fix coloring of last `n_batch` of prompt, and refactor line input by bitRAKE · Pull Request #221 · ggml-org/llama.cpp

bitRAKE · 2023-03-17T02:24:08Z

https://github.com/ggerganov/llama.cpp/blob/721311070e31464ac12bef9a4444093eb3eaebf7/main.cpp#L980-L983
This can fail to colorize the last params.n_batch part of the prompt correctly because embd was just loaded with those tokens and not printed, yet.

sw · 2023-03-19T11:21:35Z

Can confirm the bug and the PR fixes it and is certainly cleaner. Can you resolve the conflicts and maybe take along the one-liner in #283, also related to colors? Thanks.

ggerganov · 2023-03-19T18:48:51Z

@sw merge if you approve it

…ml-org#221) * fix coloring of last `n_batch` of prompt, and refactor line input * forgot the newline that needs to be sent to the model * (per ggml-org#283) try to force flush of color reset in SIGINT handler

The _mem probe measured only VRAM; host RAM was a blind spot all of the 4B memory work. Measured peak host anon = 4.81GB (not the assumed ~10GB): model is GPU-resident via device_map (+0.13GB host), the one real host chunk is the init_idx .cpu() snapshot (~3.5GB, the ggml-org#221 NVMe-reread target). RssFile is reclaimable mmap, not pressure. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

… Ada: ggml-org#218 + ggml-org#215 hybrid PTQ1_0 dispatch, in-place q4_0/q8_0 K/V flash attention (integration checkpoint) (ggml-org#221) * Add: planar-transposed activation layout for the PTQ1_0 mat-vec path The mmvq path gave every thread one 128-weight PTQ1_0 block and read the activations as 32 scattered 4-byte loads per column out of 36-byte block_q8_1 structs, so every extra column cost a full pass: 47 / 104 / 154 / 201 us for 1 / 2 / 3 / 4 columns on a 4096 x 14336 matrix (RTX 3060, test-backend-ops perf). Quantize PTQ1_0 activations into a planar-transposed layout instead: per column, 8 planes of 16-byte quant pieces and one plane of the 4 (d, s) half2 scales, 16 bytes per 128-element block each, with the same bytes per row and the same column stride as block_q8_1. A thread then reads its block's activations as 9 aligned 16-byte loads with adjacent lanes on adjacent pieces: 46 / 55 / 65 / 75 us. The trit decode is unchanged; the byte-wise minus one is (q + 0x7F7F7F7F) ^ 0x80808080 instead of __vsub4. nwarps is pinned to 4 for every column count so the K partition, and with it the fp32 summation order, does not depend on the batch. mmvq now handles PTQ1_0 up to 8 columns (MMQ took 395 us at 8). HIP keeps the old layout. * Add: dedicated PTQ1_0 mat-vec kernel with full lane utilization With 128-element blocks the generic kernel's 128 threads cover 128 K blocks per iteration, so a K = 5120 projection (40 blocks) keeps 31% of the lanes busy and K = 17408 (136 blocks) 53%. This kernel handles plain 2D MUL_MAT (no batch dims, no expert ids, K a multiple of 128, up to 8 columns): it flattens (row group, K block) pairs into one index space with rows_per_cta chosen on the host to fill whole 128-thread iterations, gives each thread 4 rows per K block (2 at 5 to 8 columns) for latency hiding and activation reuse, stores one fp32 partial per (row, column, K block) in dynamic shared memory and has one warp per (row, column) sum them in a fixed order (lane-strided sequential, then a butterfly). The per-block partial is d * sum_k d8_k * sumi_k with exact integer sumi_k, written as __fmaf_rn and __fmul_rn so every instantiation rounds alike, and the order depends only on the weight shape: a column's result is bit-identical for every column count 1 to 8. Fusion (bias, gated GLU epilogue) is supported for one column as in the generic kernel, which remains the path for batched and MoE calls. Bonsai 2 shapes, RTX 3060 12GB, us per call, 1 column: attn_qkv 5120 x 10240 72.4 to 41.8, ffn_up 5120 x 17408 120.0 to 66.7, attn_gate 5120 x 6144 44.8 to 27.7. Whole model llama-bench tg32 26.1 to 39.8 tok/s, pp2 34.1 to 61.1, pp3 33.3 to 72.0. * Test: add Bonsai 2 projection shapes to the mul_mat perf cases * Add: GGML_CUDA_BATCH_INVARIANT for batch-invariant small-batch kernels Off by default, nothing changes. When set, kernels are picked so that the per-column arithmetic of a token does not depend on how many columns (1 to 8) share the launch: F16 and BF16 matrices with at most 512 rows take the mul_mat_vec_f kernel for 1 to 8 columns on Ampere and newer (its K partition depends on K only), flash attention with up to 8 queries takes the vector kernel on Turing and newer instead of switching to the tensor-core kernel at 2 queries, and the KV split (parallel_blocks) is sized as for one query tile so the partial-softmax combine order does not depend on the batch. Together with the batch-invariant PTQ1_0 mat-vec this makes the logits of a token bit-identical whether it is decoded alone or verified inside a speculative batch of 2, 3 or 4 tokens (measured on Ternary Bonsai 2 27B: greedy output with draft-mtp equals greedy output without it on three prompts at n-max 1, 2 and 3). Batches of 5 to 8 agree with each other but not with 1 to 4. Costs throughput at depth (the vector attention kernel at 3 queries over 42K keys). * Fix: use the mat-vec kernel for bf16 matrices under 64 rows at 2 to 8 columns The qwen35 gated-delta-net gate projections are bf16 [5120 x 48]. At 2 to 8 columns on Ampere they took a 10.5 us path where mul_mat_vec_f takes 4.4 to 7.3 us (test-backend-ops perf, RTX 3060); at one column both already used mul_mat_vec_f (3.3 us). 96 calls per token on the 27B model. * cuda: fold the recurrent-state gather into the GATED_DELTA_NET kernel (+3.8% TG on Ada) build_rs gathers each layer's live recurrent state via GET_ROWS(cache, s_copy) into a temp (3 MB per layer for Bonsai 2 27B) that only the GDN kernel reads. Traced in-graph with CUPTI on an RTX 4070 the gather kernel costs ~8 us/layer, and its dirty output sits in L2 until the FFN weight stream evicts it, which made the gate/up GEMVs in GDN layers ~15% slower than the identical GEMVs in attention layers. When the GET_ROWS output has no consumer other than (optionally through a RESHAPE) the GDN node's state input, the graph evaluator skips the GET_ROWS node and records the gather (cache base, ids, row stride) for that GDN node. The kernel then indexes the cache row s_ids[seq] directly; no temp, no kernel, no writeback. Single-sequence only: with several sequences a gathered row may alias a row another sequence writes in the same op. GGML_CUDA_GDN_GATHER_FUSION=0 disables it. The registrations live in the evaluating ggml_backend_cuda_context (ggml_cuda_gdn_gather_context, beside the concurrent-stream state), keyed by node pointer and reset at the start of every graph evaluation, so two contexts or host threads decoding at once never see each other's skips. This replaces the process-wide static map in the first revision of ggml-org#215. Independent of the activation layout change; split out of ggml-org#215 so that layout can be compared with ggml-org#218 on its own. RTX 4070 12 GB, Ternary-Bonsai-2-27B-PTQ1_0, llama-bench -fa 1 -ctk/-ctv q4_0, stock clocks, three alternating rounds of r=3: tg128 56.4 -> 58.5 t/s on top of ggml-org#215 (+3.8%). pp512 unchanged. test-backend-ops CUDA0 vs CPU: GATED_DELTA_NET 39/39 supported cases, GET_ROWS unchanged. 400 greedy tokens identical with GGML_CUDA_GDN_GATHER_FUSION=0 and 1. * cuda: PTQ1_0 decode on consumer Ampere/Ada: SoA q8 activations, warp-per-row small-K GEMV (+14% TG) Merged onto the planar-transposed activation layout and dedicated 2-8 column kernel (PrismML-Eng#218, sudoingX). One decision, ggml_cuda_q8_1_layout_for(type, ncols_dst, ids), is evaluated by both the quantizer call and the kernel switch: PTQ1_0, 1 column, no ids -> SOA_ISUM (this change): exact int16 sums, warp-per-row small-K geometry PTQ1_0, 2-8 columns / MoE -> PT (ggml-org#218): shared weight decode, full lane utilization anything else -> plain block_q8_1 GGML_CUDA_BATCH_INVARIANT=1 forces the one-column case onto PT as well so a token decoded alone and a token verified inside a speculative batch still see identical arithmetic. * cuda: PTQ1_0 hybrid dispatch: template y_soa, MMQ from 5 columns The SoA/planar choice is a kernel template parameter (only instantiated true for PTQ1_0 at one column) instead of a runtime branch inside the K loop; the runtime branch cost 5% TG on Ada. ggml_cuda_should_use_mmvq keeps PTQ1_0 on the mat-vec path up to 4 columns: with the branch-free MMQ tile loader the tile path overtakes the planar mat-vec kernel at n=5 (RTX 4070, Bonsai 2 27B: 163 vs 137 t/s at n=5, 258 vs 148 at n=8). * cuda: flash attention MMA reads q4_0/q8_0 K/V in place (no F16 scratch copy) The tensor-core flash attention path converted the whole quantized K/V cache to F16 in a scratch buffer on every micro-batch: 1 GB per context at 262k with q4_0 caches, 2 GB with a draft context, and a full re-conversion per ubatch during prefill. With Bonsai 2 27B + MTP that pushed the 4070 12 GB over the edge and WDDM spilled to system RAM (41 tok/s). flash_attn_ext_f16 now takes type_KV as a template parameter. For q4_0 and q8_0 the tile loader dequantizes blocks straight from the cache into the shared-memory half2 tiles (aligned 32-bit word reads, scale from the block header), the cp.async multi-stage pipeline is disabled for those types (cp.async cannot dequantize), and K/V strides are byte strides. The allocator no longer reserves the F16 copy when the native kernel handles the call. Dispatch is limited to D=128/256 heads with matching K/V types and word-aligned strides; anything else keeps the old conversion path. test-backend-ops: 48 new FLASH_ATTN_EXT cases for q4_0/q8_0 at D=128/256, GQA, batch 1-8, KV lengths incl. non-multiple of the tile, permuted layouts. All pass against the CPU reference; the full suite is 2984/2984. Bonsai 2 27B PTQ1_0 + MTP, RTX 4070 12 GB, 262144 context, ub 128: resident, 222 MiB shared (no spill), 86 tok/s mean over 3 prompts. * cuda: PTQ1_0 multi-column mat-vec: raw digits with exact activation sums, per-pair epilogue (+13% 3-col, +24% 4-col) Two changes to mul_mat_vec_ptq1_0_pt, found with Nsight Compute on an RTX 4070 (sm_89). 1. The trit decode returns the raw base-3 digits {0,1,2} and the dot product uses a mixed-sign dp4a (u8 x s8) against the activations, subtracting the exact int16 sum of the 32 quantized activations once per sub-block: sum((q-1)*a) = sum(q*a) - sum(a). The PT quantizer now stores that sum bit-cast in the ds.y half slot (the SoA layout already did). Two ALU ops fewer per four weights; the result is the same integer, so output is bit-identical. 2. The epilogue summed each (row, column) pair with one warp: a runtime integer division per pair, a lane-strided loop and a 5-step shuffle butterfly. Under the profiler that was 23% of issued instructions and ~50% of warp stall samples (FADD on SHFL scoreboard, MUFU.RCP/I2F/F2I from the division) while the CTA held its slot and moved no bytes. Now one thread per pair sums the K blocks sequentially from shared memory with four interleaved accumulators, over a padded (odd) row stride so the reads are bank-conflict-free. The order still depends only on the weight shape, so the batch-invariant guarantee holds. Profile of the 3-column gate|up GEMV (K=5120, N=17408) before -> after, ncu locked clocks: 2135 -> 1493 instructions per warp, 56.3 -> 43.5 us, DRAM 62% -> 83% of peak. llama-bench, +1500 MHz memory: pp3 150 -> 169 tok/s, pp4 172 -> 212. pp1 unchanged (SoA path). test-backend-ops -b CUDA0 -o MUL_MAT: 1283/1283. * qwen35: MTP graph publishes only the output rows of h_nextn when embeddings_nextn_masked llama_context::decode copies the first n_outputs rows of t_h_nextn when embeddings_nextn_masked is set, but the QWEN35 MTP builder assigned t_h_nextn before the inp_out_ids gather, so the tensor had n_tokens rows. Any MTP batch with more tokens than outputs (e.g. catch-up rows decoded together with a draft row) returned row 0's hidden state for the draft row and the next draft step ran on the wrong input. Never triggered before because every MTP decode had n_tokens == n_outputs or no outputs. Gather before publishing in the masked case, exactly as the trunk graph does at its last layer; the unmasked case is unchanged. * speculative: draft-mtp decodes the catch-up rows together with the first draft row process() used to run the draft over every row of the target's verify batch immediately, purely to fill the draft's memory, and partly on rows the target then rejected. That is one graph launch and one host round trip per step spent on nothing the next draft needs yet. Single-head MTP with its own memory now keeps those rows (tokens, positions, shifted h) and decodes only the accepted prefix at the start of draft(), in the same llama_decode as the first draft row. Positions are contiguous and the draft row attends to the catch-up rows in-batch, so the draft is unchanged; accept() trims the stash to the accepted prefix, and rows are decoded on their own when a batch does not lead straight into a draft (prompt fed in tiny ubatches, drafting stopped, rollback replay overwrites them). Shared-memory and chained-head drafters keep the eager path, as does LLAMA_MTP_EAGER_CATCHUP=1 for A/B. RTX 4070, Bonsai-2-27B PTQ1_0 + MTP head, 262k context, q4_0 K/V, greedy, --backend-sampling: 23.0 -> 22.4 ms/step, three-prompt mean 101.0 -> 103.3 tok/s with identical draft acceptance (0.86 / 0.46 / 0.65) and byte-identical output vs the draft head off. Draft-side launches per step 4 -> 3. * cuda: Hadamard transform quantizes its own output when every consumer is a PTQ1_0 mat-vec A Hadamard-rotated activation ([MUL signs, RESHAPE,] MUL_MAT with GGML_HINT_SRC0_IS_HADAMARD) whose every use is src1 of a PTQ1_0 mat-vec on the mmvq path never needs to exist in F32: each consumer immediately quantizes it to q8_1. The transform kernel now quantizes as it goes and writes the q8_1 rows, in the layout the consumers expect (AoS / PT / SoA with exact integer sums and the same padding rule as ggml_cuda_mul_mat_vec_q), into its own output buffer; the mat-vecs find the rows through a per-context registry keyed by tensor identity and skip their quantize launch. On Bonsai 2 this removes ~390 launches per decode step. Safety rules the pass enforces: - every later node that reads the transform output (directly or via a whole-tensor reshape view) must be a PTQ1_0 MUL_MAT that ggml_cuda_mul_mat routes to mmvq, and the graph use count must match, so nothing ever reads the buffer as F32 - the registry is keyed by ggml_tensor pointer, never by data pointer: ggml-alloc recycles a dead output's block for later tensors in the same graph - when the output buffer aliases the input (ggml-alloc does this for the output-projection input on every Bonsai layer: x is a reshape view of a cont, there is no in-place reuse through a view parent, so the cont is released at the MUL and mm takes its block two nodes later) the rows go to a pool block held until the next graph evaluation. Writing into the aliased buffer raced the kernel's own input reads and produced sporadic garbage that, on a hybrid model, entered the recurrent state and poisoned the rest of the sequence (surfaced as INT_MAX from the backend argmax over an all-NaN logits row, about once per ~170 requests) - GGML_CUDA_FWHT_FUSION=0 disables the pass fwht.cuh: the block butterfly is a shared inline so fwht.cu and the fused quantizer stay one implementation. Validated: 340 consecutive requests (probe.py x 20 at 262144 ctx, q4_0 KV, draft-mtp n_max=2, backend sampling) with zero errors; output byte-identical to the unfused path. * sampling: reject out-of-vocab token ids from the backend sampler and the draft A backend-sampled id outside [0, n_vocab) (the CUDA argmax/top-k pad sentinel INT_MAX when a logits row is all NaN) used to travel as-is into the server, where the tokenizer's vector::at() threw "invalid vector subscript" and killed the request. Log the origin (idx, n_vocab, first candidate) and fall back to the CPU chain instead; the draft-mtp loop drops a draft whose top candidate id is out of vocab rather than feeding it to the verify decode. * speculative: draft-mtp drops stale deferred rows when a new task starts on the slot begin() used to decode whatever catch-up rows the previous task left pending. That is right only when the new prompt reuses them (same tokens at the same positions, continuing what ctx_dft holds). After a cancelled task or a different conversation landing on the slot, the server has already trimmed ctx_dft past those rows and decoding them writes stale positions into the recurrent draft state. Check first, flush when they belong, discard otherwise. * cuda: release the FWHT-q8 held pool blocks LIFO and before the pools are destroyed The registry keeps pool blocks alive across a graph when the fused Hadamard quantize would otherwise write over its own input. Its implicit destructor released them oldest first, and it is declared before the pools, so at context teardown the frees hit a destroyed VMM pool: GGML_ASSERT(ptr == pool_addr + pool_used) at the end of llama-bench. Release newest first in a destructor, and reset() from the context destructor ahead of the pools. * server: --spec-draft-depth-max stops drafting once the sequence is deep Deep in the context a decode step is bound by reading the KV cache. The draft passes and the multi-column verify add to that read without shortening it, so past some depth speculation costs more than the accepted tokens save: with Bonsai 2 27B on an RTX 4070 the MTP draft is +85% at zero depth, even near 24-32k tokens and -27% at 64k. Past the cutoff the slot decodes one token per step and the process() hook is skipped for batches that sit entirely beyond it (by position, so the early ubatches of a long prompt still reach the draft context and remain reusable as a prefix). Default 0 keeps the old behaviour. 0 / 32k / 64k tokens: 101 / 48 / 37 tok/s with a 24k cutoff against 101 / 45 / 27 without. * cuda: restrict off PTQ1_0 kernel params; Ampere uses the PT 1-col path. C2912 / stub mismatch on sm_90 and sm_120 when GGML_CUDA_RESTRICT is on the formal parameters of mul_mat_vec_ptq1_0_pt (same hunk as 2578fdf on ggml-org#218). Ampere (cc 800-889) takes the planar 1-col kernel; Ada and newer keep SoA. 3060: ggml-org#218 was +5.9% tg128 vs ggml-org#215 at one column. * cuda: address Copilot review on native q4/q8 FA, MTP catch-up, and BOM. Disable drafting when deferred catch-up decode fails. Gate in-place quantized K/V FA to NVIDIA and require ggml_cuda_is_aligned on K and V. Strip UTF-8 BOMs. The FWHT fusion comment no longer opens with the GDN gather notes. * cuda: share the PTQ1_0 mat-vec smem budget between guard and launch The combo kernel pads each partials row by one float. Use the same byte count in the entry guard as the launch, including that pad, so oversized gated shapes fall back instead of requesting more than smpb. * spec: flush MTP catch-up when it plus anchors would overflow n_batch process() can stash n_batch rows; draft() then added one more anchor. Flush first when the combined count would exceed llama_n_batch. * cuda: restore the scalar PTQ1_0 vec-dot on MUSA Same hole as ggml-org#215: the generic entry returned zero once MUSA was kept off the SoA path. MUSA now takes the HIP scalar implementation. * spec: keep MTP catch-up across a failed draft decode catchup_failed was set on a flush failure and never cleared, so one bad decode disabled that slot for every later task. begin() now resets it with the rest of the per-task state. draft() also cleared pending while building the batch, so a failed llama_decode dropped those rows from both the stash and ctx_dft. pending stays set until that first decode succeeds, and a failure marks the slot so it is not drafted again until the next begin(). flush_deferred waits for success the same way. The layout_for comment now says it is the column-count helper. ggml_cuda_q8_1_layout_host is the decision both sides call. * cuda: define the PTQ1_0 layout helper after ggml_cuda_info The host layout decision reads the current device's compute capability. It was declared above ggml_cuda_info and ggml_cuda_get_device, so a clean sm_89 build failed in every CUDA translation unit. The helper now sits below those declarations. * cuda: land remaining ggml-org#221 review notes: pass q8 layout down, skip HIP/MUSA q4/q8 FA instances, exercise ALiBi/softcap. * cuda: wait on the PDL dependency in the PTQ1_0 planar mat-vec (Hopper/Blackwell). * cuda: restore warp-reduce epilogue under BATCH_INVARIANT so MTP-on matches MTP-off. The four-accumulator path (5300cd1) is faster but a 5080 run flipped a late near-tie. GDN now waits on PDL before reading s_ids. --------- Co-authored-by: sudoingX <200180104+sudoingX@users.noreply.github.com> Co-authored-by: Cary Palmer <24235924+professorpalmer@users.noreply.github.com> (cherry picked from commit 5244cea) Assisted-by: GitHub Copilot

Add --spec-draft-depth-max and stop feeding draft context rows beyond its cutoff. Keep early prompt rows reusable and fall back to ordinary target decoding after the cutoff. Reject out-of-vocabulary MTP draft candidates. Based on Cary Palmer and sudoingX's speculative controls and candidate guard from Prism ggml-org#221. Source: PrismML-Eng@5244cea Co-authored-by: sudoingX <200180104+sudoingX@users.noreply.github.com> Co-authored-by: Cary Palmer <24235924+professorpalmer@users.noreply.github.com> Assisted-by: GitHub Copilot Assisted-by: OpenAI Codex Ported and rebased by Codex.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com>

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.

…B on Intel Arc (port of Torchit1/arc-b580) (ggml-org#356) * Bonsai 2 27B (PTQ1_0) on Intel Arc B580: SYCL + Vulkan performance work - SYCL: PTQ1_0 weights repacked to 2-bit at load and multiplied on XMX via the TernSYCL int8 x int2 DPAS kernels (vendored, BSD 3-Clause), activations quantized to int8 with round-to-nearest (GGML_SYCL_PTQ1_T2=ffn|all). - SYCL: decode attention for q4_0 KV serving 1-4 query tokens per launch; blocked gated delta-net; PTQ1_0 mat-vec kernels (single column, decode-once multi column, same-input merge, fused gate/up); narrow concat. - Vulkan: matching PTQ1_0 multi-column, decode attention, gated delta-net and coopmat work. - Speculative decoding: draft KV cache type (-ctkd/-ctvd) honoured by the MTP draft context, LLAMA_ARG_SPEC_DRAFT_UBATCH to shrink its compute buffer. - docs/bonsai-arc-b580.md: results at 128K on a B580, build and run instructions, switches, credits. Based on the bonsai-combo branch of professorpalmer/llama.cpp-ada-ternary (PrismML PR ggml-org#221). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 753323e on Torchit1/llama.cpp arc-b580) Excluded from this port: .github docs ggml/src/ggml-vulkan Conflicts resolved against prism: fwht.cpp keeps prism's FWHT (ggml-org#302) and adds this commit's signed variant beside it as ggml_sycl_op_fwht_signed; ggml_sycl_op_mul_mat_sycl takes fp16 for PQ2_0/PTQ1_0 as before (ggml-org#278) plus this commit's large-batch rule; the T2 path runs before the PQ2_0 XMX path (ggml-org#294), which still serves PQ2_0/PTQ1_0 when GGML_SYCL_PTQ1_T2 is unset. * sycl: PQ2_0 weights on the TernSYCL XMX path too (GGML_SYCL_PTQ1_T2) PQ2_0 (2-bit codes 00=-1 01=0 10=+1, fp16 scale per 128) is the same size as the 2-bit XMX layout, so it is remapped in place. Code 3 (+2) has no 2-bit two's-complement form: the repack reports it and the matmul aborts with a message. test-backend-ops MUL_MAT type_a=pq2_0 with GGML_SYCL_PTQ1_T2=all: 84/84; ptq1 94/94. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 5056bde on Torchit1/llama.cpp arc-b580) * sycl: PTQ1 T2 path only on Xe2+ GPUs, fall back with a warning elsewhere The T2 kernels use 16-lane int8 x int2 DPAS builtins that only Xe2 and newer drivers provide; on an Arc 140T (Xe-LPG+) GGML_SYCL_PTQ1_T2=all failed with 'No kernel named ... was found'. Now checked once against the device architecture. GGML_SYCL_PTQ1_T2_ANYGPU=1 skips the check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 731c1a8 on Torchit1/llama.cpp arc-b580) * sycl/ternsycl: 64-bit surface base address on Windows The 2D block I/O builtins took the surface base as 'long', which is 32 bits in SYCL device code on Windows (the device follows the host's LLP64 data model), so the weight loads would have used truncated addresses. 'long long' is 64 bits on both; identical code on Linux. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 2f0cf6e on Torchit1/llama.cpp arc-b580) * sycl: GGML_SYCL_PTQ1_T2_TILE lab knob for the TernSYCL large-M tile (default stays best) B580 pp1024 (ub 1024, all weights on XMX): 8x128/8x2 939 t/s (default), 8x128/16x1 949, others 480-857. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 244c4d4 on Torchit1/llama.cpp arc-b580) * sycl: two-pass top-k for long rows (GGML_SYCL_TOPK_OLD=1 restores the one-group kernel) The one-work-group top-k took ~550 us per MTP draft token on the B580 (248K-entry vocabulary). Now up to 256 groups per row with a pairwise local-memory merge, then one group merges the winners; ties keep the lower index. test-backend-ops TOP_K 325/325. 200K entries: k=1 241 -> 9.6 us, k=10 647 -> 71 us. Live settings: fresh +1-2%, rename/edit within noise (the speculative step is not purely GPU-bound). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit afe4c82 on Torchit1/llama.cpp arc-b580) * sycl: q4_0 decode attention on XMX via s8 x s4 DPAS (opt-in GGML_SYCL_FA_DEC_DPAS) Q.K uses intel_sub_group_i8_i4_matrix_mad_k32 (IGC provides it on Xe2 though the public extension only lists i4_i4_k64): the raw 16-byte q4_0 block of a key XOR 0x88888888 is the B operand, one key per lane, against int8 Q with one RNE scale per row and block (q4_0 byte order absorbed by pairing dims l and l+16 in A). P.V computes O^T = V^T P^T on f16 DPAS so each lane dequantizes its own key's V block from 5 dword loads. Token chunks of 8 let one kernel serve 1..32 (or =N) query tokens, so n-gram verify batches stop falling back to oneDNN + a full-KV f16 conversion. Per layer (Bonsai 27B shape, real cache layout) old -> new, us: 16K 1 tok 266 -> 123, 4 tok 457 -> 244, 16 tok 2217 -> 776; 48K 1 tok 768 -> 433, 4 tok 1295 -> 671, 16 tok 6629 -> 1915. FLASH_ATTN_EXT decode cases incl. 12/16/17/32 tokens and the permuted layout: 57/57. GGML_SYCL_FA_DEC_DPAS_PROBE skips phases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit e5a4d8a on Torchit1/llama.cpp arc-b580) * sycl: MKL flash-attention softmax, one work-group per query row mkl_fa_online_softmax_chunk gave each query row to one work-item that walked the whole KV chunk twice, so neighbouring work-items read addresses chunk_size floats apart. Above the oneDNN KV cap this kernel was 72% of generation time (Bonsai 27B, 115K-token context: verify batches > 32 tokens and every new prompt). Now 256 work-items per row with coalesced reads and group reductions; same math. Bonsai 27B, live flags, 114.6K-token code context: new code 17.6 -> 44.0 t/s, edit 10.8 -> 27.5 t/s, identical output; short prompts unchanged. FLASH_ATTN_EXT with the MKL path forced (ONEDNN_MAX_KV=512): 82/84, the same two pre-existing prompt-batch failures as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit a122b2b on Torchit1/llama.cpp arc-b580) * sycl: GGML_SYCL_MMVQ_CHUNK_MAX, largest batch run as MMVQ column chunks Quantized mul_mat with 33+ columns fell through to dequantize-all-of-src0 + GEMM per call. For speculative-verify sized batches that is ruinous: Gemma 4 12B q4_0 with n-gram drafts of 33-64 tokens spent ~94% of GPU time in dequantize_row_q4_0_sycl_reorder. Default stays 4 x MMVQ_MAX_BATCH_SIZE (32); with 128, Gemma 4 12B + QAT assistant (ngram n-max 64): rename 79 -> 186 t/s, edit 57 -> 136. Bonsai 27B unchanged within noise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit e8bd534 on Torchit1/llama.cpp arc-b580) * sycl: oneDNN / MKL flash attention only for packed interleaved K/V Reported from an Arc 140T and reproduced on a B580: FLASH_ATTN_EXT with quantized K/V in the permuted layout returned ERR ~1.2 for every KV length from 1024 up (ggml-org#27769). The failing test builds the cache view with a gap between tokens (2x the heads allocated, half viewed); the oneDNN SDPA and MKL dequant paths take the token stride from the head count. llama.cpp's own cache view is always packed, so real models were unaffected (Bonsai 27B perplexity at 4K: oneDNN 8.7273, MKL 8.7311, TILE 8.7330, and still 8.7273 with this guard). Interleaved K/V with a gap now falls back to TILE. FLASH_ATTN_EXT 4137/4137 (was 2 failing), plus a new q4_0 real-layout sweep (kv 1024..8192, nb 33/64/512) 29/29. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 226166a on Torchit1/llama.cpp arc-b580) * sycl: XMX decode attention only on Xe2+, fall back with a warning elsewhere GGML_SYCL_FA_DEC_DPAS uses the 16-lane int8 x int4 DPAS builtin that only Xe2 and newer provide; on Alchemist / Xe-LPG it would fail at the first launch (like the T2 path did before 9c25b48). Now checked once against the device architecture, sharing ggml_sycl_device_is_xe2() with the T2 gate; GGML_SYCL_FA_DEC_DPAS_ANYGPU=1 skips the check. B580 unchanged: 4-token verify at 48K 671 us (1295 without). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit d441076 on Torchit1/llama.cpp arc-b580) * sycl: XMX q4_0 decode attention for Gemma 4 (head 512 x GQA 16 global, head 256 x GQA 2 sliding) The s8 x s4 DPAS decode kernel takes the head size as a template parameter (K read block by block, P.V subgroups own D/16 dims) and serves two more shapes when GGML_SYCL_FA_DEC_DPAS is on: Gemma 4's global layers (512, 1 KV x 16 Q heads, token chunks of 2) and sliding layers (256, 2 per KV head, chunks of 8). Splits target ~256 work-groups in all (a single KV head used to leave the GPU idle); combine step templated on D. Global-layer attention at 48K, us: 1 token 1114 (f16 TILE) -> 236, 4 tokens 2011 -> 635. Gemma 4 12B QAT + assistant, q4_0 KV, 128K context: short 92 -> 100 t/s new code; at 48K new code 47 -> 65, edit 44 -> 57 (f16 KV was 55 / 47). Bonsai shape unchanged (4-token verify at 48K 638 us). FLASH_ATTN_EXT: Gemma global 20/20, sliding 10/10, Bonsai 36/36. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit c759ddb on Torchit1/llama.cpp arc-b580) * sycl: reordered q4_0 small-batch GEMM on XMX via s8 x s4 DPAS (opt-in GGML_SYCL_Q4_0_DPAS) Verify batches of 5..256 columns on q4_0 weights used MMVQ column chunks (weights re-read per 8 columns) or dequantize-everything + GEMM. The new kernel streams each weight row once for up to 64 columns (chunks of 64 beyond): one lane per weight row, B = the row's 16 q4_0 bytes XOR 0x88888888 through intel_sub_group_i8_i4_matrix_mad_k32, A = activations quantized once per call to int8 (per row and 32-block, elements l / l+16 paired). Xe2 only (ggml_sycl_device_is_xe2). GGML_SYCL_Q4_0_DPAS_MIN / _MAX (5 / 256). 15360x3840 q4_0: 8 columns 150 -> 81 us, 16 columns 292 -> 89 us. Gemma 4 12B QAT + assistant: rename 188 -> 275 t/s, edit 138 -> 168 (ngram n-max 96), new code unchanged, rename/edit outputs byte-identical. MUL_MAT q4_0 Gemma shapes 1..256 columns: 20/20. Also Gemma 4 q4_0 mat-mul perf cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 03594d5 on Torchit1/llama.cpp arc-b580) * sycl: large T2 prompt batches through oneDNN's int8 GEMM (GGML_SYCL_T2_W8A8_MIN) Before the TernSYCL path went live, PTQ1_0 prompts ran on the W8A8 oneDNN GEMM (pp512 1013); GemmMT is better for verify-sized batches but slower on full prompt batches. New ggml_sycl_w8a8_mul_mat_t2 converts the repacked 2-bit layout to int8 (per-row scale, group ratios) and uses the same gemm_s8. The conversion is a fixed cost per call, so it only pays from ~1024 columns: Bonsai 27B pp1024 (ub 1024) 941 -> 1062, pp512 901 -> 818 (so MIN=1024 on the live config); 48.6K-token prompt read 67.7 -> 61.6 s (719 -> 789 t/s). PPL c4096 8.7273 -> 8.7359 (+0.1%, int8 rounding as the old W8A8 path). Off by default (0). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit a68382d on Torchit1/llama.cpp arc-b580) * SYCL decode attention: SLM-staged K/V tiles for 1 query token, one-wave split count; PTQ1_0 T2 NSG knob fattn-dec (DPAS q4_0 path, D 256): single-token decode copies each 128-key K and V tile into SLM with overlapped 16-byte loads and reads lane = key from there; the split count now fills one wave of work-groups (Xe cores x work-groups per core / KV heads) instead of a fixed 64 per head. B580: 48K 1 token 433 -> 244 us, 4 tokens 640 -> 456 us; whole model plain decode +12% at 48K, +21% at 115K, same KLD and output. GGML_SYCL_FA_DEC_STAGE=0 and GGML_SYCL_FA_DEC_SPLITS=64 restore the old behaviour. ptq1-t2: GGML_SYCL_PTQ1_T2_NSG=4 (4 sub-groups for 2-8 row GEMV, +~1%, default off). Co-Authored-By: Claude <noreply@anthropic.com> (cherry picked from commit 9cfffcb on Torchit1/llama.cpp arc-b580) * SYCL: s8 x s4 DPAS as inline vISA instead of the intel_sub_group_i8_i4_matrix_mad_k32 builtin The builtin is not in the public subgroup matrix extension list. On a Lunar Lake (Arc 140V) driver that does not provide it, loading the device image failed with "No device image found for external symbol _Z36intel_sub_group_i8_i4_matrix_mad_k32" and every model crashed at the first decode. xe2-dpas.hpp emits the dpas.s4.s8 instruction directly (as libxsmm/TernSYCL's xe2.hpp helpers), so the symbol is gone from libggml-sycl. Used by the q4_0 decode attention and the q4_0 small-batch GEMM. B580: FA 4167/4167, MUL_MAT q4_0 (GGML_SYCL_Q4_0_DPAS=1) 59/59; whole model output byte-identical to the builtin at short / 48K / 115K context, plain and MTP. Single-token attention kernel ~6% slower (48K 244 -> 259 us), whole model within 1-2%. Co-Authored-By: Claude <noreply@anthropic.com> (cherry picked from commit f31a4f6 on Torchit1/llama.cpp arc-b580) * SYCL: no IGC builtins left (TernSYCL main, f16 DPAS via vISA), device code split per kernel - ternsycl/: libxsmm/TernSYCL main (11484da) + the long long address fix of TernSYCL#3. Inline vISA only, so the undocumented builtins the old copy declared (intel_sub_group_i8_i2_matrix_mad_k32, __builtin_IB_* 2D block I/O) are gone; it also drops our three JIT workarounds. The per-32 activation scale variant (off by default) is removed. - ptq1-t2.cpp is compiled with -fp-model=precise: under icpx's default fast model the TernSYCL GemmMT returns NaN (upstream's own driver fails validation built that way). - fattn-dec: the f16 P.V DPAS also goes through inline vISA (xe2-dpas.hpp dpas_hf_r8). - CMake: -fsycl-device-code-split=per_kernel (GGML_SYCL_DEVICE_CODE_SPLIT_PER_KERNEL, default ON): the Xe2 XMX kernels are only JIT-compiled when launched, so GPUs without DPAS (Xe-LP, Gen9) don't compile them at all. - test-backend-ops: FA cases for 64-256 query tokens and ~115K history. B580: libggml-sycl has 0 matrix_mad / __builtin_IB references (was f16, i8_i2, 9 IB). FA 4167/4167, MUL_MAT q4_0 59/59, ptq1_0 T2 94/94. KLD (ub 1 / 4 / 512) identical to the previous build. Whole model output identical; MTP 48K 73.6 t/s, 115K 59.6 (was 69.6 / 57.2), plain unchanged. Startup unchanged (first request 4 s). Co-Authored-By: Claude <noreply@anthropic.com> (cherry picked from commit 2e8a0e6 on Torchit1/llama.cpp arc-b580) * SYCL decode attention: default DPAS cap 32 -> 256 query tokens Long n-gram verify batches now stay on the q4_0 DPAS kernel instead of oneDNN's path, which converts the whole KV cache to f16 each call. B580 per layer: 48K 64 tokens 28.8 -> 4.9 ms, 115K 256 tokens 253 -> 44 ms. KLD (ub 64-256) 0.012699 vs 0.012800, same top p 92.60 vs 92.89 %, equal within error. Whole model MTP: 48K 73.2 -> 76.1 t/s, 115K 60.3 -> 65.1. GGML_SYCL_FA_DEC_DPAS=32 restores the old cap. Co-Authored-By: Claude <noreply@anthropic.com> (cherry picked from commit 064461a on Torchit1/llama.cpp arc-b580) * SYCL decode attention: 5-token query chunks for MTP verifies MTP with 4 drafts verifies 5 tokens per step, which ran the 8-token DPAS kernel with 3 empty tokens (48 rows of Q.K / softmax for 30). A 5-token chunk (128-key tiles) cuts the kernel from 637 to 502 us per layer at 48K (215 -> 175 us at 16K); whole model MTP 48K 77.4 -> 78.1 t/s, 115K 65.7 -> 67.1, output identical. GGML_SYCL_FA_DEC_NQ5=0 turns it off (=2 SLM-staged and =3 256-key tiles were slower). test-backend-ops: 5/6-token real-layout cases. Co-Authored-By: Claude <noreply@anthropic.com> (cherry picked from commit 081d868 on Torchit1/llama.cpp arc-b580) * lab: runtime-switchable SYCL tuning knobs (GGML_SYCL_KNOBS_FILE); LLAMA_ARG_SPEC_NGRAM_FRESH - ggml_sycl_knob(name, def): the attention (DPAS cap, staging, 5-token chunks, split count, probe) and PTQ1_0 T2 (tile, sub-groups, GEMM off) switches are read through it. With GGML_SYCL_KNOBS_FILE set, the NAME=VALUE file is re-read at the start of a graph compute when it changes, so one server can A/B variants. Without it, each knob is read from the environment once, as before. - LLAMA_ARG_SPEC_NGRAM_FRESH=1: the n-gram drafter forgets earlier requests at the start of each one, so repeated benchmark prompts don't replay earlier answers as drafts. Co-Authored-By: Claude <noreply@anthropic.com> (cherry picked from commit 9db9408 on Torchit1/llama.cpp arc-b580) Excluded from this port: common * sycl: T2 quantized-activation cache + fused Hadamard rotate/quantize Back-to-back T2 mat-vecs on one tensor reuse its int8 activations (GGML_SYCL_PTQ1_T2_QCACHE), and the signed 1024-point FWHT that feeds them runs 128 lanes per block and writes Aq/SA itself (GGML_SYCL_PTQ1_T2_FWHTQ). B580 Bonsai 27B plain decode +3.6-4.9%, KLD unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 254dfb2 on Torchit1/llama.cpp arc-b580) * sycl: fuse residual add + RMS norm into the T2 rotate/quantize kernel ADD + RMS_NORM + MUL(weight) [+ MUL(signs) + RESHAPE + Hadamard MUL_MAT] runs as one work-group per token (GGML_SYCL_PTQ1_T2_NORMQ). Outputs only used inside the chain are not written; across tokens, outputs must not partly overlap inputs. B580 Bonsai 27B plain decode +1.9-2.6%, KLD unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 4d50f84 on Torchit1/llama.cpp arc-b580) * sycl: vendored TernSYCL -> 8b0ae42, drop the -fp-model=precise override Upstream fixed the GemmMT row-scale broadcast that returned NaN under icpx's default -fp-model=fast (libxsmm/TernSYCL#1). B580: MUL_MAT ptq1_0/pq2_0 178/178 with fast math, KLD unchanged, decode speed unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 546f94c on Torchit1/llama.cpp arc-b580) * sycl: lab knob GGML_SYCL_T2_ZERO_WEIGHTS (zeroes T2 weights to profile non-weight cost) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 2dbd839 on Torchit1/llama.cpp arc-b580) * sycl: lab instrumentation - host token timing (GGML_SYCL_HOST_TIMING), W8A8 threshold as a runtime knob; test shapes GGML_SYCL_HOST_TIMING=1 splits decode graphs into between / enqueue / wait host time plus per-op enqueue cost, without a tracer. test-backend-ops: Bonsai attn k/v and output-head mat-vec shapes, FA nh=1,2 at 48K. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit ec71714 on Torchit1/llama.cpp arc-b580) * sycl: fuse the DeltaNet conv chain for one-token decode (GGML_SYCL_CONV_CHAIN) CONCAT + conv-cache CPY + SSM_CONV + SILU + L2_NORM in one kernel per layer (one work-group per 128-channel head). Outputs are written at the CONCAT's position, so nodes in between must not touch them (checked); an output may sit exactly on the qkv input (same element per channel), and an L2 output on top of the old conv state falls back to its own L2_NORM kernel. Also GGML_SYCL_FAST_SET_TENSOR (opt-in pinned input uploads). B580 Bonsai 27B plain decode +1.3-1.8%, KLD ub1 unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit b56951e on Torchit1/llama.cpp arc-b580) * sycl: fuse the DeltaNet gate block (GGML_SYCL_DN_GATES) bf16 alpha + beta mat-vecs (2 x 48 rows, same input, up to 8 tokens) with the dt-bias add, softplus, ssm_a multiply and sigmoid in one kernel: 9 graph nodes / 7 launches per layer -> 1 (2 when an output aliases the input and goes through a scratch buffer). GGML_SYCL_DN_GATES_DEBUG=1 prints why a block is not fused. B580 Bonsai 27B plain decode +0.5-1.4%, KLD unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 38c6be5 on Torchit1/llama.cpp arc-b580) * sycl: same-input T2 mat-vecs in one launch (GGML_SYCL_T2_MULTI) q+k+v, GDN qkv+z and FFN gate+up (<= 8 tokens) share one quantization and one GEMV launch: TernSYCL's Gemv body now takes its work-group indices (run()), and a wrapper maps each group to its matrix through a single call site (three inlined copies spilled: 34.6 vs 46.3 t/s). Grouping and output staging as the PTQ1_0 merge. B580 Bonsai 27B plain decode +1.8-2.0%, KLD unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 31ae86e on Torchit1/llama.cpp arc-b580) * sycl: fold FFN SwiGLU and the DeltaNet pre-norm into the T2 rotation SWIGLU -> MUL(signs) -> RESHAPE -> Hadamard MUL_MAT: the rotate/quantize kernel computes silu(gate) * up on its load (GGML_SYCL_GLU_ROT). ADD + RMS_NORM + MUL whose rotation comes ~9 nodes later (DeltaNet layers): the rotation is computed with the norm when no node in between touches its output (GGML_SYCL_NORM_ROT_FAR). Same arithmetic, byte-identical outputs. B580 Bonsai 27B plain +0.3-0.8%, MTP rename +2%, edit +1.1%. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit cf4b155 on Torchit1/llama.cpp arc-b580) * sycl: per-batch-size GemmMT tile for T2 (small tile up to 32 rows); GEMV_MAX knob arctune sweep on the B580: tile {8,128,2,4} is 7-11% faster per pass for 12-32 rows (n-gram verify batches), the default {8,128,8,2} from 48 rows up. GGML_SYCL_PTQ1_T2_TILE now forces one tile for all sizes; GGML_SYCL_PTQ1_T2_TILE_SMALL_MAX (default 32) picks the crossover. GGML_SYCL_PTQ1_T2_GEMV_MAX (default 8) makes the GEMV/GemmMT switch a knob; the sweep confirmed 8. Outputs and KLD identical; MTP rename/edit +0.5%, 48K copy +1.3%. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit a70fe64 on Torchit1/llama.cpp arc-b580) * sycl: decode attention prefetches the next phase's key rows (GGML_SYCL_FA_DEC_PF) Phase probes (GGML_SYCL_FA_DEC_DPAS_PROBE) showed Q.K and P.V serialised behind the K / V loads. Each tile now prefetches its V rows before Q.K and the next tile's K rows before P.V (sycl_ext_oneapi_prefetch, L1). Per layer at 1 token: 48K 263 -> 226 us, 115K 554 -> 496 us. Whole model: plain 48K +2.7%, 115K +4.4%; MTP 48K +4.2%, 115K +7.5%; outputs identical. FA Bonsai cases 5/5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit dc575f4 on Torchit1/llama.cpp arc-b580) * tests: arctune fusion tests for the SYCL fusions; fusion guard fixes test-backend-ops ARCLAB_GLU_ROT / ADD_NORM_ROT / CONV_CHAIN / DN_GATES / T2_MULTI build the exact node pattern each fusion matches, at Bonsai 27B shapes and 1-16 tokens, 3 seeds each, and compare every fused output with the CPU backend: 75/75 pass, each fusion confirmed firing (GGML_SYCL_FUSION_LOG, GGML_SYCL_T2_MULTI_LOG). Found while writing them: the far norm->rotation fusion now requires the signs to be a weight (they are read early); the conv chain no longer refuses a later reader of the cache slot; the same-input merge repacks weights on first use, so it also merges on the first graph. Whole model unchanged (plain short / 48K / 115K md5s identical). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit bfb2d76 on Torchit1/llama.cpp arc-b580) Conflict resolved against prism: test-backend-ops keeps prism's fused SwiGLU/sign-flip FWHT cases and adds these. * ptq1-t2: GGML_SYCL_PTQ1_T2_TUNED per-shape GEMV tile table (plumbing, empty table) ARC-LAB: GGML_SYCL_PTQ1_T2_TUNED=1 (default 0) looks up a (N, K, rows, merged) row in t2_tuned_table() for every 1..8-row T2 GEMV (t2_dispatch and ggml_sycl_t2_mul_mat_multi) and launches TernSYCL's Gemv<SGM, NSG, LS, U> (TernSYCL, BSD 3-Clause) under a GRF mode and work-group order through GemvTuned (= gsw::Sweep of /lab/exp/gemv-sweep). Tiles are compiled from the T2_TUNED_KERNELS X-macro; t2_precompile warms them when the knob is on. Table and kernel list are empty until the B580 sweep results are in, so =1 changes nothing. =2 is a plumbing self-test: the default tiles through the tuned launcher (outputs must stay identical). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit a0d82ad on Torchit1/llama.cpp arc-b580) * ptq1-t2: GGML_SYCL_PTQ1_T2_TUNED table from the B580 GEMV sweep (bit-exact tiles) ARC-LAB: five rows from /lab/exp/gemv-sweep (JIT build, medians of 5 interleaved rounds), each with the same LS as the default tile, so the outputs are bit-identical: ffn_down n=5 0.775, gate+up n=5 0.896, output head n=5 0.882, q+k+v n=5 0.923, q+k+v n=1 0.974 (time vs the default tile). Tiles are TernSYCL's Gemv (BSD 3-Clause) under a different launch geometry and work-group order. Batch sizes 2-4 and other shapes keep the default tiles. Knob still defaults to 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 4bd894b on Torchit1/llama.cpp arc-b580) * ptq1-t2: tuned GEMV tiles on by default on a B580 (GGML_SYCL_PTQ1_T2_TUNED=0 turns them off) B580 A/B (sd2 head): MTP-only +2.9/+3.1%; live MTP + n-gram short +4.4%, 48K +3.1%, 115K flat; outputs identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 0729c9d on Torchit1/llama.cpp arc-b580) * sycl: default to the legacy Level Zero adapter unless SYCL_UR_USE_LEVEL_ZERO_V2 is set Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 5336e12 on Torchit1/llama.cpp arc-b580) * sycl: tuned Q6_K mat-vec and fused Q6_K gate+up+SWIGLU for the MTP draft head (knobs, off) The draft head's Q6_K mat-vecs are ~60% of a draft step and read at ~70% of the B580's bandwidth (op timing, 2026-10-10). - GGML_SYCL_Q6K_GEMV=1: a single-column reordered Q6_K mat-vec with the original per-row arithmetic (same lane -> quant-int mapping, blocks in order, same vec_dot expression accumulated in the same order, same sub-group reduction) but U blocks' operands loaded before use, aligned 32-bit ql/qh loads, and GGML_SYCL_Q6K_SG sub-groups per work-group. Sweep knobs GGML_SYCL_Q6K_U (1/2/4, default 2) and GGML_SYCL_Q6K_SG (4/8/16, default 8). GGML_SYCL_Q6K_CHECK=1 reruns the original kernel per launch and counts rows that differ in any bit (stderr), the in-server identity check. - GGML_SYCL_Q6K_GLU=1: Q6_K joins the reorder gate+up+GLU fusion (1..8 columns), whose dot products are the reorder mat-vec's and whose epilogue is the SWIGLU kernel's silu(gate) * up. - test-backend-ops: Q6_K mat-vec and gate+up fusion cases at the head's shapes (eval: 1, 2, 6 tokens; perf: 1..5 tokens). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 6dd4481 on Torchit1/llama.cpp arc-b580) * sycl fattn-dec: transposed 2D block loads for the DPAS kernel's lane = key reads (GGML_SYCL_FA_DEC_B2D, default off) xe2dp::ld2d_t32_8x16 (lsc_load_block2d d32.8x16tn): one message gives each lane 8 dwords of its own key row, replacing the per-lane dword gathers in Q.K (5 messages per 16 keys instead of 40) and P.V (1 instead of 5). Same bytes and math, 89/89 FLASH_ATTN_EXT cases. Engages only on 64-byte aligned K/V surfaces (logged once). Slower on its own at 128K (register spills): 5 tokens 1069 -> 1220 us, 16 tokens 2945 -> 3610 us. Kernel launch now goes through a named lambda. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit fb75ee9 on Torchit1/llama.cpp arc-b580) * sycl fattn-dec: 256-register file for the DPAS decode kernel at 3+ query tokens (GGML_SYCL_FA_DEC_GRF, default 256) The 3+ token launch is one wave of 2 work-groups x 16 subgroups per Xe core = 32 threads, which is what the 256-register mode allows, so it costs no occupancy and removes the spills. 1-2 tokens keep the default register file (their wave puts 4 work-groups on a core). GGML_SYCL_FA_DEC_GRF=0 = off. B580, us per layer at 128K (real cache layout) 5 / 8 / 12 / 16 tokens: 1073 / 1434 / 2920 / 2939 -> 1004 / 1240 / 2465 / 2445; 1 token unchanged. Whole model (arctune x3, MTP + n-gram, sd3 head): 48K 106.1 -> 108.2 t/s (+2.0%), 115K 86.8 -> 90.6 (+4.4%), output md5 identical (d09ea3 / 291b0b). FLASH_ATTN_EXT q4_0: 480/480. Perf cases: kv 16K/64K/128K x 1/5/8/12/16 tokens. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 1eb2307 on Torchit1/llama.cpp arc-b580) Also carries the FLASH_ATTN_EXT perf cases this commit widens, first added by 1fa26f1 (Jesse Symons), which is not picked separately. * sycl: one in-place layout owner for PQ2_0/PTQ1_0 weights The port's MMVQ reorder (opt_for_reorder, also called by the fused gate/up path) and the PQ2_0 XMX rewrite (ggml-org#294) both change these weights in place, and neither checked the other's flag. With GGML_SYCL_PTQ1_T2 unset a weight could get both layouts, so 27B PTQ1_0 perplexity went to 139615 on the B390 while every op test passed. On 16-wide DPAS devices the XMX layout now owns PQ2_0/PTQ1_0 and the reorder skips them; the fused paths that need the reorder decline and fall back to the unfused XMX path. The XMX path also refuses a weight that is already reordered or T2-repacked. * sycl: drop PQ2_0/PTQ1_0 duplicates left by the arc-b580 merge Keep prism's dequantize, vec_dot and MMVQ dispatch; keep the port's reorder-aware convert/MMVQ cases. Prism (ggml-org#235/ggml-org#294) and the port both added PQ2_0/PTQ1_0 support, so the merged tree had duplicate dequant functions, switch cases and two MMVQ implementations with the same names and did not compile. Deletions only, found and built on the B390 (oneAPI 2025.3.2). * sycl: Xe3 defaults to the MMVQ reorder for PQ2_0/PTQ1_0 and to XMX decode attention Measured on an Arc B390 (Panther Lake) with GGML_SYCL_PTQ1_T2 unset: - PQ2_0/PTQ1_0 weights go to the MMVQ reorder layout instead of the XMX layout: decode +16%, prompt +6%, about 0.9 GB less memory; KLD vs prism 6.7e-4 (XMX 2.4e-4). GGML_SYCL_DISABLE_XMX=0 restores the XMX layout, =1 still forces the reorder on any device. - GGML_SYCL_FA_DEC_DPAS defaults to 1: decode at 16K context +4-6% with a q4_0 KV cache; FLASH_ATTN_EXT head size 256 264/264, KLD with and without it within run-to-run error. =0 turns it off. Other devices keep their previous defaults. --------- Co-authored-by: Jesse Symons <155509533+Torchit1@users.noreply.github.com>

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection. Fold (r172, PR ggml-org#247, overdoingism): the derived kq mask covers batches of several sequences, so a `--parallel > 1 --kv-unified` context stops reserving the packed mask (n_kv x n_ubatch x 2 bytes on each device and again pinned on the host). Sequence k of the ubatch has its cells and its tokens' [lo, hi] shifted by k * 2^23 (KQ_DERIVED_SEQ_STRIDE), so the existing `tok_lo <= cell_pos <= tok_hi` predicate keeps every token inside its own sequence's range and the FA kernels need no change. The form is taken only when one KV stream holds the batch, every token is in exactly one sequence, no cell carries two of the batch's sequences (`llama_kv_cells` now counts those, so the check is O(1) unless `llama_memory_seq_cp` was used) and every position is below the stride; a multi-sequence batch of at most 32 tokens (the multi-slot verify steps) and every other refused shape keeps the packed mask. `kq_mask_packed_reachable()` drops its `n_seq_max > 1` term, and `decode()` latches `kq_mask_multiseq_seen` on the first pre-split batch the form cannot serve and re-reserves once with the packed mask (the r160 2-D latch pattern, so no later batch grows the compute buffer - issue ggml-org#42). Measured (gfx1201, dense 27B UD-Q4_K_XL, `-c 245760 -ub 2048 --parallel 2 --kv-unified -ctk/-ctv q8_0`): device memory 27.51 -> 26.41 GB and server RSS 1.97 -> 0.98 GiB after load, two concurrent 16K prefills unchanged (26.96 vs 27.23 s at 1 GPU; 17.07 vs 17.09 s at 2 GPU `-sm tensor`), single-sequence greedy text byte-identical (dense / 35B-A3B MoE / hybrid 4B) and `plain == draft-mtp` identical. Purity: the derived predicate *is* the packed one - the author's host oracle compares the two fills cell by cell, re-run here with 0 mismatches on llama-perplexity at 4 sequences per ubatch (dense 27B at 1 and 2 GPU tensor, Qwen3.5-4B hybrid, Gemma4-12B and gpt-oss-20b iSWA) and on the 2-slot server shape; the prefill-logit gate at `--ub 2048 --ab "LLAMA_KQ_MASK_DERIVED_MULTISEQ=0"` reads 0.000000 / 100.000 % at both f16 and q8_0 KV, and `gate-park-idle-slots.sh` passes at 3 GPU tensor. No effect on a graph that also holds an `allow_derived = false` builder (qwen4exp/QSA, MLA): its packed mask is created and filled anyway - measured identical VRAM and RSS on Flash-Next. `LLAMA_KQ_MASK_DERIVED_MULTISEQ=0` restores the single-sequence rule and the packed reserve.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection. Fold (r172, PR ggml-org#247, overdoingism): the derived kq mask covers batches of several sequences, so a `--parallel > 1 --kv-unified` context stops reserving the packed mask (n_kv x n_ubatch x 2 bytes on each device and again pinned on the host). Sequence k of the ubatch has its cells and its tokens' [lo, hi] shifted by k * 2^23 (KQ_DERIVED_SEQ_STRIDE), so the existing `tok_lo <= cell_pos <= tok_hi` predicate keeps every token inside its own sequence's range and the FA kernels need no change. The form is taken only when one KV stream holds the batch, every token is in exactly one sequence, no cell carries two of the batch's sequences (`llama_kv_cells` now counts those, so the check is O(1) unless `llama_memory_seq_cp` was used) and every position is below the stride; a multi-sequence batch of at most 32 tokens (the multi-slot verify steps) and every other refused shape keeps the packed mask. `kq_mask_packed_reachable()` drops its `n_seq_max > 1` term, and `decode()` latches `kq_mask_multiseq_seen` on the first pre-split batch the form cannot serve and re-reserves once with the packed mask (the r160 2-D latch pattern, so no later batch grows the compute buffer - issue ggml-org#42). Measured (gfx1201, dense 27B UD-Q4_K_XL, `-c 245760 -ub 2048 --parallel 2 --kv-unified -ctk/-ctv q8_0`): device memory 27.51 -> 26.41 GB and server RSS 1.97 -> 0.98 GiB after load, two concurrent 16K prefills unchanged (26.96 vs 27.23 s at 1 GPU; 17.07 vs 17.09 s at 2 GPU `-sm tensor`), single-sequence greedy text byte-identical (dense / 35B-A3B MoE / hybrid 4B) and `plain == draft-mtp` identical. Purity: the derived predicate *is* the packed one - the author's host oracle compares the two fills cell by cell, re-run here with 0 mismatches on llama-perplexity at 4 sequences per ubatch (dense 27B at 1 and 2 GPU tensor, Qwen3.5-4B hybrid, Gemma4-12B and gpt-oss-20b iSWA) and on the 2-slot server shape; the prefill-logit gate at `--ub 2048 --ab "LLAMA_KQ_MASK_DERIVED_MULTISEQ=0"` reads 0.000000 / 100.000 % at both f16 and q8_0 KV, and `gate-park-idle-slots.sh` passes at 3 GPU tensor. No effect on a graph that also holds an `allow_derived = false` builder (qwen4exp/QSA, MLA): its packed mask is created and filled anyway - measured identical VRAM and RSS on Flash-Next. `LLAMA_KQ_MASK_DERIVED_MULTISEQ=0` restores the single-sequence rule and the packed reserve.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection. Fold (r172, PR ggml-org#247, overdoingism): the derived kq mask covers batches of several sequences, so a `--parallel > 1 --kv-unified` context stops reserving the packed mask (n_kv x n_ubatch x 2 bytes on each device and again pinned on the host). Sequence k of the ubatch has its cells and its tokens' [lo, hi] shifted by k * 2^23 (KQ_DERIVED_SEQ_STRIDE), so the existing `tok_lo <= cell_pos <= tok_hi` predicate keeps every token inside its own sequence's range and the FA kernels need no change. The form is taken only when one KV stream holds the batch, every token is in exactly one sequence, no cell carries two of the batch's sequences (`llama_kv_cells` now counts those, so the check is O(1) unless `llama_memory_seq_cp` was used) and every position is below the stride; a multi-sequence batch of at most 32 tokens (the multi-slot verify steps) and every other refused shape keeps the packed mask. `kq_mask_packed_reachable()` drops its `n_seq_max > 1` term, and `decode()` latches `kq_mask_multiseq_seen` on the first pre-split batch the form cannot serve and re-reserves once with the packed mask (the r160 2-D latch pattern, so no later batch grows the compute buffer - issue ggml-org#42). Measured (gfx1201, dense 27B UD-Q4_K_XL, `-c 245760 -ub 2048 --parallel 2 --kv-unified -ctk/-ctv q8_0`): device memory 27.51 -> 26.41 GB and server RSS 1.97 -> 0.98 GiB after load, two concurrent 16K prefills unchanged (26.96 vs 27.23 s at 1 GPU; 17.07 vs 17.09 s at 2 GPU `-sm tensor`), single-sequence greedy text byte-identical (dense / 35B-A3B MoE / hybrid 4B) and `plain == draft-mtp` identical. Purity: the derived predicate *is* the packed one - the author's host oracle compares the two fills cell by cell, re-run here with 0 mismatches on llama-perplexity at 4 sequences per ubatch (dense 27B at 1 and 2 GPU tensor, Qwen3.5-4B hybrid, Gemma4-12B and gpt-oss-20b iSWA) and on the 2-slot server shape; the prefill-logit gate at `--ub 2048 --ab "LLAMA_KQ_MASK_DERIVED_MULTISEQ=0"` reads 0.000000 / 100.000 % at both f16 and q8_0 KV, and `gate-park-idle-slots.sh` passes at 3 GPU tensor. No effect on a graph that also holds an `allow_derived = false` builder (qwen4exp/QSA, MLA): its packed mask is created and filled anyway - measured identical VRAM and RSS on Flash-Next. `LLAMA_KQ_MASK_DERIVED_MULTISEQ=0` restores the single-sequence rule and the packed reserve.

… Ada: ggml-org#218 + ggml-org#215 hybrid PTQ1_0 dispatch, in-place q4_0/q8_0 K/V flash attention (integration checkpoint) (ggml-org#221) * Add: planar-transposed activation layout for the PTQ1_0 mat-vec path The mmvq path gave every thread one 128-weight PTQ1_0 block and read the activations as 32 scattered 4-byte loads per column out of 36-byte block_q8_1 structs, so every extra column cost a full pass: 47 / 104 / 154 / 201 us for 1 / 2 / 3 / 4 columns on a 4096 x 14336 matrix (RTX 3060, test-backend-ops perf). Quantize PTQ1_0 activations into a planar-transposed layout instead: per column, 8 planes of 16-byte quant pieces and one plane of the 4 (d, s) half2 scales, 16 bytes per 128-element block each, with the same bytes per row and the same column stride as block_q8_1. A thread then reads its block's activations as 9 aligned 16-byte loads with adjacent lanes on adjacent pieces: 46 / 55 / 65 / 75 us. The trit decode is unchanged; the byte-wise minus one is (q + 0x7F7F7F7F) ^ 0x80808080 instead of __vsub4. nwarps is pinned to 4 for every column count so the K partition, and with it the fp32 summation order, does not depend on the batch. mmvq now handles PTQ1_0 up to 8 columns (MMQ took 395 us at 8). HIP keeps the old layout. * Add: dedicated PTQ1_0 mat-vec kernel with full lane utilization With 128-element blocks the generic kernel's 128 threads cover 128 K blocks per iteration, so a K = 5120 projection (40 blocks) keeps 31% of the lanes busy and K = 17408 (136 blocks) 53%. This kernel handles plain 2D MUL_MAT (no batch dims, no expert ids, K a multiple of 128, up to 8 columns): it flattens (row group, K block) pairs into one index space with rows_per_cta chosen on the host to fill whole 128-thread iterations, gives each thread 4 rows per K block (2 at 5 to 8 columns) for latency hiding and activation reuse, stores one fp32 partial per (row, column, K block) in dynamic shared memory and has one warp per (row, column) sum them in a fixed order (lane-strided sequential, then a butterfly). The per-block partial is d * sum_k d8_k * sumi_k with exact integer sumi_k, written as __fmaf_rn and __fmul_rn so every instantiation rounds alike, and the order depends only on the weight shape: a column's result is bit-identical for every column count 1 to 8. Fusion (bias, gated GLU epilogue) is supported for one column as in the generic kernel, which remains the path for batched and MoE calls. Bonsai 2 shapes, RTX 3060 12GB, us per call, 1 column: attn_qkv 5120 x 10240 72.4 to 41.8, ffn_up 5120 x 17408 120.0 to 66.7, attn_gate 5120 x 6144 44.8 to 27.7. Whole model llama-bench tg32 26.1 to 39.8 tok/s, pp2 34.1 to 61.1, pp3 33.3 to 72.0. * Test: add Bonsai 2 projection shapes to the mul_mat perf cases * Add: GGML_CUDA_BATCH_INVARIANT for batch-invariant small-batch kernels Off by default, nothing changes. When set, kernels are picked so that the per-column arithmetic of a token does not depend on how many columns (1 to 8) share the launch: F16 and BF16 matrices with at most 512 rows take the mul_mat_vec_f kernel for 1 to 8 columns on Ampere and newer (its K partition depends on K only), flash attention with up to 8 queries takes the vector kernel on Turing and newer instead of switching to the tensor-core kernel at 2 queries, and the KV split (parallel_blocks) is sized as for one query tile so the partial-softmax combine order does not depend on the batch. Together with the batch-invariant PTQ1_0 mat-vec this makes the logits of a token bit-identical whether it is decoded alone or verified inside a speculative batch of 2, 3 or 4 tokens (measured on Ternary Bonsai 2 27B: greedy output with draft-mtp equals greedy output without it on three prompts at n-max 1, 2 and 3). Batches of 5 to 8 agree with each other but not with 1 to 4. Costs throughput at depth (the vector attention kernel at 3 queries over 42K keys). * Fix: use the mat-vec kernel for bf16 matrices under 64 rows at 2 to 8 columns The qwen35 gated-delta-net gate projections are bf16 [5120 x 48]. At 2 to 8 columns on Ampere they took a 10.5 us path where mul_mat_vec_f takes 4.4 to 7.3 us (test-backend-ops perf, RTX 3060); at one column both already used mul_mat_vec_f (3.3 us). 96 calls per token on the 27B model. * cuda: fold the recurrent-state gather into the GATED_DELTA_NET kernel (+3.8% TG on Ada) build_rs gathers each layer's live recurrent state via GET_ROWS(cache, s_copy) into a temp (3 MB per layer for Bonsai 2 27B) that only the GDN kernel reads. Traced in-graph with CUPTI on an RTX 4070 the gather kernel costs ~8 us/layer, and its dirty output sits in L2 until the FFN weight stream evicts it, which made the gate/up GEMVs in GDN layers ~15% slower than the identical GEMVs in attention layers. When the GET_ROWS output has no consumer other than (optionally through a RESHAPE) the GDN node's state input, the graph evaluator skips the GET_ROWS node and records the gather (cache base, ids, row stride) for that GDN node. The kernel then indexes the cache row s_ids[seq] directly; no temp, no kernel, no writeback. Single-sequence only: with several sequences a gathered row may alias a row another sequence writes in the same op. GGML_CUDA_GDN_GATHER_FUSION=0 disables it. The registrations live in the evaluating ggml_backend_cuda_context (ggml_cuda_gdn_gather_context, beside the concurrent-stream state), keyed by node pointer and reset at the start of every graph evaluation, so two contexts or host threads decoding at once never see each other's skips. This replaces the process-wide static map in the first revision of ggml-org#215. Independent of the activation layout change; split out of ggml-org#215 so that layout can be compared with ggml-org#218 on its own. RTX 4070 12 GB, Ternary-Bonsai-2-27B-PTQ1_0, llama-bench -fa 1 -ctk/-ctv q4_0, stock clocks, three alternating rounds of r=3: tg128 56.4 -> 58.5 t/s on top of ggml-org#215 (+3.8%). pp512 unchanged. test-backend-ops CUDA0 vs CPU: GATED_DELTA_NET 39/39 supported cases, GET_ROWS unchanged. 400 greedy tokens identical with GGML_CUDA_GDN_GATHER_FUSION=0 and 1. * cuda: PTQ1_0 decode on consumer Ampere/Ada: SoA q8 activations, warp-per-row small-K GEMV (+14% TG) Merged onto the planar-transposed activation layout and dedicated 2-8 column kernel (PrismML-Eng#218, sudoingX). One decision, ggml_cuda_q8_1_layout_for(type, ncols_dst, ids), is evaluated by both the quantizer call and the kernel switch: PTQ1_0, 1 column, no ids -> SOA_ISUM (this change): exact int16 sums, warp-per-row small-K geometry PTQ1_0, 2-8 columns / MoE -> PT (ggml-org#218): shared weight decode, full lane utilization anything else -> plain block_q8_1 GGML_CUDA_BATCH_INVARIANT=1 forces the one-column case onto PT as well so a token decoded alone and a token verified inside a speculative batch still see identical arithmetic. * cuda: PTQ1_0 hybrid dispatch: template y_soa, MMQ from 5 columns The SoA/planar choice is a kernel template parameter (only instantiated true for PTQ1_0 at one column) instead of a runtime branch inside the K loop; the runtime branch cost 5% TG on Ada. ggml_cuda_should_use_mmvq keeps PTQ1_0 on the mat-vec path up to 4 columns: with the branch-free MMQ tile loader the tile path overtakes the planar mat-vec kernel at n=5 (RTX 4070, Bonsai 2 27B: 163 vs 137 t/s at n=5, 258 vs 148 at n=8). * cuda: flash attention MMA reads q4_0/q8_0 K/V in place (no F16 scratch copy) The tensor-core flash attention path converted the whole quantized K/V cache to F16 in a scratch buffer on every micro-batch: 1 GB per context at 262k with q4_0 caches, 2 GB with a draft context, and a full re-conversion per ubatch during prefill. With Bonsai 2 27B + MTP that pushed the 4070 12 GB over the edge and WDDM spilled to system RAM (41 tok/s). flash_attn_ext_f16 now takes type_KV as a template parameter. For q4_0 and q8_0 the tile loader dequantizes blocks straight from the cache into the shared-memory half2 tiles (aligned 32-bit word reads, scale from the block header), the cp.async multi-stage pipeline is disabled for those types (cp.async cannot dequantize), and K/V strides are byte strides. The allocator no longer reserves the F16 copy when the native kernel handles the call. Dispatch is limited to D=128/256 heads with matching K/V types and word-aligned strides; anything else keeps the old conversion path. test-backend-ops: 48 new FLASH_ATTN_EXT cases for q4_0/q8_0 at D=128/256, GQA, batch 1-8, KV lengths incl. non-multiple of the tile, permuted layouts. All pass against the CPU reference; the full suite is 2984/2984. Bonsai 2 27B PTQ1_0 + MTP, RTX 4070 12 GB, 262144 context, ub 128: resident, 222 MiB shared (no spill), 86 tok/s mean over 3 prompts. * cuda: PTQ1_0 multi-column mat-vec: raw digits with exact activation sums, per-pair epilogue (+13% 3-col, +24% 4-col) Two changes to mul_mat_vec_ptq1_0_pt, found with Nsight Compute on an RTX 4070 (sm_89). 1. The trit decode returns the raw base-3 digits {0,1,2} and the dot product uses a mixed-sign dp4a (u8 x s8) against the activations, subtracting the exact int16 sum of the 32 quantized activations once per sub-block: sum((q-1)*a) = sum(q*a) - sum(a). The PT quantizer now stores that sum bit-cast in the ds.y half slot (the SoA layout already did). Two ALU ops fewer per four weights; the result is the same integer, so output is bit-identical. 2. The epilogue summed each (row, column) pair with one warp: a runtime integer division per pair, a lane-strided loop and a 5-step shuffle butterfly. Under the profiler that was 23% of issued instructions and ~50% of warp stall samples (FADD on SHFL scoreboard, MUFU.RCP/I2F/F2I from the division) while the CTA held its slot and moved no bytes. Now one thread per pair sums the K blocks sequentially from shared memory with four interleaved accumulators, over a padded (odd) row stride so the reads are bank-conflict-free. The order still depends only on the weight shape, so the batch-invariant guarantee holds. Profile of the 3-column gate|up GEMV (K=5120, N=17408) before -> after, ncu locked clocks: 2135 -> 1493 instructions per warp, 56.3 -> 43.5 us, DRAM 62% -> 83% of peak. llama-bench, +1500 MHz memory: pp3 150 -> 169 tok/s, pp4 172 -> 212. pp1 unchanged (SoA path). test-backend-ops -b CUDA0 -o MUL_MAT: 1283/1283. * qwen35: MTP graph publishes only the output rows of h_nextn when embeddings_nextn_masked llama_context::decode copies the first n_outputs rows of t_h_nextn when embeddings_nextn_masked is set, but the QWEN35 MTP builder assigned t_h_nextn before the inp_out_ids gather, so the tensor had n_tokens rows. Any MTP batch with more tokens than outputs (e.g. catch-up rows decoded together with a draft row) returned row 0's hidden state for the draft row and the next draft step ran on the wrong input. Never triggered before because every MTP decode had n_tokens == n_outputs or no outputs. Gather before publishing in the masked case, exactly as the trunk graph does at its last layer; the unmasked case is unchanged. * speculative: draft-mtp decodes the catch-up rows together with the first draft row process() used to run the draft over every row of the target's verify batch immediately, purely to fill the draft's memory, and partly on rows the target then rejected. That is one graph launch and one host round trip per step spent on nothing the next draft needs yet. Single-head MTP with its own memory now keeps those rows (tokens, positions, shifted h) and decodes only the accepted prefix at the start of draft(), in the same llama_decode as the first draft row. Positions are contiguous and the draft row attends to the catch-up rows in-batch, so the draft is unchanged; accept() trims the stash to the accepted prefix, and rows are decoded on their own when a batch does not lead straight into a draft (prompt fed in tiny ubatches, drafting stopped, rollback replay overwrites them). Shared-memory and chained-head drafters keep the eager path, as does LLAMA_MTP_EAGER_CATCHUP=1 for A/B. RTX 4070, Bonsai-2-27B PTQ1_0 + MTP head, 262k context, q4_0 K/V, greedy, --backend-sampling: 23.0 -> 22.4 ms/step, three-prompt mean 101.0 -> 103.3 tok/s with identical draft acceptance (0.86 / 0.46 / 0.65) and byte-identical output vs the draft head off. Draft-side launches per step 4 -> 3. * cuda: Hadamard transform quantizes its own output when every consumer is a PTQ1_0 mat-vec A Hadamard-rotated activation ([MUL signs, RESHAPE,] MUL_MAT with GGML_HINT_SRC0_IS_HADAMARD) whose every use is src1 of a PTQ1_0 mat-vec on the mmvq path never needs to exist in F32: each consumer immediately quantizes it to q8_1. The transform kernel now quantizes as it goes and writes the q8_1 rows, in the layout the consumers expect (AoS / PT / SoA with exact integer sums and the same padding rule as ggml_cuda_mul_mat_vec_q), into its own output buffer; the mat-vecs find the rows through a per-context registry keyed by tensor identity and skip their quantize launch. On Bonsai 2 this removes ~390 launches per decode step. Safety rules the pass enforces: - every later node that reads the transform output (directly or via a whole-tensor reshape view) must be a PTQ1_0 MUL_MAT that ggml_cuda_mul_mat routes to mmvq, and the graph use count must match, so nothing ever reads the buffer as F32 - the registry is keyed by ggml_tensor pointer, never by data pointer: ggml-alloc recycles a dead output's block for later tensors in the same graph - when the output buffer aliases the input (ggml-alloc does this for the output-projection input on every Bonsai layer: x is a reshape view of a cont, there is no in-place reuse through a view parent, so the cont is released at the MUL and mm takes its block two nodes later) the rows go to a pool block held until the next graph evaluation. Writing into the aliased buffer raced the kernel's own input reads and produced sporadic garbage that, on a hybrid model, entered the recurrent state and poisoned the rest of the sequence (surfaced as INT_MAX from the backend argmax over an all-NaN logits row, about once per ~170 requests) - GGML_CUDA_FWHT_FUSION=0 disables the pass fwht.cuh: the block butterfly is a shared inline so fwht.cu and the fused quantizer stay one implementation. Validated: 340 consecutive requests (probe.py x 20 at 262144 ctx, q4_0 KV, draft-mtp n_max=2, backend sampling) with zero errors; output byte-identical to the unfused path. * sampling: reject out-of-vocab token ids from the backend sampler and the draft A backend-sampled id outside [0, n_vocab) (the CUDA argmax/top-k pad sentinel INT_MAX when a logits row is all NaN) used to travel as-is into the server, where the tokenizer's vector::at() threw "invalid vector subscript" and killed the request. Log the origin (idx, n_vocab, first candidate) and fall back to the CPU chain instead; the draft-mtp loop drops a draft whose top candidate id is out of vocab rather than feeding it to the verify decode. * speculative: draft-mtp drops stale deferred rows when a new task starts on the slot begin() used to decode whatever catch-up rows the previous task left pending. That is right only when the new prompt reuses them (same tokens at the same positions, continuing what ctx_dft holds). After a cancelled task or a different conversation landing on the slot, the server has already trimmed ctx_dft past those rows and decoding them writes stale positions into the recurrent draft state. Check first, flush when they belong, discard otherwise. * cuda: release the FWHT-q8 held pool blocks LIFO and before the pools are destroyed The registry keeps pool blocks alive across a graph when the fused Hadamard quantize would otherwise write over its own input. Its implicit destructor released them oldest first, and it is declared before the pools, so at context teardown the frees hit a destroyed VMM pool: GGML_ASSERT(ptr == pool_addr + pool_used) at the end of llama-bench. Release newest first in a destructor, and reset() from the context destructor ahead of the pools. * server: --spec-draft-depth-max stops drafting once the sequence is deep Deep in the context a decode step is bound by reading the KV cache. The draft passes and the multi-column verify add to that read without shortening it, so past some depth speculation costs more than the accepted tokens save: with Bonsai 2 27B on an RTX 4070 the MTP draft is +85% at zero depth, even near 24-32k tokens and -27% at 64k. Past the cutoff the slot decodes one token per step and the process() hook is skipped for batches that sit entirely beyond it (by position, so the early ubatches of a long prompt still reach the draft context and remain reusable as a prefix). Default 0 keeps the old behaviour. 0 / 32k / 64k tokens: 101 / 48 / 37 tok/s with a 24k cutoff against 101 / 45 / 27 without. * cuda: restrict off PTQ1_0 kernel params; Ampere uses the PT 1-col path. C2912 / stub mismatch on sm_90 and sm_120 when GGML_CUDA_RESTRICT is on the formal parameters of mul_mat_vec_ptq1_0_pt (same hunk as 2578fdf on ggml-org#218). Ampere (cc 800-889) takes the planar 1-col kernel; Ada and newer keep SoA. 3060: ggml-org#218 was +5.9% tg128 vs ggml-org#215 at one column. * cuda: address Copilot review on native q4/q8 FA, MTP catch-up, and BOM. Disable drafting when deferred catch-up decode fails. Gate in-place quantized K/V FA to NVIDIA and require ggml_cuda_is_aligned on K and V. Strip UTF-8 BOMs. The FWHT fusion comment no longer opens with the GDN gather notes. * cuda: share the PTQ1_0 mat-vec smem budget between guard and launch The combo kernel pads each partials row by one float. Use the same byte count in the entry guard as the launch, including that pad, so oversized gated shapes fall back instead of requesting more than smpb. * spec: flush MTP catch-up when it plus anchors would overflow n_batch process() can stash n_batch rows; draft() then added one more anchor. Flush first when the combined count would exceed llama_n_batch. * cuda: restore the scalar PTQ1_0 vec-dot on MUSA Same hole as ggml-org#215: the generic entry returned zero once MUSA was kept off the SoA path. MUSA now takes the HIP scalar implementation. * spec: keep MTP catch-up across a failed draft decode catchup_failed was set on a flush failure and never cleared, so one bad decode disabled that slot for every later task. begin() now resets it with the rest of the per-task state. draft() also cleared pending while building the batch, so a failed llama_decode dropped those rows from both the stash and ctx_dft. pending stays set until that first decode succeeds, and a failure marks the slot so it is not drafted again until the next begin(). flush_deferred waits for success the same way. The layout_for comment now says it is the column-count helper. ggml_cuda_q8_1_layout_host is the decision both sides call. * cuda: define the PTQ1_0 layout helper after ggml_cuda_info The host layout decision reads the current device's compute capability. It was declared above ggml_cuda_info and ggml_cuda_get_device, so a clean sm_89 build failed in every CUDA translation unit. The helper now sits below those declarations. * cuda: land remaining ggml-org#221 review notes: pass q8 layout down, skip HIP/MUSA q4/q8 FA instances, exercise ALiBi/softcap. * cuda: wait on the PDL dependency in the PTQ1_0 planar mat-vec (Hopper/Blackwell). * cuda: restore warp-reduce epilogue under BATCH_INVARIANT so MTP-on matches MTP-off. The four-accumulator path (5300cd1) is faster but a 5080 run flipped a late near-tie. GDN now waits on PDL before reading s_ids. --------- Co-authored-by: sudoingX <200180104+sudoingX@users.noreply.github.com> Co-authored-by: Cary Palmer <24235924+professorpalmer@users.noreply.github.com>

Add --spec-draft-depth-max and stop feeding draft context rows beyond its cutoff. Keep early prompt rows reusable and fall back to ordinary target decoding after the cutoff. Reject out-of-vocabulary MTP draft candidates. Based on Cary Palmer and sudoingX's speculative controls and candidate guard from Prism ggml-org#221. Source: PrismML-Eng@5244cea Co-authored-by: sudoingX <200180104+sudoingX@users.noreply.github.com> Co-authored-by: Cary Palmer <24235924+professorpalmer@users.noreply.github.com> Assisted-by: GitHub Copilot Assisted-by: OpenAI Codex Ported and rebased by Codex.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection. Fold (r172, PR ggml-org#247, overdoingism): the derived kq mask covers batches of several sequences, so a `--parallel > 1 --kv-unified` context stops reserving the packed mask (n_kv x n_ubatch x 2 bytes on each device and again pinned on the host). Sequence k of the ubatch has its cells and its tokens' [lo, hi] shifted by k * 2^23 (KQ_DERIVED_SEQ_STRIDE), so the existing `tok_lo <= cell_pos <= tok_hi` predicate keeps every token inside its own sequence's range and the FA kernels need no change. The form is taken only when one KV stream holds the batch, every token is in exactly one sequence, no cell carries two of the batch's sequences (`llama_kv_cells` now counts those, so the check is O(1) unless `llama_memory_seq_cp` was used) and every position is below the stride; a multi-sequence batch of at most 32 tokens (the multi-slot verify steps) and every other refused shape keeps the packed mask. `kq_mask_packed_reachable()` drops its `n_seq_max > 1` term, and `decode()` latches `kq_mask_multiseq_seen` on the first pre-split batch the form cannot serve and re-reserves once with the packed mask (the r160 2-D latch pattern, so no later batch grows the compute buffer - issue ggml-org#42). Measured (gfx1201, dense 27B UD-Q4_K_XL, `-c 245760 -ub 2048 --parallel 2 --kv-unified -ctk/-ctv q8_0`): device memory 27.51 -> 26.41 GB and server RSS 1.97 -> 0.98 GiB after load, two concurrent 16K prefills unchanged (26.96 vs 27.23 s at 1 GPU; 17.07 vs 17.09 s at 2 GPU `-sm tensor`), single-sequence greedy text byte-identical (dense / 35B-A3B MoE / hybrid 4B) and `plain == draft-mtp` identical. Purity: the derived predicate *is* the packed one - the author's host oracle compares the two fills cell by cell, re-run here with 0 mismatches on llama-perplexity at 4 sequences per ubatch (dense 27B at 1 and 2 GPU tensor, Qwen3.5-4B hybrid, Gemma4-12B and gpt-oss-20b iSWA) and on the 2-slot server shape; the prefill-logit gate at `--ub 2048 --ab "LLAMA_KQ_MASK_DERIVED_MULTISEQ=0"` reads 0.000000 / 100.000 % at both f16 and q8_0 KV, and `gate-park-idle-slots.sh` passes at 3 GPU tensor. No effect on a graph that also holds an `allow_derived = false` builder (qwen4exp/QSA, MLA): its packed mask is created and filled anyway - measured identical VRAM and RSS on Flash-Next. `LLAMA_KQ_MASK_DERIVED_MULTISEQ=0` restores the single-sequence rule and the packed reserve. Fold (r208): RDNA2 (gfx1030) FA prefill, reported and validated by @slavap (PR ggml-org#270/ggml-org#271; Radeon Pro V620 gfx1030, Qwen3.6-35B-A3B IQ4_XS, -b 8192 -ub 2048 -fa 1). This is the RDNA2 long-context deficit of issue ggml-org#214, and it is folded here rather than cut as new blocks. - q_global threshold 16 -> 64: the two kernel-side conditions, the Q_tmp_count stub and all seven launcher q_h2 arguments move together (a launcher that omits the argument is the r88 null-Q_h2 fault). At ncols 16..32 the global-half2 Q path re-reads Q per K-tile pass and the checked tail spilled ~180 VGPRs; with Q back in LDS it does not. - rdna_cap_ncols2 lifted for DKQ == 256 (D=128 keeps the cap). The cap dates from r76, when HIP's RDNA2 occupancy query returned 0 for the wide D=256 variants; launch_fattn computes RDNA2 occupancy itself now, so the cap is stale. Checked against the tree's own formula (4*waves_per_simd/waves_per_block, LDS cap 128K/lds) over the patched gfx1030 HSACO: every D=256 shape the launcher can now pick reports >= 2 blocks, so the GGML_ASSERT(max_blocks_per_sm > 0) abort the cap guarded against is unreachable. - the checked tail clamps out-of-range rows to the last in-bounds row instead of branching to a local zero array. The tail values are discarded by the k_VKQ_sup guards, every read stays inside the cache, and the pointer-select the branch existed to avoid (the kv=113 flat-scratch fault) is gone with it. i_sup - 1 may be -1 (a V chunk wholly past the KV end): row -1 is the last VALID row, and every weight in that chunk is zero, so the comment in the loader says so and warns against "fixing" it to max(i_sup - 1, 0), which would read one row past the cache. Verified here on ROCm 10.1.0 against this tree: both levers compile for gfx1030 (all 12 template-instances tile TUs, no warnings), the D=256 4x8/16x2 checked-tail scratch goes 480/500 B -> 0 B (the submitter's 488/520 on their toolchain), the largest static LDS in the whole tile family stays 63488 B of gfx1030's 65536 B, and for gfx1201 every tile kernel's machine code is bit-identical before and after (88/88 and 106/106 kernels hashed per kernel), so RDNA4, RDNA3_0, RDNA3_5, NVIDIA and MUSA/SYCL keep their codegen and numerics exactly. The timings stay the submitter's (pp2048 @ d32768 1041.0 -> 1259.8 t/s, tg256 @ d131072 50.9 -> 62.4 across their two arms); RDNA2 is best-effort per CONTAINERS.md and carries no gate in this repo. Not taken from the PRs: their own patches-rdna2-r205/ + release-rdna2-r205.json + scripts/apply-rdna2-r205.sh re-packaging (the same call as r88), and the block 16 two-lane prefill the branches were stacked on (PR ggml-org#268, a separate candidate).

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection. Fold (r172, PR ggml-org#247, overdoingism): the derived kq mask covers batches of several sequences, so a `--parallel > 1 --kv-unified` context stops reserving the packed mask (n_kv x n_ubatch x 2 bytes on each device and again pinned on the host). Sequence k of the ubatch has its cells and its tokens' [lo, hi] shifted by k * 2^23 (KQ_DERIVED_SEQ_STRIDE), so the existing `tok_lo <= cell_pos <= tok_hi` predicate keeps every token inside its own sequence's range and the FA kernels need no change. The form is taken only when one KV stream holds the batch, every token is in exactly one sequence, no cell carries two of the batch's sequences (`llama_kv_cells` now counts those, so the check is O(1) unless `llama_memory_seq_cp` was used) and every position is below the stride; a multi-sequence batch of at most 32 tokens (the multi-slot verify steps) and every other refused shape keeps the packed mask. `kq_mask_packed_reachable()` drops its `n_seq_max > 1` term, and `decode()` latches `kq_mask_multiseq_seen` on the first pre-split batch the form cannot serve and re-reserves once with the packed mask (the r160 2-D latch pattern, so no later batch grows the compute buffer - issue ggml-org#42). Measured (gfx1201, dense 27B UD-Q4_K_XL, `-c 245760 -ub 2048 --parallel 2 --kv-unified -ctk/-ctv q8_0`): device memory 27.51 -> 26.41 GB and server RSS 1.97 -> 0.98 GiB after load, two concurrent 16K prefills unchanged (26.96 vs 27.23 s at 1 GPU; 17.07 vs 17.09 s at 2 GPU `-sm tensor`), single-sequence greedy text byte-identical (dense / 35B-A3B MoE / hybrid 4B) and `plain == draft-mtp` identical. Purity: the derived predicate *is* the packed one - the author's host oracle compares the two fills cell by cell, re-run here with 0 mismatches on llama-perplexity at 4 sequences per ubatch (dense 27B at 1 and 2 GPU tensor, Qwen3.5-4B hybrid, Gemma4-12B and gpt-oss-20b iSWA) and on the 2-slot server shape; the prefill-logit gate at `--ub 2048 --ab "LLAMA_KQ_MASK_DERIVED_MULTISEQ=0"` reads 0.000000 / 100.000 % at both f16 and q8_0 KV, and `gate-park-idle-slots.sh` passes at 3 GPU tensor. No effect on a graph that also holds an `allow_derived = false` builder (qwen4exp/QSA, MLA): its packed mask is created and filled anyway - measured identical VRAM and RSS on Flash-Next. `LLAMA_KQ_MASK_DERIVED_MULTISEQ=0` restores the single-sequence rule and the packed reserve. Fold (r208): RDNA2 (gfx1030) FA prefill, reported and validated by @slavap (PR ggml-org#270/ggml-org#271; Radeon Pro V620 gfx1030, Qwen3.6-35B-A3B IQ4_XS, -b 8192 -ub 2048 -fa 1). This is the RDNA2 long-context deficit of issue ggml-org#214, and it is folded here rather than cut as new blocks. - q_global threshold 16 -> 64: the two kernel-side conditions, the Q_tmp_count stub and all seven launcher q_h2 arguments move together (a launcher that omits the argument is the r88 null-Q_h2 fault). At ncols 16..32 the global-half2 Q path re-reads Q per K-tile pass and the checked tail spilled ~180 VGPRs; with Q back in LDS it does not. - rdna_cap_ncols2 lifted for DKQ == 256 (D=128 keeps the cap). The cap dates from r76, when HIP's RDNA2 occupancy query returned 0 for the wide D=256 variants; launch_fattn computes RDNA2 occupancy itself now, so the cap is stale. Checked against the tree's own formula (4*waves_per_simd/waves_per_block, LDS cap 128K/lds) over the patched gfx1030 HSACO: every D=256 shape the launcher can now pick reports >= 2 blocks, so the GGML_ASSERT(max_blocks_per_sm > 0) abort the cap guarded against is unreachable. - the checked tail clamps out-of-range rows to the last in-bounds row instead of branching to a local zero array. The tail values are discarded by the k_VKQ_sup guards, every read stays inside the cache, and the pointer-select the branch existed to avoid (the kv=113 flat-scratch fault) is gone with it. i_sup - 1 may be -1 (a V chunk wholly past the KV end): row -1 is the last VALID row, and every weight in that chunk is zero, so the comment in the loader says so and warns against "fixing" it to max(i_sup - 1, 0), which would read one row past the cache. Verified here on ROCm 10.1.0 against this tree: both levers compile for gfx1030 (all 12 template-instances tile TUs, no warnings), the D=256 4x8/16x2 checked-tail scratch goes 480/500 B -> 0 B (the submitter's 488/520 on their toolchain), the largest static LDS in the whole tile family stays 63488 B of gfx1030's 65536 B, and for gfx1201 every tile kernel's machine code is bit-identical before and after (88/88 and 106/106 kernels hashed per kernel), so RDNA4, RDNA3_0, RDNA3_5, NVIDIA and MUSA/SYCL keep their codegen and numerics exactly. The timings stay the submitter's (pp2048 @ d32768 1041.0 -> 1259.8 t/s, tg256 @ d131072 50.9 -> 62.4 across their two arms); RDNA2 is best-effort per CONTAINERS.md and carries no gate in this repo. Not taken from the PRs: their own patches-rdna2-r205/ + release-rdna2-r205.json + scripts/apply-rdna2-r205.sh re-packaging (the same call as r88), and the block 16 two-lane prefill the branches were stacked on (PR ggml-org#268, a separate candidate).

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection. Fold (r172, PR ggml-org#247, overdoingism): the derived kq mask covers batches of several sequences, so a `--parallel > 1 --kv-unified` context stops reserving the packed mask (n_kv x n_ubatch x 2 bytes on each device and again pinned on the host). Sequence k of the ubatch has its cells and its tokens' [lo, hi] shifted by k * 2^23 (KQ_DERIVED_SEQ_STRIDE), so the existing `tok_lo <= cell_pos <= tok_hi` predicate keeps every token inside its own sequence's range and the FA kernels need no change. The form is taken only when one KV stream holds the batch, every token is in exactly one sequence, no cell carries two of the batch's sequences (`llama_kv_cells` now counts those, so the check is O(1) unless `llama_memory_seq_cp` was used) and every position is below the stride; a multi-sequence batch of at most 32 tokens (the multi-slot verify steps) and every other refused shape keeps the packed mask. `kq_mask_packed_reachable()` drops its `n_seq_max > 1` term, and `decode()` latches `kq_mask_multiseq_seen` on the first pre-split batch the form cannot serve and re-reserves once with the packed mask (the r160 2-D latch pattern, so no later batch grows the compute buffer - issue ggml-org#42). Measured (gfx1201, dense 27B UD-Q4_K_XL, `-c 245760 -ub 2048 --parallel 2 --kv-unified -ctk/-ctv q8_0`): device memory 27.51 -> 26.41 GB and server RSS 1.97 -> 0.98 GiB after load, two concurrent 16K prefills unchanged (26.96 vs 27.23 s at 1 GPU; 17.07 vs 17.09 s at 2 GPU `-sm tensor`), single-sequence greedy text byte-identical (dense / 35B-A3B MoE / hybrid 4B) and `plain == draft-mtp` identical. Purity: the derived predicate *is* the packed one - the author's host oracle compares the two fills cell by cell, re-run here with 0 mismatches on llama-perplexity at 4 sequences per ubatch (dense 27B at 1 and 2 GPU tensor, Qwen3.5-4B hybrid, Gemma4-12B and gpt-oss-20b iSWA) and on the 2-slot server shape; the prefill-logit gate at `--ub 2048 --ab "LLAMA_KQ_MASK_DERIVED_MULTISEQ=0"` reads 0.000000 / 100.000 % at both f16 and q8_0 KV, and `gate-park-idle-slots.sh` passes at 3 GPU tensor. No effect on a graph that also holds an `allow_derived = false` builder (qwen4exp/QSA, MLA): its packed mask is created and filled anyway - measured identical VRAM and RSS on Flash-Next. `LLAMA_KQ_MASK_DERIVED_MULTISEQ=0` restores the single-sequence rule and the packed reserve. Fold (r208): RDNA2 (gfx1030) FA prefill, reported and validated by @slavap (PR ggml-org#270/ggml-org#271; Radeon Pro V620 gfx1030, Qwen3.6-35B-A3B IQ4_XS, -b 8192 -ub 2048 -fa 1). This is the RDNA2 long-context deficit of issue ggml-org#214, and it is folded here rather than cut as new blocks. - q_global threshold 16 -> 64: the two kernel-side conditions, the Q_tmp_count stub and all seven launcher q_h2 arguments move together (a launcher that omits the argument is the r88 null-Q_h2 fault). At ncols 16..32 the global-half2 Q path re-reads Q per K-tile pass and the checked tail spilled ~180 VGPRs; with Q back in LDS it does not. - rdna_cap_ncols2 lifted for DKQ == 256 (D=128 keeps the cap). The cap dates from r76, when HIP's RDNA2 occupancy query returned 0 for the wide D=256 variants; launch_fattn computes RDNA2 occupancy itself now, so the cap is stale. Checked against the tree's own formula (4*waves_per_simd/waves_per_block, LDS cap 128K/lds) over the patched gfx1030 HSACO: every D=256 shape the launcher can now pick reports >= 2 blocks, so the GGML_ASSERT(max_blocks_per_sm > 0) abort the cap guarded against is unreachable. - the checked tail clamps out-of-range rows to the last in-bounds row instead of branching to a local zero array. The tail values are discarded by the k_VKQ_sup guards, every read stays inside the cache, and the pointer-select the branch existed to avoid (the kv=113 flat-scratch fault) is gone with it. i_sup - 1 may be -1 (a V chunk wholly past the KV end): row -1 is the last VALID row, and every weight in that chunk is zero, so the comment in the loader says so and warns against "fixing" it to max(i_sup - 1, 0), which would read one row past the cache. Verified here on ROCm 10.1.0 against this tree: both levers compile for gfx1030 (all 12 template-instances tile TUs, no warnings), the D=256 4x8/16x2 checked-tail scratch goes 480/500 B -> 0 B (the submitter's 488/520 on their toolchain), the largest static LDS in the whole tile family stays 63488 B of gfx1030's 65536 B, and for gfx1201 every tile kernel's machine code is bit-identical before and after (88/88 and 106/106 kernels hashed per kernel), so RDNA4, RDNA3_0, RDNA3_5, NVIDIA and MUSA/SYCL keep their codegen and numerics exactly. The timings stay the submitter's (pp2048 @ d32768 1041.0 -> 1259.8 t/s, tg256 @ d131072 50.9 -> 62.4 across their two arms); RDNA2 is best-effort per CONTAINERS.md and carries no gate in this repo. Not taken from the PRs: their own patches-rdna2-r205/ + release-rdna2-r205.json + scripts/apply-rdna2-r205.sh re-packaging (the same call as r88), and the block 16 two-lane prefill the branches were stacked on (PR ggml-org#268, a separate candidate).

bitRAKE added 2 commits March 16, 2023 20:23

fix coloring of last n_batch of prompt, and refactor line input

8470458

forgot the newline that needs to be sent to the model

07d2da0

gjmulder added the bug Something isn't working label Mar 17, 2023

bitRAKE added 2 commits March 19, 2023 11:21

resolve conflicts

9a1d2c7

(per ggml-org#283) try to force flush of color reset in SIGINT handler

77117c7

sw approved these changes Mar 19, 2023

View reviewed changes

sw merged commit 5c19c70 into ggml-org:master Mar 19, 2023

dmahurin pushed a commit to dmahurin/llama.cpp that referenced this pull request May 31, 2023

Fix llama_cpp and Llama type signatures. Closes ggml-org#221

71ef19c

dmahurin pushed a commit to dmahurin/llama.cpp that referenced this pull request May 31, 2023

Fix llama_cpp and Llama type signatures. Closes ggml-org#221

f2e4aff

dmahurin pushed a commit to dmahurin/llama.cpp that referenced this pull request Jun 1, 2023

Fix llama_cpp and Llama type signatures. Closes ggml-org#221

fda33dd

Deadsg pushed a commit to Deadsg/llama.cpp that referenced this pull request Dec 19, 2023

Fix llama_cpp and Llama type signatures. Closes ggml-org#221

01a010b

Bearsaerker mentioned this pull request Mar 12, 2025

Eval bug: Gemma 3 extremly slow prompt processing when using quantized kv cache. #12352

Closed

Navigation Menu

Repository navigation

fix coloring of last `n_batch` of prompt, and refactor line input - #221

fix coloring of last `n_batch` of prompt, and refactor line input#221
sw merged 4 commits into
ggml-org:masterfrom
bitRAKE:master

bitRAKE commented Mar 17, 2023

Uh oh!

sw commented Mar 19, 2023

Uh oh!

ggerganov commented Mar 19, 2023

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

bitRAKE commented Mar 17, 2023

Uh oh!

sw commented Mar 19, 2023

Uh oh!

ggerganov commented Mar 19, 2023

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants