Skip to content

ggml-hrx: no V transpose for MLA strided V (GLM-4.7-Flash prompts >= 512 tokens: PPL 315,664 -> 5.92) - #70

Merged
bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
1bit/hrx-fa-mla-transpose
Oct 2, 2026
Merged

bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
1bit/hrx-fa-mla-transpose

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

ggml-hrx: no V transpose for MLA's strided V (GLM-4.7-Flash prompts >= 512 tokens)

prepare_flash_attention_value transposes V with ggml_copy_transpose_f16
once a prompt batch has >= 256 queries and >= 512 KV rows. That kernel
reads kv_heads x value_head_size-wide contiguous rows, but MLA
(deepseek2/GLM) stores V with the QK head-size stride, the layout the
prefill matcher accepts since #95: 576 elements per row, of which V is the
first 512. The transpose then read misaligned rows and attention used the
wrong values. A V whose rows aren't contiguous now keeps the row-major
layout; that kernel takes the real row stride (value->nb[1]).

GLM-4.7-Flash Q4_K_M, llama-perplexity -c 512 -b 512, 4 chunks
(transpose path):
HRX0 before PPL 315,664 +/- 40,417
HRX0 after PPL 5.9160 +/- 0.479
CPU PPL 5.9592 +/- 0.484
Non-MLA models have contiguous V rows and keep the transpose:
Qwen3.8-27B-Q4_0-H32 PPL 5.8714 before and after (4 chunks).

🤖 Generated with Claude Code

…= 512 tokens)

prepare_flash_attention_value transposes V with ggml_copy_transpose_f16
once a prompt batch has >= 256 queries and >= 512 KV rows. That kernel
reads `kv_heads x value_head_size`-wide contiguous rows, but MLA
(deepseek2/GLM) stores V with the QK head-size stride, the layout the
prefill matcher accepts since #95: 576 elements per row, of which V is the
first 512. The transpose then read misaligned rows and attention used the
wrong values. A V whose rows aren't contiguous now keeps the row-major
layout; that kernel takes the real row stride (value->nb[1]).

GLM-4.7-Flash Q4_K_M, llama-perplexity -c 512 -b 512, 4 chunks
(transpose path):
  HRX0 before  PPL 315,664 +/- 40,417
  HRX0 after   PPL 5.9160 +/- 0.479
  CPU          PPL 5.9592 +/- 0.484
Non-MLA models have contiguous V rows and keep the transpose:
Qwen3.8-27B-Q4_0-H32 PPL 5.8714 before and after (4 chunks).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added the ggml label Oct 2, 2026
@bong-water-water-bong

Copy link
Copy Markdown
Author

Review (PR-Agent duty): approve. This is a minimal guard (+5/-1) in AMD's flash-attention dispatch: the V transpose path is taken only when V rows are contiguous (nb[1] == columns * sizeof(f16)). MLA's strided V (deepseek2/GLM, QK-head stride 576) keeps the row-major layout, whose kernel honours the real stride. Evidence: GLM-4.7-Flash Q4_K_M PPL 315,664 -> 5.916 (CPU 5.959) at prompts >= 512 tokens; the non-MLA 27B H32 is unchanged (5.8714). It's release-relevant: it goes in the next engine pin.

@bong-water-water-bong
bong-water-water-bong merged commit 6339066 into 1bit/hrx-vulkan-patched Oct 2, 2026
10 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant