Repository navigation
Pin gpu-kernels 4f586cac: split-K HIP decode attention for gated head-256 layers (Qwen3.5-4B 32K decode 23.7 -> 47.1 tok/s, Qwen3.8-27B 7.5 -> 11.0) - #500
Merged
Conversation
…-256 layers (Qwen3.5-4B 32K decode 23.7 -> 47.1 tok/s, Qwen3.8-27B 7.5 -> 11.0) gpu-kernels 4f586cac adds, since cf222e7d: - #24: hip.fa_decode_d256.gate, one dispatch for the gated full attention of Qwen3.5 / Qwen3.6 / Qwen3.8 (FLASH_ATTN_EXT -> RESHAPE -> MUL by SIGMOID(gate)) at 1..8 query tokens (decode and MTP verify), head 256, f16 KV, n_kv >= 1024. The Loom gated kernel ran it on one workgroup per query head (16-24 workgroups); profile at 26,644 tokens: 55% of a Qwen3.5-4B decode step and 34% of a Qwen3.8-27B one, about 41 GB/s of KV. The HIP kernel splits the cache over T x Hkv x S workgroups, reads each K/V row once per KV head and merges with agent-scope fences and runtime-zeroed counters (about 217 GB/s at n_kv 32768 in its bench). Measured on strixhalo gfx1151, performance mode, private build (llama.cpp af04c9f2 + hrx-system 82804197 + the add-on), llama-bench -fa 1 -p 0 -n 64 -d D, interleaved x3, medians, tok/s (ours): d=0 4096 16384 32768 Qwen3.5-4B before 61.13 50.93 34.17 23.71 Qwen3.5-4B after 61.20 59.19 53.31 47.07 Qwen3.8-27B before 12.34 11.36 9.27 7.49 Qwen3.8-27B after 12.32 12.11 11.62 10.97 (Lemonade's Vulkan llama.cpp b11393, same command and files: 4B 64.03 / 60.99 / 54.80 / 48.27, 27B 12.47 / 9.06 / 11.78 / 11.14; 27B d<32768 rows on the 08bb2241 base.) Qwen3.8-27B --mtp, llama-server, 26,644-token prompt, 64 greedy tokens: 16.40 -> 23.78 tok/s (medians of 4). Correctness: kernel bench vs a double-precision reference on 17 shapes (G 1..8, T 1..8, gate on/off) PASS, 5 repeats bit-identical; greedy text vs the Loom kernel identical on Qwen3.5-4B at 180 / 6,662 / 26,644 tokens (mean top-10 KL <= 1.7e-4), Qwen3.8-27B 26,644 tokens mean KL 3.7e-5 then a near-tie flip at token 10 (logprob gap 0.008); identical requests repeat exactly; no GPU fault in dmesg. Needs the hw-gate run (label) before merge: goldens with Qwen3.8-27B decode past 1024 KV cells may drift on near-ties. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR Reviewer Guide 🔍(Review updated until commit 6b898ff)Here are some key observations to aid the review process:
|
|
Persistent review updated to latest commit 5b979e4 |
|
Persistent review updated to latest commit 6b898ff |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
gpu-kernels 4f586cac adds, since cf222e7d:
(FLASH_ATTN_EXT -> RESHAPE -> MUL by SIGMOID(gate)) at 1..8 query tokens (decode and MTP verify), head
256, f16 KV, n_kv >= 1024. The Loom gated kernel ran it on one workgroup per query head (16-24
workgroups); profile at 26,644 tokens: 55% of a Qwen3.5-4B decode step and 34% of a Qwen3.8-27B one,
about 41 GB/s of KV. The HIP kernel splits the cache over T x Hkv x S workgroups, reads each K/V row
once per KV head and merges with agent-scope fences and runtime-zeroed counters (about 217 GB/s at
n_kv 32768 in its bench).
Measured on strixhalo gfx1151, performance mode, private build (llama.cpp af04c9f2 + hrx-system 82804197 +
the add-on), llama-bench -fa 1 -p 0 -n 64 -d D, interleaved x3, medians, tok/s (ours):
d=0 4096 16384 32768
Qwen3.5-4B before 61.13 50.93 34.17 23.71
Qwen3.5-4B after 61.20 59.19 53.31 47.07
Qwen3.8-27B before 12.34 11.36 9.27 7.49
Qwen3.8-27B after 12.32 12.11 11.62 10.97
(Lemonade's Vulkan llama.cpp b11393, same command and files: 4B 64.03 / 60.99 / 54.80 / 48.27,
27B 12.47 / 9.06 / 11.78 / 11.14; 27B d<32768 rows on the 08bb2241 base.)
Qwen3.8-27B --mtp, llama-server, 26,644-token prompt, 64 greedy tokens: 16.40 -> 23.78 tok/s (medians of 4).
Correctness: kernel bench vs a double-precision reference on 17 shapes (G 1..8, T 1..8, gate on/off) PASS,
5 repeats bit-identical; greedy text vs the Loom kernel identical on Qwen3.5-4B at 180 / 6,662 / 26,644
tokens (mean top-10 KL <= 1.7e-4), Qwen3.8-27B 26,644 tokens mean KL 3.7e-5 then a near-tie flip at token
10 (logprob gap 0.008); identical requests repeat exactly; no GPU fault in dmesg.
Needs the hw-gate run (label) before merge: goldens with Qwen3.8-27B decode past 1024 KV cells may drift on
near-ties.
🤖 Generated with Claude Code