Skip to content

Pin gpu-kernels 4f586cac: split-K HIP decode attention for gated head-256 layers (Qwen3.5-4B 32K decode 23.7 -> 47.1 tok/s, Qwen3.8-27B 7.5 -> 11.0) - #500

Merged
bong-water-water-bong merged 3 commits into
mainfrom
1bit/pin-gk-4f586cac
Oct 22, 2026
Merged

bong-water-water-bong merged 3 commits into
mainfrom
1bit/pin-gk-4f586cac

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

gpu-kernels 4f586cac adds, since cf222e7d:

  • Docs: the onebit recipe lives in our Lemonade fork #24: hip.fa_decode_d256.gate, one dispatch for the gated full attention of Qwen3.5 / Qwen3.6 / Qwen3.8
    (FLASH_ATTN_EXT -> RESHAPE -> MUL by SIGMOID(gate)) at 1..8 query tokens (decode and MTP verify), head
    256, f16 KV, n_kv >= 1024. The Loom gated kernel ran it on one workgroup per query head (16-24
    workgroups); profile at 26,644 tokens: 55% of a Qwen3.5-4B decode step and 34% of a Qwen3.8-27B one,
    about 41 GB/s of KV. The HIP kernel splits the cache over T x Hkv x S workgroups, reads each K/V row
    once per KV head and merges with agent-scope fences and runtime-zeroed counters (about 217 GB/s at
    n_kv 32768 in its bench).

Measured on strixhalo gfx1151, performance mode, private build (llama.cpp af04c9f2 + hrx-system 82804197 +
the add-on), llama-bench -fa 1 -p 0 -n 64 -d D, interleaved x3, medians, tok/s (ours):
d=0 4096 16384 32768
Qwen3.5-4B before 61.13 50.93 34.17 23.71
Qwen3.5-4B after 61.20 59.19 53.31 47.07
Qwen3.8-27B before 12.34 11.36 9.27 7.49
Qwen3.8-27B after 12.32 12.11 11.62 10.97
(Lemonade's Vulkan llama.cpp b11393, same command and files: 4B 64.03 / 60.99 / 54.80 / 48.27,
27B 12.47 / 9.06 / 11.78 / 11.14; 27B d<32768 rows on the 08bb2241 base.)
Qwen3.8-27B --mtp, llama-server, 26,644-token prompt, 64 greedy tokens: 16.40 -> 23.78 tok/s (medians of 4).
Correctness: kernel bench vs a double-precision reference on 17 shapes (G 1..8, T 1..8, gate on/off) PASS,
5 repeats bit-identical; greedy text vs the Loom kernel identical on Qwen3.5-4B at 180 / 6,662 / 26,644
tokens (mean top-10 KL <= 1.7e-4), Qwen3.8-27B 26,644 tokens mean KL 3.7e-5 then a near-tie flip at token
10 (logprob gap 0.008); identical requests repeat exactly; no GPU fault in dmesg.

Needs the hw-gate run (label) before merge: goldens with Qwen3.8-27B decode past 1024 KV cells may drift on
near-ties.

🤖 Generated with Claude Code

…-256 layers (Qwen3.5-4B 32K decode 23.7 -> 47.1 tok/s, Qwen3.8-27B 7.5 -> 11.0)

gpu-kernels 4f586cac adds, since cf222e7d:
- #24: hip.fa_decode_d256.gate, one dispatch for the gated full attention of Qwen3.5 / Qwen3.6 / Qwen3.8
  (FLASH_ATTN_EXT -> RESHAPE -> MUL by SIGMOID(gate)) at 1..8 query tokens (decode and MTP verify), head
  256, f16 KV, n_kv >= 1024. The Loom gated kernel ran it on one workgroup per query head (16-24
  workgroups); profile at 26,644 tokens: 55% of a Qwen3.5-4B decode step and 34% of a Qwen3.8-27B one,
  about 41 GB/s of KV. The HIP kernel splits the cache over T x Hkv x S workgroups, reads each K/V row
  once per KV head and merges with agent-scope fences and runtime-zeroed counters (about 217 GB/s at
  n_kv 32768 in its bench).

Measured on strixhalo gfx1151, performance mode, private build (llama.cpp af04c9f2 + hrx-system 82804197 +
the add-on), llama-bench -fa 1 -p 0 -n 64 -d D, interleaved x3, medians, tok/s (ours):
                    d=0     4096    16384   32768
  Qwen3.5-4B before 61.13   50.93   34.17   23.71
  Qwen3.5-4B after  61.20   59.19   53.31   47.07
  Qwen3.8-27B before 12.34  11.36    9.27    7.49
  Qwen3.8-27B after  12.32  12.11   11.62   10.97
  (Lemonade's Vulkan llama.cpp b11393, same command and files: 4B 64.03 / 60.99 / 54.80 / 48.27,
   27B 12.47 / 9.06 / 11.78 / 11.14; 27B d<32768 rows on the 08bb2241 base.)
Qwen3.8-27B --mtp, llama-server, 26,644-token prompt, 64 greedy tokens: 16.40 -> 23.78 tok/s (medians of 4).
Correctness: kernel bench vs a double-precision reference on 17 shapes (G 1..8, T 1..8, gate on/off) PASS,
5 repeats bit-identical; greedy text vs the Loom kernel identical on Qwen3.5-4B at 180 / 6,662 / 26,644
tokens (mean top-10 KL <= 1.7e-4), Qwen3.8-27B 26,644 tokens mean KL 3.7e-5 then a near-tie flip at token
10 (logprob gap 0.008); identical requests repeat exactly; no GPU fault in dmesg.

Needs the hw-gate run (label) before merge: goldens with Qwen3.8-27B decode past 1024 KV cells may drift on
near-ties.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong bong-water-water-bong added the hw-gate Run the HRX hardware gate on Strix Halo (docs/hw-gate.md) label Oct 22, 2026
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 6b898ff)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

24 - Partially compliant

Compliant requirements:

  • Update docs/lemonade.md to reflect that the Lemonade recipe lives in the 1bit-MONSTER/lemonade fork
  • Update the README status to reflect the change
  • Update PORTING step 1 to reflect the change

Non-compliant requirements:

  • None

Requires further human verification:

  • None
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Outdated Reference

The documentation still refers to the old commit cf222e7d for the private HIP kernels, while the actual pinned commit in config/gpu-kernels.config is 4f586caca4147b072b5056756a24852cd4a08b1b. This discrepancy could mislead users into thinking the wrong version is being used.

| private HIP kernels | [1bit-MONSTER/gpu-kernels](https://github.com/1bit-MONSTER/gpu-kernels) (not a submodule: a build points at a checkout with `-DONEBIT_GPU_PRIVATE`) | `4f586cac` — the commit the release builds and the hardware gate were run at |

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 5b979e4

@bong-water-water-bong
bong-water-water-bong merged commit df054b4 into main Oct 22, 2026
8 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the 1bit/pin-gk-4f586cac branch October 22, 2026 03:00
@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 6b898ff

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hw-gate Run the HRX hardware gate on Strix Halo (docs/hw-gate.md) Review effort 2/5

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant