Repository navigation
hrx: one-token MUL_MAT_ID as a GEMV over the selected experts (ZAYA1-8B decode +34%) - #32
Merged
bong-water-water-bong merged 1 commit intoSep 29, 2026
Conversation
…code) At decode the WMMA mul_mat_id kernel reads a single token's expert at ~45 GB/s (ZAYA1-8B: 80 calls x 85.6 us = 40% of a token's device time). The new ggml_mul_mat_id_decode_f32_wave64 (ops/mul_mat_id_decode_f32.loom, ours) runs the decode GEMV's per-block dequant/dot loop from ops/mul_mat_f32_f32_decode.loom (linked as a library) over rows expert*N + row, one wave64 per (row pair, token*slot); the router id is clamped to the expert count. Registered at priority 150 for <= 4 tokens and <= 64 token*slot rows, below the fused routed-FFN and SwiGLU paths. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bong-water-water-bong
merged commit Sep 29, 2026
8de144e
into
1bit/hrx-vulkan-patched
10 of 24 checks passed
bong-water-water-bong
added a commit
to 1bit-MONSTER/engine
that referenced
this pull request
Sep 29, 2026
…xperts (ZAYA1-8B +34%) (#218) Brings 1bit-MONSTER/llama.cpp#32: a one-token MUL_MAT_ID kernel (ops/mul_mat_id_decode_f32.loom) that reads only the routed expert's rows, ahead of the WMMA mul_mat_id kernel at <= 4 tokens. HRX0 decode: ZAYA1-8B 46.3 -> 62.0 tok/s, GLM-4.7-Flash 22.1 -> 25.3; ZAYA tf_cmp32 69/96 on both paths; batch-1 perplexity within 0.1% (ZAYA) / 0.35% (GLM) of the CPU backend. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Engine RFC #213, gate 1 (HRX decode speed). A dispatch profile of ZAYA1-8B decode on HRX0 (
HRX_PROFILE_MODE=dispatch, read withiree-profile executable) showed the MoE expert matmuls as the largest cost:ggml_mul_mat_id_f32_f32_wmma, 80 calls per token at 85.6 us each, 40% of the token's device time. ZAYA routes one expert per token, so each call reads ~3.8 MB: about 45 GB/s. The WMMA kernel is built for batches, and at one token it wastes most of each tile.Change
ops/mul_mat_id_decode_f32.loom(new, ours):ggml_mul_mat_id_decode_f32_wave64. It is a GEMV with one wave64 per (output-row pair, token × slot). It reads the slot's expert from the route ids (clamped to the expert count, so a bad router id cannot read past the weights), then runs the per-block dequant/dot loop ofops/mul_mat_f32_f32_decode.loomover rowsexpert * N + row. That loop is linked as a library and copied unchanged apart from the weight and input row indices. Per-slot inputs (input_rows= slot count) and ggml's strided route ids are supported.dispatch-mul-mat-id-decode.cpp(new, ours):common.mul_mat_id.decode_f32_wave64, priority 150, for ≤ 4 tokens and ≤ 64 token×slot rows. That is above the plain WMMA path (100) and below the fused SwiGLU/postops paths (180/200) and Qwen's routed-FFN kernels (900–1200), which are unchanged.dispatch-common.cpp, and two lines inCMakeLists.txt.Measured (Strix Halo,
llama-bench -p 0 -n 64 -r 2, A/B in one binary viaGGML_HRX_DISABLE_DISPATCH=mul_mat_id.decode, interleaved)Accuracy
cache_prompt): 69/96 before and after.-c 512 -b 1; every token goes through the decode path), against the CPU backend as an independent reference:The differences come from rounding. The decode GEMV rounds weights to F16 before the dot product, as HRX's dense decode matmuls already do. Both paths are deterministic run to run (288/288 identical tokens each).
test-backend-ops -o MUL_MAT_IDdoes not reach this path: HRX reports stand-alone MUL_MAT_ID as not supported, because its matchers need the model graph. Hence the model-level checks above.🤖 Generated with Claude Code