Pin ROCmFPX d348b83: row-tiled Q4_0 verify kernel (KernelForge), DFlash2 decode 46 / 28 / 18 tok/s

README.md

@@ -62,7 +62,7 @@ Windows and 1bit OS ([docs/releases.md](docs/releases.md)).
 > ([docs/vulkan.md](docs/vulkan.md#zaya1-zyphra-from-our-llamacpp)); the rest of Zyphra's
 > family (Zamba, Zamba2, BlackMamba) runs on Vulkan too, and so do its vision models, ZAYA1-VL-8B
 > and Zamba2-VL, through `1bit serve --mmproj`. Qwen3.8-27B runs in one ROCm server with Hadamard
-> W4A4 prompt processing and DFlash2 decode: 516 tok/s on a 1,800-token prompt and 42 tok/s
+> W4A4 prompt processing and DFlash2 decode: 518 tok/s on a 1,800-token prompt and 46 tok/s
 > decode on code ([docs/lean.md](docs/lean.md#hadamard-rotated-q4_0-w4a4-prompt-processing)). W4A4 covers MoE
 > experts too, and `1bit serve --long-model` sends long prompts there and short ones to Vulkan: on
 > Qwen3-Coder-30B-A3B a 16K-token prompt gets its answer at 13.1 effective tok/s against 8.6 on

docs/lean.md

@@ -309,15 +309,24 @@ included). Decode is 256 greedy tokens on the code / prose / short prompts (best
 
 | Qwen3.8-27B-Q4_0-H32, lean ROCm route | Prompt t/s | Decode tok/s |
 |---|---|---|
-| no drafter | 558 / 550 | 13.2 / 13.2 / 13.6 |
-| `--dflash` (DFlash2 Q8_0) | **521** / **516** | **42.1** / **24.9** / 13.6 |
+| no drafter | 559 / 546 | 13.1 / 13.1 / 13.5 |
+| `--dflash` (DFlash2 Q8_0) | **518** / 486 | **46.0** / **28.4** / **17.6** |
 
-Before the prefill kernel of ROCmFPX#5, the same run measured 502 / 494 and 470 / 467.
+The `--dflash` prompt median is low because one noisy round pulled it down; its best matches the
+earlier runs.
 
-The `--dflash` row drafted with p-min 0. Serve now passes 0.4 on this route (recipe
-`rocm-dflash-p-min-0.4`, [recipes.md](recipes.md)). That stops a block at the first guess the
-selector is less than 40% sure of. Medians, code / prose / short: 41.9 / 26.8 / 16.7 tok/s, against
-41.9 / 24.8 / 13.4 at 0 in the same run.
+How the decode got there, as measured with this tool:
+
+- **p-min 0.4.** Serve passes it on this route (recipe `rocm-dflash-p-min-0.4`,
+  [recipes.md](recipes.md)). A block stops at the first guess the selector is less than 40% sure
+  of. Medians, code / prose / short: 41.9 / 26.8 / 16.7 tok/s, against 41.9 / 24.8 / 13.4 at 0.
+- **A row-tiled Q4_0 kernel for 8-column verification batches** (ROCmFPX#6, found by KernelForge,
+  [tools/kernelforge](../tools/kernelforge/README.md)). One warp computes 4 rows, so each
+  column's activations are loaded once and each weight is unpacked once for all 8 columns. Its
+  output is bit for bit the old kernel's. Qwen3.8-27B 8-token batches: 71.3 -> 78.3 tok/s.
+
+The earlier states, same tool: before ROCmFPX#5's prefill kernel, 502 / 494 (no drafter) and 470 /
+467 (DFlash2) prompt; with it but before these two, 516 prompt and 42.1 / 24.9 / 13.6 decode.
 
 Mean accepted block: 6.54 tokens on code, 4.23 on prose, in line with upstream on this drafter
 (6.71 / 4.25). Greedy output matches the no-drafter run on the code and short prompts; on prose

docs/serve.md

@@ -184,9 +184,9 @@ drafter is faster than BF16 (42.5 vs 38.9 on code, direct llama-server). A draft
 kept only when the model agrees, so the output is the model's own.
 
 `--dflash` also works on the lean ROCm route: a Hadamard-rotated Q4_0 file gets W4A4 prompt
-processing and DFlash2 decode from one server, 516 t/s prompt and 42.1 / 24.9 tok/s decode
-([lean.md](lean.md#hadamard-rotated-q4_0-w4a4-prompt-processing)). `--dflash` sets
-`--spec-draft-p-min 0` unless `--mtp-p-min` is given, and without `--mtp-max` drafts the
+processing and DFlash2 decode from one server, 518 t/s prompt and 46.0 / 28.4 / 17.6 tok/s decode
+([lean.md](lean.md#hadamard-rotated-q4_0-w4a4-prompt-processing)). `--dflash` gets
+`--spec-draft-p-min` from a [recipe](recipes.md) (0.4 on rocm, 0 elsewhere) unless `--mtp-p-min` is given, and without `--mtp-max` drafts the
 drafter's block minus one (its `dflash.block_size`; 16 when the file does not say).
 
 ## Recipes (`--recipes`, `--no-recipes`)

tools/kernelforge/README.md

@@ -32,6 +32,7 @@ Halo's gfx1151 instead. Each task packages one of the engine's kernels with:
 | Task | Kernel | Where it runs in the engine |
 |---|---|---|
 | [`gdn-prefill/`](gdn-prefill/) | gated delta-net prefill (head size 128, optional rollback snapshots) | Qwen3.8 / Qwen3.6 prompt processing on ROCm, from ROCmFPX's `gated_delta_net.cu` |
+| [`gemv-q4_0/`](gemv-q4_0/) | Q4_0 x Q8_1 for 1-8 columns (`mul_mat_vec_q`) | speculative verification (and plain decode) on ROCm, from ROCmFPX's `mmvq.cu` |
 
 ## Run one
 
@@ -78,7 +79,28 @@ task driver's gentler gate range had let it pass. The rest went into ROCmFPX#5:
 | Qwen3.8-27B-H32 serve prompt | 484-495 tok/s | 542-555 tok/s |
 | Same, with DFlash2 | 456-465 tok/s | 510-522 tok/s |
 
-The driver now draws gates from ggml's own test range. Two setup traps:
+**gemv-q4_0, 2026-09-29.** A 2-hour campaign, about $40, kept a row-tiled body for 5-8 columns:
+
+- 4 rows per warp;
+- activations loaded once per k-block;
+- weights unpacked once for all columns;
+- a reduce-scatter epilogue.
+
+The mean case score is 1.18x. Byte for byte it gives the old kernel's output: 40 cases, with odd
+row counts, extreme scales, edge nibbles, and saturated and zero activations. The task's driver
+now carries those adversarial inputs itself.
+
+| Result | Before | After |
+|---|---|---|
+| 5120 x 17408, 8 columns | 0.298 ms | 0.226 ms |
+| Qwen3.8-27B pp8 | 71.3 tok/s | 78.3 tok/s |
+| Serve + DFlash2 decode, code (median, 2 rounds) | 43.0 / 39.4 | 46.1 / 45.7 |
+| Serve + DFlash2 decode, prose (median, 2 rounds) | 20.2 / 25.9 | 28.1 / 28.1 |
+
+In the full model it helped only at 8 columns (5-7 were level or slower), so ROCmFPX#6 dispatches
+it for exactly 8.
+
+The gdn-prefill driver now draws gates from ggml's own test range. Two setup traps:
 
 - **Login.** KernelForge runs its planning and implementer sessions as `claude --bare`, which
   ignores a logged-in subscription and accepts only an API key. Our Hyperloom checkout on strixhalo

tools/kernelforge/gemv-q4_0/driver.py

@@ -0,0 +1,186 @@
+# Copyright 2026 bong-water-water-bong
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Measurement driver for the Q4_0 x Q8_1 few-column matmul task (KernelForge's driver contract).
+
+Correctness (default): SNR of every case against a float64 oracle computed from the same quantized
+blocks, over ordinary inputs AND adversarial ones (huge and tiny scales, all-edge nibbles 0 / 15,
+saturated int8 activations, a zero column), so a kernel that is only right on friendly data fails.
+--bench-mode: median kernel time per case (HIP events), `case_ms` lines.
+--profile-run: warm the kernel, then run it a few times for the profiler.
+
+Cases are Qwen3.8-27B's FFN matrices (5120 -> 17408 and 17408 -> 5120) with 8 columns (a
+speculative verification batch), 4 columns, and 1 column (plain decode, which must not regress).
+The pip ROCm torch has no gfx1151 kernels of its own, so everything but the kernel under test runs
+on the CPU.
+"""
+from __future__ import annotations
+
+import argparse
+import math
+import sys
+
+import numpy as np
+import torch
+
+from gemv_q4_0_kernel import QK, gemv_q4_0
+
+CASES = [  # (ncols_x = K, nrows = N, ncols = batch)
+    (5120, 17408, 8),
+    (17408, 5120, 8),
+    (5120, 17408, 4),
+    (5120, 17408, 1),
+]
+CHECK_CASES = [(512, 96, c) for c in range(1, 9)] + [(5120, 1024, 8), (17408, 256, 3)]
+_SEED = 5
+
+
+def quantize_q4_0(w, rng=None, mode="normal"):
+    """w: float [N, K] -> (bytes [N*K/32*18], dequant ints q in 0..15 [N, K/32, 32], scales d [N, K/32])."""
+    N, K = w.shape
+    b = w.reshape(N, K // QK, QK)
+    amax_idx = np.abs(b).argmax(axis=-1)
+    maxv = np.take_along_axis(b, amax_idx[..., None], axis=-1)[..., 0]
+    d = (maxv / -8.0).astype(np.float16)
+    df = d.astype(np.float64)
+    inv = np.where(df != 0, 1.0 / np.where(df != 0, df, 1), 0.0)
+    q = np.clip(np.floor(b * inv[..., None] + 8.5), 0, 15).astype(np.uint8)
+    if mode == "edges":
+        q = np.where(rng.random(q.shape) < 0.5, 0, 15).astype(np.uint8)
+    packed = (q[..., :16] | (q[..., 16:] << 4)).astype(np.uint8)
+    raw = np.zeros((N, K // QK, 18), dtype=np.uint8)
+    raw[..., 0:2] = d.view(np.uint8).reshape(N, K // QK, 2)
+    raw[..., 2:] = packed
+    return raw.reshape(-1), q, d
+
+
+def quantize_q8_1(a):
+    """a: float [C, K] -> (bytes [C*K/32*36], ints q [C, K/32, 32], d [C, K/32], s = d*sum(q) as half)."""
+    C, K = a.shape
+    b = a.reshape(C, K // QK, QK)
+    amax = np.abs(b).max(axis=-1)
+    d = amax / 127.0
+    inv = np.where(d != 0, 1.0 / np.where(d != 0, d, 1), 0.0)
+    q = np.round(b * inv[..., None]).clip(-127, 127).astype(np.int8)
+    d16 = d.astype(np.float16)
+    s16 = (d * q.astype(np.float64).sum(axis=-1)).astype(np.float16)
+    raw = np.zeros((C, K // QK, 36), dtype=np.uint8)
+    raw[..., 0:2] = d16.view(np.uint8).reshape(C, K // QK, 2)
+    raw[..., 2:4] = s16.view(np.uint8).reshape(C, K // QK, 2)
+    raw[..., 4:] = q.view(np.uint8)
+    return raw.reshape(-1), q, d16, s16
+
+
+def reference(q4, d4, q8, d8, s8):
+    """float64 out [C, N] = sum_b d4 * (d8 * sum_j q4_j q8_j - 8 * s8), as the kernel defines it."""
+    dots = np.einsum("nbj,cbj->cnb", q4.astype(np.float64), q8.astype(np.float64))
+    return np.einsum("nb,cnb->cn", d4.astype(np.float64),
+                     d8.astype(np.float64)[:, None, :] * dots - 8.0 * s8.astype(np.float64)[:, None, :])
+
+
+def make_case(K, N, C, mode, seed=_SEED):
+    rng = np.random.default_rng(seed + K + 7 * N + 13 * C)
+    w = rng.standard_normal((N, K))
+    a = rng.standard_normal((C, K))
+    if mode == "scales":        # scales across many orders of magnitude, per block
+        w *= 10.0 ** rng.uniform(-4, 3, size=(N, K // QK, 1)).repeat(QK, axis=2).reshape(N, K)
+        a *= 10.0 ** rng.uniform(-3, 2, size=(C, K // QK, 1)).repeat(QK, axis=2).reshape(C, K)
+    if mode == "saturated":     # activations at the int8 limits, one column all zero
+        a = np.sign(a) * 50.0
+        a[C - 1] = 0.0
+    xr, q4, d4 = quantize_q4_0(w, rng, "edges" if mode == "edges" else "normal")
+    yr, q8, d8, s8 = quantize_q8_1(a)
+    x = torch.from_numpy(xr).cuda()
+    y = torch.from_numpy(yr).cuda()
+    return x, y, (q4, d4, q8, d8, s8)
+
+
+def snr_db(ref, test):
+    noise = float(((test - ref) ** 2).mean())
+    sig = float((ref ** 2).mean())
+    if noise <= 0.0:
+        return 200.0   # exact: above any real SNR, so the minimum over cases stays informative
+    return 10.0 * math.log10(sig / noise) if sig > 0 else 0.0
+
+
+def _run_correctness():
+    worst, ok = float("inf"), True
+    for mode in ("normal", "scales", "edges", "saturated"):
+        for K, N, C in CHECK_CASES:
+            x, y, parts = make_case(K, N, C, mode)
+            out = gemv_q4_0(x, y, K, N, C)
+            torch.cuda.synchronize()
+            out = out.cpu().double().numpy()
+            ref = reference(*parts)
+            finite = bool(np.isfinite(out).all())
+            s = snr_db(ref, out) if finite else float("nan")
+            close = finite and np.allclose(out, ref, rtol=1e-3, atol=1e-3 * max(1e-30, float(np.abs(ref).max())))
+            if not finite or not close or s < 80:
+                print(f"# {mode} K{K} N{N} C{C}: SNR {s:.2f} dB, finite {finite}, allclose {close}")
+            worst = min(worst, s) if finite else float("nan")
+            ok = ok and close
+    print(f"SNR: {worst:.2f} dB")
+    print(f"allclose: {ok}")
+    return 0
+
+
+def _run_bench(warmup, iters):
+    for K, N, C in CASES:
+        x, y, _ = make_case(K, N, C, "normal")
+        for _ in range(warmup):
+            gemv_q4_0(x, y, K, N, C)
+        torch.cuda.synchronize()
+        times = []
+        for _ in range(iters):
+            a, b = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)
+            a.record()
+            gemv_q4_0(x, y, K, N, C)
+            b.record()
+            b.synchronize()
+            times.append(a.elapsed_time(b))
+        times.sort()
+        for t in times:
+            print(f"wall_ms: {t:.6f}")
+        print(f"case_ms: K{K}_N{N}_C{C} {times[len(times) // 2]:.6f}")
+    return 0
+
+
+def _run_profile():
+    for K, N, C in CASES:
+        x, y, _ = make_case(K, N, C, "normal")
+        for _ in range(6):
+            gemv_q4_0(x, y, K, N, C)
+    torch.cuda.synchronize()
+    return 0
+
+
+def main():
+    ap = argparse.ArgumentParser(description="Q4_0 x Q8_1 few-column matmul task driver")
+    ap.add_argument("--bench-mode", action="store_true")
+    ap.add_argument("--profile-run", action="store_true")
+    ap.add_argument("--warmup", type=int, default=10)
+    ap.add_argument("--iters", type=int, default=50)
+    args, _ = ap.parse_known_args()
+    if not torch.cuda.is_available():
+        print("error: no GPU available (torch.cuda.is_available() is False)")
+        return 1
+    if args.profile_run:
+        return _run_profile()
+    if args.bench_mode:
+        return _run_bench(args.warmup, args.iters)
+    return _run_correctness()
+
+
+if __name__ == "__main__":
+    sys.exit(main())