[dhiltgen]

Proposed changes

Use a narrower output tile for large NVFP4 matrix-vector products on M5 Max GPUs. Reducing the results per SIMD-group from four to two lowers register pressure for output dimensions of 4096 or greater.

Keep the existing kernel configuration for other architectures, quantization modes, and smaller output dimensions.

Add coverage for batched and unbatched large-output QMV across the supported floating-point types.

Performance

Apple M5 Max, 40-core GPU, 128 GB, macOS 26.5.2.

The optimized path is intentionally limited to M5 Max (applegpu_g17s), which is the only M5 configuration I was able to test. The narrower tile may also benefit other M5 GPUs, but I do not have the hardware to validate that tradeoff without risking regressions.

python -m mlx_lm.benchmark \
  --model mlx-community/Qwen3.5-4B-nvfp4 \
  -p 2048 -g 128 -n 9

After a warmup, baseline and candidate were run in B/A/A/B order. Results are the midpoint of the paired nine-trial medians.

Model	Prompt tok/s	Generation tok/s
Qwen3.5 4B NVFP4	5394.5 → 5416.0 (+0.4%)	128.0 → 131.5 (+2.7%)

Ollama MLX generation improved by 2.8% on Qwen3.5 4B NVFP4 and 3.9% on Qwen3.5 9B NVFP4; prompt throughput was effectively unchanged.

Checklist

Put an x in the boxes that apply.

• I have read the CONTRIBUTING document
• I have run pre-commit run --all-files to format my code / installed pre-commit prior to committing changes
• I have added tests that prove my fix is effective or that my feature works
• I have updated the necessary documentation (if needed)

[zcbenz]

Can you rebase the PR to fix conflicts with the global scale changes?

[zcbenz]

I can not observe performance improvement when benchmarking individual matmul:

script

import mlx.core as mx
import time

Ms = [1, 2]
# (N, K)
shapes = [(4096, 512), (4096, 4096), (16384, 16384), (16384, 28672)]


def timed(fn, reps=20):
    mx.eval(fn())
    ts = []
    for _ in range(reps):
        t0 = time.perf_counter()
        mx.eval(fn())
        ts.append(time.perf_counter() - t0)
    return sorted(ts)[int(0.95 * len(ts))] * 1000


def run_benchmark():
    results = {}
    for M in Ms:
        for N, K in shapes:
            w = mx.random.normal((N, K)).astype(mx.bfloat16)
            qw, sc = mx.quantize(w, mode="nvfp4")
            x = mx.random.normal((M, K)).astype(mx.bfloat16)
            t = timed(lambda x=x, qw=qw, sc=sc: mx.quantized_matmul(
                x, qw, sc, transpose=True, mode="nvfp4"))
            results[(M, N, K)] = t
    return results


best_results = {}

for run_idx in range(8):
    results = run_benchmark()
    for key, t in results.items():
        M, N, K = key
        if key not in best_results or t < best_results[key]:
            best_results[key] = t

for M in Ms:
    for N, K in shapes:
        t = best_results[(M, N, K)]
        print(f"{M}x{N}x{K} : {t:.3f} ms")

This branch:

1x4096x512 : 0.278 ms
1x4096x4096 : 0.290 ms
1x16384x16384 : 0.600 ms
1x16384x28672 : 0.787 ms
2x4096x512 : 0.278 ms
2x4096x4096 : 0.298 ms
2x16384x16384 : 0.631 ms
2x16384x28672 : 0.822 ms

Main branch:

1x4096x512 : 0.278 ms
1x4096x4096 : 0.285 ms
1x16384x16384 : 0.588 ms
1x16384x28672 : 0.833 ms
2x4096x512 : 0.281 ms
2x4096x4096 : 0.294 ms
2x16384x16384 : 0.601 ms
2x16384x28672 : 0.818 ms

But I can definitely observe the improvement on inference:

This branch:

Trial 1:  prompt_tps=5408.563, generation_tps=171.683, peak_memory=4.207, total_time=1.209
Trial 2:  prompt_tps=5430.998, generation_tps=171.576, peak_memory=4.208, total_time=1.208
Trial 3:  prompt_tps=5428.169, generation_tps=171.626, peak_memory=4.209, total_time=1.209
Trial 4:  prompt_tps=5406.959, generation_tps=171.568, peak_memory=4.209, total_time=1.211
Trial 5:  prompt_tps=5433.920, generation_tps=171.501, peak_memory=4.210, total_time=1.210
Trial 6:  prompt_tps=5416.891, generation_tps=171.406, peak_memory=4.210, total_time=1.212
Trial 7:  prompt_tps=5426.455, generation_tps=171.126, peak_memory=4.211, total_time=1.213
Trial 8:  prompt_tps=5414.991, generation_tps=171.208, peak_memory=4.211, total_time=1.212
Trial 9:  prompt_tps=5429.892, generation_tps=171.124, peak_memory=4.212, total_time=1.212
Averages: prompt_tps=5421.871, generation_tps=171.424, peak_memory=4.210

Main branch:

Trial 1:  prompt_tps=5434.188, generation_tps=168.376, peak_memory=4.206, total_time=1.223
Trial 2:  prompt_tps=5426.434, generation_tps=168.329, peak_memory=4.208, total_time=1.224
Trial 3:  prompt_tps=5389.171, generation_tps=168.336, peak_memory=4.208, total_time=1.226
Trial 4:  prompt_tps=5418.096, generation_tps=168.274, peak_memory=4.209, total_time=1.225
Trial 5:  prompt_tps=5425.207, generation_tps=167.070, peak_memory=4.210, total_time=1.234
Trial 6:  prompt_tps=5390.096, generation_tps=165.589, peak_memory=4.210, total_time=1.250
Trial 7:  prompt_tps=5366.185, generation_tps=164.369, peak_memory=4.211, total_time=1.248
Trial 8:  prompt_tps=5408.535, generation_tps=167.631, peak_memory=4.211, total_time=1.231
Trial 9:  prompt_tps=5406.592, generation_tps=167.742, peak_memory=4.211, total_time=1.228
Averages: prompt_tps=5407.167, generation_tps=167.302, peak_memory=4.209

I hope there is a easier way to measure improvements like this.