Open
Description
Currently, Qwen3 dense (0.6B-32B) isn't implemented, despite support for Qwen2 and Qwen3.5. Implementation itself would match almost perfectly onto existing ops (similar to Qwen2), with an additional per-head QK-RMSNorm.
I plan to approach this in 2 PRs:
- Integrating Qwen3 using existing ops, with the same shape as the Qwen2 implementation (alongside a
Qwen3RMSNormcoveringq_normandk_norm). - Fuse q norm, k norm, and RoPE into one
fused_qk_norm_ropekernel that is GQA-aware. I'd implement it as a core op, following the existing_rope_kerneltiling and containing params to apply the general op to all supported models that can use it (Qwen3, Gemma3, Qwen3.5).
I'll be developing on my 3060ti with sm_86.
Some quick questions:
- Anything already being worked on internally for Qwen3 dense?
- Qwen3.5 already uses
apply_tilegym_kernel_to_qwen3. Should I name the new one with_qwen3_dense, or rename the old one? - Does fused kernel fit as a core op, since its reusable accross Gemma3/Qwen3.5, or should I keep it local to Qwen3?
Metadata
Metadata
Assignees
Labels
No labels
Activity
lirundong commentedon Oct 8, 2026
Hi, Nayan, thanks for considering a contribution! Your PR plan looks good to us. Regarding your questions:
_qwen3_densewould be more preferrable;cc @hannahli-nv