Skip to content

E2E support for Qwen3 dense; fused QK-RMSNorm + RoPE kernel #210

Open

Description

@nayanp38

Currently, Qwen3 dense (0.6B-32B) isn't implemented, despite support for Qwen2 and Qwen3.5. Implementation itself would match almost perfectly onto existing ops (similar to Qwen2), with an additional per-head QK-RMSNorm.

I plan to approach this in 2 PRs:

  1. Integrating Qwen3 using existing ops, with the same shape as the Qwen2 implementation (alongside a Qwen3RMSNorm covering q_norm and k_norm).
  2. Fuse q norm, k norm, and RoPE into one fused_qk_norm_rope kernel that is GQA-aware. I'd implement it as a core op, following the existing _rope_kernel tiling and containing params to apply the general op to all supported models that can use it (Qwen3, Gemma3, Qwen3.5).

I'll be developing on my 3060ti with sm_86.

Some quick questions:

  • Anything already being worked on internally for Qwen3 dense?
  • Qwen3.5 already uses apply_tilegym_kernel_to_qwen3. Should I name the new one with _qwen3_dense, or rename the old one?
  • Does fused kernel fit as a core op, since its reusable accross Gemma3/Qwen3.5, or should I keep it local to Qwen3?

Activity

  1. lirundong commented on Oct 8, 2026

    @lirundong
    Collaborator

    Hi, Nayan, thanks for considering a contribution! Your PR plan looks good to us. Regarding your questions:

    1. Anything already being worked on internally for Qwen3 dense?: No, Qwen3 dense family, e.g., Qwen/Qwen3-0.6B, was not in our support plan;
    2. Qwen3.5 already uses apply_tilegym_kernel_to_qwen3. Should I name the new one with _qwen3_dense, or rename the old one?: A new patch func with _qwen3_dense would be more preferrable;
    3. Does fused kernel fit as a core op, since its reusable accross Gemma3/Qwen3.5, or should I keep it local to Qwen3?: Let's keep it local to qwen3 module.

    cc @hannahli-nv

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions