[rosenrodt]

Summary by CodeRabbit

Release Notes

• New Features

  • Added support for Qwen3.5 Mixture of Experts (MoE) model variant with proper weight mapping and layer configuration.
  • Enhanced hybrid Qwen3 model detection and layer type handling for improved architecture compatibility.
  • Improved chat template parameter handling with automatic detection of thinking-related features.

• Bug Fixes

  • Fixed KV cache manager selection for hybrid models to ensure correct caching behavior.
  • Enhanced FP8 quantization support for newer hardware architectures.

Description

This adds a thin wrapper for Qwen3.5 text model over the Qwen3-next implementation in order to reuse as much components as possible.

Verified on B200 BF16 and FP8 (deepgemm MoE backend)

Test Coverage

pytest -q -s tests/integration/defs/accuracy/test_llm_api_pytorch.py -k 'TestQwen3_5_35B_A3B and test_bf16'
pytest -q -s tests/integration/defs/accuracy/test_llm_api_pytorch.py -k 'TestQwen3_5_35B_A3B and test_fp8'

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds support for Qwen3.5 MoE model architecture including new model class Qwen3_5MoeForCausalLM, specialized weight mapper with FP8 dequantization and projection tensor packing, dynamic layer type selection for hybrid models, and chat template utilities for template argument handling.

Changes

Cohort / File(s)	Summary
Model Definition & Registry tensorrt_llm/_torch/models/modeling_qwen3_5.py, tensorrt_llm/_torch/models/__init__.py	Introduces Qwen3_5MoeForCausalLM class wrapping Qwen3NextModel with overridden load_weights and post_load_weights methods for weight mapping and layer normalization linking. Adds the class to public API exports.
Weight Mapper Implementation tensorrt_llm/_torch/models/checkpoints/hf/qwen3_5_weight_mapper.py, tensorrt_llm/_torch/models/checkpoints/__init__.py	Adds Qwen3_5MoeHfWeightMapper extending Qwen3NextHfWeightMapper with specialized weight normalization, MoE instance module handling, FP8 dequantization for linear-attention weights, and projection tensor packing into grouped-interleaved layout. Exports mapper in public API.
Model Config & Detection tensorrt_llm/_torch/model_config.py, tensorrt_llm/_torch/pyexecutor/config_utils.py	Updates get_num_attention_layers to use is_qwen3_hybrid for Qwen3 detection. Introduces comprehensive Qwen3 utilities: is_qwen3_5, is_qwen3_hybrid, get_qwen3_hybrid_layer_types, get_qwen3_hybrid_layer_masks, get_qwen3_hybrid_num_attention_layers. Adds _Qwen35ConfigCompat class for normalizing Qwen3.5 HF configs to Qwen3NextConfig shape with quantization and RoPE handling.
MoE Infrastructure tensorrt_llm/_torch/models/modeling_qwen3_next.py, tensorrt_llm/_torch/modules/fused_moe/fused_moe_trtllm_gen.py	Introduces weight_loading_mode selection for MoE blocks based on model_type. Replaces hardcoded layer types with dynamic lookup via get_qwen3_hybrid_layer_types. Removes router_logits argument from FP8 block-scale MoE backend call.
Quantization & SM100 Support tensorrt_llm/_torch/modules/fused_moe/quantization.py	Adds SM 100f-specific preprocessing for FP8 weight scales: detects expert IDs and applies resmoothing to weight_scale_inv entries via resmooth_to_fp8_e8m0 before delegating to base load_weights.
Executor & KV Cache tensorrt_llm/_torch/pyexecutor/_util.py, tensorrt_llm/_torch/pyexecutor/py_executor_creator.py	Replaces is_qwen3_next with is_qwen3_hybrid for hybrid model detection. Introduces _get_model_kv_cache_manager_cls for dynamic KV cache manager selection. Uses get_qwen3_hybrid_layer_masks for per-model layer mask computation in draft scenarios.
Evaluation Chat Templates tensorrt_llm/evaluate/interface.py, tensorrt_llm/evaluate/lm_eval.py	Adds get_chat_template_kwargs helper to inspect tokenizer chat_template and set enable_thinking=False by default when template contains "enable_thinking". Integrates into do_apply_chat_template and LmEvalWrapper/MultimodalLmEvalWrapper apply_chat_template flows.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

🚥 Pre-merge checks | ✅ 1 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 34.88% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check	❓ Inconclusive	The PR description lacks key details required by the template. Missing comprehensive explanation of changes, test coverage details, and proper justification for the implementation.	Expand the description section with concrete details about what was changed, why this approach was chosen, and how it reuses Qwen3-next components. Include full test command context and expected outcomes.

✅ Passed checks (1 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately describes the main change: adding initial Qwen3.5 text model support for the PyTorch backend with BF16/FP8 precision. The changeset comprehensively introduces Qwen3.5 MoE modeling, weight mapping, config utilities, and evaluation support across multiple modules, aligning perfectly with the title.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

📝 Coding Plan

• Generate coding plan for human review comments

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Tip

CodeRabbit can scan for known vulnerabilities in your dependencies using OSV Scanner.

OSV Scanner will automatically detect and report security vulnerabilities in your project's dependencies. No additional configuration is required.

[coderabbitai]

Actionable comments posted: 8

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

tensorrt_llm/_torch/pyexecutor/_util.py (1)

940-980: ⚠️ Potential issue | 🟠 Major

Preserve caller-supplied masks for Qwen3 hybrid KV caches.

Unlike the Nemotron branch just above, this path ignores layer_mask and never extends the masks for appended speculative layers. In one-engine speculative decoding, or when building a separate draft KV cache, the Qwen3 hybrid manager will still size/cache only the base model layers and miss the draft-only layers.

Suggested direction

-        hybrid_layer_mask, mamba_layer_mask = get_qwen3_hybrid_layer_masks(
-            config)
-        num_layers = sum(hybrid_layer_mask)
-        num_mamba_layers = sum(mamba_layer_mask)
+        base_hybrid_mask, base_mamba_mask = get_qwen3_hybrid_layer_masks(
+            config)
+        if layer_mask is not None:
+            hybrid_layer_mask = []
+            mamba_layer_mask = []
+            for i, include in enumerate(layer_mask):
+                is_attention = base_hybrid_mask[i] if i < len(base_hybrid_mask) else True
+                is_mamba = base_mamba_mask[i] if i < len(base_mamba_mask) else False
+                hybrid_layer_mask.append(is_attention and include)
+                mamba_layer_mask.append(is_mamba and include)
+        else:
+            hybrid_layer_mask = list(base_hybrid_mask)
+            mamba_layer_mask = list(base_mamba_mask)
+            if spec_config is not None:
+                num_spec_layers = get_num_spec_layers(spec_config)
+                hybrid_layer_mask.extend([True] * num_spec_layers)
+                mamba_layer_mask.extend([False] * num_spec_layers)
+        num_layers = sum(hybrid_layer_mask)
+        num_mamba_layers = sum(mamba_layer_mask)

🧹 Nitpick comments (2)

tensorrt_llm/_torch/models/checkpoints/hf/qwen3_5_weight_mapper.py (1)

46-47: Make _SUPPORTED_SUFFIXES immutable.

This is acting as a constant, but a mutable class-level set is shared across all instances and is what Ruff is warning about here. frozenset(...) keeps the intent and removes the shared-mutation footgun.

♻️ Proposed fix

-    _SUPPORTED_SUFFIXES = {"weight", "bias", "weight_scale_inv"}
+    _SUPPORTED_SUFFIXES = frozenset({"weight", "bias", "weight_scale_inv"})

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tensorrt_llm/_torch/models/checkpoints/hf/qwen3_5_weight_mapper.py` around
lines 46 - 47, The class-level constant _SUPPORTED_SUFFIXES is defined as a
mutable set which can be accidentally changed at runtime; change its definition
to an immutable frozenset to preserve constant semantics and avoid
shared-mutation issues (replace the set literal in _SUPPORTED_SUFFIXES with
frozenset({"weight","bias","weight_scale_inv"}) so code referencing
_SUPPORTED_SUFFIXES continues to work).

tensorrt_llm/_torch/models/modeling_qwen3_5.py (1)

11-61: Consider reusing the existing Qwen3NextForCausalLM wrapper instead of copying its setup.

This class appears to differ mainly by registration name, but __init__, load_weights(), and post_load_weights() now need to stay in sync with the Qwen3Next wrapper in tensorrt_llm/_torch/models/modeling_qwen3_next.py. A thin subclass or shared helper would reduce drift.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tensorrt_llm/_torch/models/modeling_qwen3_5.py` around lines 11 - 61, The
Qwen3_5MoeForCausalLM class duplicates logic from Qwen3NextForCausalLM; replace
the copy with reuse by making Qwen3_5MoeForCausalLM a thin subclass or
delegating to Qwen3NextForCausalLM so __init__, load_weights, and
post_load_weights remain shared; specifically, have Qwen3_5MoeForCausalLM
inherit from Qwen3NextForCausalLM (or call its methods) and only override what’s
necessary (the registration/name), keeping references to Qwen3_5MoeForCausalLM,
Qwen3NextForCausalLM, __init__, load_weights, and post_load_weights to locate
the code to change.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@tensorrt_llm/_torch/models/checkpoints/hf/qwen3_5_weight_mapper.py`:
- Around line 1-13: This file is missing the required NVIDIA Apache-2.0 header
and uses symbol-level imports that violate the repo namespace-import rule; add
the standard NVIDIA Apache-2.0 license header with the current year at the top
of tensorrt_llm/_torch/models/checkpoints/hf/qwen3_5_weight_mapper.py, then
convert the symbol imports to module-scoped imports (e.g., import
tensorrt_llm._torch.models.checkpoints.hf.qwen3_next_weight_mapper as
qwen3_next_weight_mapper and import tensorrt_llm._torch.models.modeling_utils as
modeling_utils and import tensorrt_llm._torch.modules.fused_moe.interface as
fused_moe_interface) and update usages of Qwen3NextHfWeightMapper,
register_mapper, MoE, and MoEWeightLoadingMode to reference the module-qualified
names (qwen3_next_weight_mapper.Qwen3NextHfWeightMapper,
modeling_utils.register_mapper, fused_moe_interface.MoE,
fused_moe_interface.MoEWeightLoadingMode).

In `@tensorrt_llm/_torch/models/modeling_qwen3_5.py`:
- Around line 1-8: Add the required NVIDIA Apache-2.0 license header (with the
current year) at the top of the file, then convert all symbol-level imports to
module-scoped imports: import the modules that contain Qwen3NextConfig,
BaseWeightMapper, ModelConfig, Qwen3NextModel, SpecDecOneEngineForCausalLM, and
register_auto_model instead of importing those symbols directly, and update any
usages in this file to reference the symbols via their module namespace (e.g.,
module.Qwen3NextConfig, module.BaseWeightMapper, module.ModelConfig,
module.Qwen3NextModel, module.SpecDecOneEngineForCausalLM,
module.register_auto_model).

In `@tensorrt_llm/_torch/modules/fused_moe/quantization.py`:
- Around line 1113-1118: Avoid caching CUDA-resident presmoothed tensors in the
global weights dict: resmooth_to_fp8_e8m0 (referenced here) moves outputs to
CUDA, and assigning them into weights[weight_name]/weights[name] holds full
experts on GPU until super().load_weights() completes and can OOM; instead
perform resmooth_to_fp8_e8m0 lazily inside the per-expert load path (where each
expert shard is copied), copy the resulting shard out to the target storage,
then immediately discard or move the temporary presmoothed tensors back to CPU
(or do not reassign them into weights) so no full expert remains resident on
CUDA between load steps.
- Around line 1102-1118: The SM100f branch is performing a full resmooth via
resmooth_to_fp8_e8m0, causing a second resmooth later when
resmooth_and_transform_fp8_scale runs; change the flow so that when is_sm_100f()
triggers and you call resmooth_to_fp8_e8m0 (in the block using
module.initial_local_expert_ids and need_load_shared_weights), you mark that the
weights were already resmoothed (e.g., set a boolean flag on the module or add a
sentinel entry to the weights dict) and then update the later post_load_weights
step that calls resmooth_and_transform_fp8_scale to check that marker and only
run the transform portion (not the resmooth) when the marker is present;
reference the functions is_sm_100f(), need_load_shared_weights(),
resmooth_to_fp8_e8m0(), resmooth_and_transform_fp8_scale(), and the
post_load_weights flow to implement the conditional skip.

In `@tensorrt_llm/_torch/pyexecutor/config_utils.py`:
- Around line 241-255: The compat branch for Qwen3.5 only matches model_type ==
"qwen3_5_moe" or architectures[0] == "Qwen3_5MoeForCausalLM", but downstream
expects flattened text configs (e.g. "qwen3_5_moe_text") and the wrapper variant
"Qwen3_5MoeForConditionalGeneration"; update the conditional in the factory that
chooses transformers.Qwen3NextConfig.from_dict(...) to also match model_type ==
"qwen3_5_moe_text" and the wrapper architecture name (e.g.
"Qwen3_5MoeForConditionalGeneration"), so that
_Qwen35ConfigCompat.normalize(config_dict) is applied for those IDs as well
before falling back to AutoConfig/other _CONFIG_REGISTRY paths (refer to
model_type, architectures, _Qwen35ConfigCompat.normalize, and
Qwen3NextConfig.from_dict).

In `@tensorrt_llm/_torch/pyexecutor/py_executor_creator.py`:
- Around line 36-39: The file currently imports is_mla twice (once from ._util
and again from .config_utils), causing a duplicate import (Flake8 F811); remove
the redundant is_mla entry from the second import line that lists is_mla,
is_nemotron_hybrid, is_qwen3_hybrid so that is_mla remains only imported from
._util while preserving imports for is_nemotron_hybrid and is_qwen3_hybrid.
- Around line 525-529: When you disable kv_cache_config.enable_block_reuse for
hybrid models in the conditional that checks is_nemotron_hybrid(config) or
is_qwen3_hybrid(config), also update the already-constructed
AttentionRuntimeFeatures passed into engines by setting its cache_reuse to match
the new value; specifically, after setting kv_cache_config.enable_block_reuse =
False, iterate the created model engines (or the single model_engine) and assign
model_engine.attn_runtime_features.cache_reuse = False so runtime features are
in sync with kv_cache_config and avoid stale behavior.

In `@tensorrt_llm/evaluate/lm_eval.py`:
- Around line 277-282: The chat-template kwargs are being filtered against
self.llm.tokenizer before the actual template/processor is chosen, which can
drop processor-specific (e.g., Qwen) args; update the code so
get_chat_template_kwargs is applied against the resolved template/processor (the
final template returned/used by inputs.utils.apply_chat_template) instead of
self.llm.tokenizer, or modify get_chat_template_kwargs to accept the resolved
template/processor and perform filtering there; locate uses around
apply_chat_template and get_chat_template_kwargs and move or extend the helper
so multimodal/processor-owned kwargs are preserved.

---

Nitpick comments:
In `@tensorrt_llm/_torch/models/checkpoints/hf/qwen3_5_weight_mapper.py`:
- Around line 46-47: The class-level constant _SUPPORTED_SUFFIXES is defined as
a mutable set which can be accidentally changed at runtime; change its
definition to an immutable frozenset to preserve constant semantics and avoid
shared-mutation issues (replace the set literal in _SUPPORTED_SUFFIXES with
frozenset({"weight","bias","weight_scale_inv"}) so code referencing
_SUPPORTED_SUFFIXES continues to work).

In `@tensorrt_llm/_torch/models/modeling_qwen3_5.py`:
- Around line 11-61: The Qwen3_5MoeForCausalLM class duplicates logic from
Qwen3NextForCausalLM; replace the copy with reuse by making
Qwen3_5MoeForCausalLM a thin subclass or delegating to Qwen3NextForCausalLM so
__init__, load_weights, and post_load_weights remain shared; specifically, have
Qwen3_5MoeForCausalLM inherit from Qwen3NextForCausalLM (or call its methods)
and only override what’s necessary (the registration/name), keeping references
to Qwen3_5MoeForCausalLM, Qwen3NextForCausalLM, __init__, load_weights, and
post_load_weights to locate the code to change.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 66b42b7e-9154-420f-8255-50258441c259

📥 Commits

Reviewing files that changed from the base of the PR and between fe77b40 and ba5480a.

📒 Files selected for processing (13)

• tensorrt_llm/_torch/model_config.py
• tensorrt_llm/_torch/models/__init__.py
• tensorrt_llm/_torch/models/checkpoints/__init__.py
• tensorrt_llm/_torch/models/checkpoints/hf/qwen3_5_weight_mapper.py
• tensorrt_llm/_torch/models/modeling_qwen3_5.py
• tensorrt_llm/_torch/models/modeling_qwen3_next.py
• tensorrt_llm/_torch/modules/fused_moe/fused_moe_trtllm_gen.py
• tensorrt_llm/_torch/modules/fused_moe/quantization.py
• tensorrt_llm/_torch/pyexecutor/_util.py
• tensorrt_llm/_torch/pyexecutor/config_utils.py
• tensorrt_llm/_torch/pyexecutor/py_executor_creator.py
• tensorrt_llm/evaluate/interface.py
• tensorrt_llm/evaluate/lm_eval.py

💤 Files with no reviewable changes (1)

• tensorrt_llm/_torch/modules/fused_moe/fused_moe_trtllm_gen.py

[rosenrodt]

/bot run

[tensorrt-cicd]

PR_Github #39178 [ run ] triggered by Bot. Commit: 291d846 Link to invocation

[tensorrt-cicd]

PR_Github #39178 [ run ] completed with state FAILURE. Commit: 291d846
/LLM/main/L0_MergeRequest_PR pipeline #30433 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[rosenrodt]

/bot run

[tensorrt-cicd]

PR_Github #39271 [ run ] triggered by Bot. Commit: f9608c5 Link to invocation

[tensorrt-cicd]

PR_Github #39271 [ run ] completed with state SUCCESS. Commit: f9608c5
/LLM/main/L0_MergeRequest_PR pipeline #30518 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[rosenrodt]

/bot run

[tensorrt-cicd]

PR_Github #39347 [ run ] triggered by Bot. Commit: 39706df Link to invocation

[yuxianq]

layer_wise_benchmarks/runner.py not updated — Qwen3.5 will fail in the benchmark tool

The PR replaces is_qwen3_next with is_qwen3_hybrid in _util.py, py_executor_creator.py, and model_config.py, but tensorrt_llm/tools/layer_wise_benchmarks/runner.py (lines 651, 845) still uses is_qwen3_next(pretrained_config).

Qwen3.5 models will:

• Skip the Mamba2Metadata setup (line 651) → crash or incorrect benchmark results
• Fall through to KVCacheManager instead of MambaHybridCacheManager (line 845) → wrong cache manager for hybrid models
• Hit the interval-based mask logic (line 856) which won't handle Qwen3.5's layer_types-based config

Suggested fix: Update runner.py to import and use is_qwen3_hybrid and get_qwen3_hybrid_layer_masks from config_utils, mirroring the refactoring pattern applied in _util.py.

[rosenrodt]

layer_wise_benchmarks/runner.py not updated — Qwen3.5 will fail in the benchmark tool

The PR replaces is_qwen3_next with is_qwen3_hybrid in _util.py, py_executor_creator.py, and model_config.py, but tensorrt_llm/tools/layer_wise_benchmarks/runner.py (lines 651, 845) still uses is_qwen3_next(pretrained_config).

Qwen3.5 models will:

• Skip the Mamba2Metadata setup (line 651) → crash or incorrect benchmark results
• Fall through to KVCacheManager instead of MambaHybridCacheManager (line 845) → wrong cache manager for hybrid models
• Hit the interval-based mask logic (line 856) which won't handle Qwen3.5's layer_types-based config

Suggested fix: Update runner.py to import and use is_qwen3_hybrid and get_qwen3_hybrid_layer_masks from config_utils, mirroring the refactoring pattern applied in _util.py.

Added fix in 19ef3b6

[tensorrt-cicd]

PR_Github #39542 [ run ] completed with state SUCCESS. Commit: dbdc4a4
/LLM/main/L0_MergeRequest_PR pipeline #30760 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

[rosenrodt]

/bot run

[tensorrt-cicd]

PR_Github #39602 [ run ] triggered by Bot. Commit: 344ffbb Link to invocation

[tensorrt-cicd]

PR_Github #39602 [ run ] completed with state SUCCESS. Commit: 344ffbb
/LLM/main/L0_MergeRequest_PR pipeline #30810 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[rosenrodt]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #39616 [ run ] triggered by Bot. Commit: 344ffbb Link to invocation

[tensorrt-cicd]

PR_Github #39616 [ run ] completed with state SUCCESS. Commit: 344ffbb
/LLM/main/L0_MergeRequest_PR pipeline #30823 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[rosenrodt]

/bot run

[tensorrt-cicd]

PR_Github #39662 [ run ] triggered by Bot. Commit: 344ffbb Link to invocation

[tensorrt-cicd]

PR_Github #39662 [ run ] completed with state SUCCESS. Commit: 344ffbb
/LLM/main/L0_MergeRequest_PR pipeline #30866 completed with status: 'SUCCESS'

CI Report

Link to invocation