[DeepSeek-V4] Loading nvidia/DeepSeek-V4-Pro-NVFP4 crashes: 'Linear' object has no attribute 'weight_scale' in fused_a loader (MIXED_PRECISION attn excluded → built BF16 but stored as FP8) #16196

Activity

1. github-actions

   added

   Customized kernels<NV>Specialized/modified CUDA kernels in TRTLLM for LLM ops, beyond standard TRT. Dev & perf.<NV>Specialized/modified CUDA kernels in TRTLLM for LLM ops, beyond standard TRT. Dev & perf.

   Model optimization<NV>Model-specific performance optimizations and tuning<NV>Model-specific performance optimizations and tuning

   on Jul 9, 2026
2. waynehacking8 commented on Jul 11, 2026

   

   waynehacking8

   on Jul 11, 2026

   More actions

   I tried to reproduce this on current main (8bcec87) with a tiny synthetic replica of the checkpoint (same hf_quant_config.json, tensor names, fp8-e4m3 weights + ue8m0 block scales, experts as ModelOpt NVFP4 quadruples) on an RTX PRO 6000, and main now fails earlier than your trace: DeepseekV4DecoderLayer hits the "MIXED_PRECISION is ambiguous" assert (modeling_deepseekv4.py:1762, added with #15414) during model construction, before load_weights ever runs. So on main this packaging needs layer-level MIXED_PRECISION resolution at construction first (the FIXME right above that assert acknowledges the gap), and then your fused_a fix -- the unguarded weight_scale deref plus raw-fp8 copy you diagnosed is still there in the else branch (modeling_deepseekv4.py:1032, same pattern at modeling_deepseekv3.py:557 for the V3/V32 arches). I have the guard+weight_dequant half ready on a branch (https://github.com/waynehacking8/TensorRT-LLM/tree/wayne/fix-16196-fused-a-dequant) if maintainers want it as a follow-on; caveats on my repro: main python over the 1.3.0rc20 wheel binaries, synthetic tiny checkpoint, not the real weights.
3. d3nb commented on Jul 11, 2026

   

   d3nb

   on Jul 11, 2026

   Author

   More actions

   Thanks @waynehacking8 — I tested your fused-A dequant fix against the real nvidia/DeepSeek-V4-Pro-NVFP4 weights (851 GB, 64 shards) on 4× B300, TP4, and it clears the loading crash.

   Setup: since the real checkpoint's deepseek_v4 model type isn't recognized by the transformers in the rc20 wheel (ValueError: ... does not recognize this architecture at config_utils.py:569, before load), I ran on 1.3.0rc15.post1 (which loads deepseek_v4 fine — the official deepseek-ai/DeepSeek-V4-Pro FP8+MXFP4 checkpoint loads + serves + generates end-to-end on the exact same setup, as a control). I ported your guard to rc15.post1's modeling_deepseekv4.py else-branch (line 1030), i.e. the same hasattr(module, "weight_scale") → weight_dequant(...) change.

   Result: the 'Linear' object has no attribute 'weight_scale' crash is gone — weights load to 100%. So the fused-A guard+dequant is confirmed on real weights, not just the synthetic replica. 🎉

   But a second, downstream blocker then surfaces during executor init (MLA attention warmup), and I believe it's the same root cause showing up at runtime:

   File ".../tensorrt_llm/_torch/modules/attention.py", line 2358, in _q_branch
       assert not self._is_fused_q_fp8_quant_enabled(num_generations=num_generations), (
   AssertionError: CuTe DSL q_b_proj path is incompatible with the fused FP8 q-quant branch
   

   _q_branch takes the CuTe-DSL path because q_b_proj.weight.dtype == torch.bfloat16 (attn is exclude_modules'd → built unquantized BF16), while _is_fused_q_fp8_quant_enabled() is still True — exactly the "future config drift" the assert's comment warns about. So dequantizing fused-A to BF16 makes the model load, but leaves the attention modules BF16 while the rest of the MLA path still assumes FP8 q-quant.

   This lines up with your point that the real fix is the construction-time, layer-level MIXED_PRECISION resolution (the MIXED_PRECISION is ambiguous assert + the FIXME above it): if the excluded-but-FP8 attn modules were built as FP8_BLOCK_SCALES at construction, they'd carry a weight_scale (so the fused-A else-branch never triggers) and the runtime _q_branch FP8-q-quant contract would hold. The guard+dequant is a good defensive fallback, but for this ModelOpt experts-only NVFP4 packaging it isn't sufficient on its own — the attn modules really need to end up FP8, not BF16.

   Happy to test a construction-time resolution patch on the real weights + B300 if you or the maintainers put one up. For reference, my env: rc15.post1 image, real nvidia/DeepSeek-V4-Pro-NVFP4, TP4 on 4× B300, moe_config.backend=TRTLLM.
4. waynehacking8 commented on Jul 11, 2026

   

   waynehacking8

   on Jul 11, 2026

   More actions

   Thanks for running the real thing -- that closes the loading half on actual weights, and your downstream _q_branch assert points at the same place my construction repro does. I think the underlying semantic bug is that this packaging's exclude_modules means "left at the base model's precision", and DeepSeek-V4-Pro's base attn IS block-scale FP8 (the repack's embedded quantization_config still says quant_method=fp8 with weight_block_size [128,128] alongside the NVFP4 moe_quant_algo), while TRT-LLM's exclude handling builds excluded modules as unquantized BF16 -- so the construction-time resolution should build the excluded attn Linears as FP8_BLOCK_SCALES, which gives them a weight_scale (the fused-A else branch then never fires) and keeps the MLA fused-q-quant contract intact, with the layer-level MIXED_PRECISION assert (modeling_deepseekv4.py:1762 on main) resolved the same way. I'll draft that construction-time patch on my branch and will gladly take you up on the B300 offer once it's up.
5. waynehacking8

   added a commit that references this issue on Jul 11, 2026

   [None][fix] Resolve MIXED_PRECISION construction for experts-only NVF…

   ...

   3972f5e
6. waynehacking8 commented on Jul 11, 2026

   

   waynehacking8

   on Jul 11, 2026

   More actions

   @d3nb the construction-time resolution is up on the same branch (https://github.com/waynehacking8/TensorRT-LLM/tree/wayne/fix-16196-fused-a-dequant, commit 3972f5e, on top of the loader guard). It does three things: derives an FP8_BLOCK_SCALES config from the checkpoint's embedded base quantization_config and builds the excluded attention / shared experts / MTP e_proj+h_proj with it (matching the native FP8 flow), resolves the decoder-layer algo from the per-layer quantized_layers table under MIXED_PRECISION (so the ambiguity assert goes away and the MoE stays NVFP4), and accepts the checkpoint-native layers.N.ffn.experts key form that these repacks emit. On my tiny replica: construction passes, weights load to 100%, the fused_a fp8 payload is bit-exact vs the checkpoint, and q_b_proj / shared experts end up FP8 with weight_scale present -- which should also satisfy the _q_branch fused-q-quant contract you hit. Not covered: runtime on real weights (your B300 loop), and MTP spec decoding (the MTP routed experts are stored at the base model's MXFP4 -- worth leaving spec-dec off for the first run).
7. d3nb commented on Jul 11, 2026

   

   d3nb

   on Jul 11, 2026

   Author

   More actions

   Update from the B300 real-weights side, @waynehacking8 — the construction-time resolution works end-to-end on the real nvidia/DeepSeek-V4-Pro-NVFP4, and the MTP path you flagged is now working too.

   Construction-time resolution — validated on real weights + B300

   I implemented the same idea you described (independently, on 1.3.0rc15.post1 in model_config.py rather than modeling_deepseekv4.py, since that's the wheel that loads deepseek_v4 configs for us): when quant_algo == MIXED_PRECISION and the checkpoint's base quantization_config is quant_method=fp8 / weight_block_size=[128,128], resolve the global to FP8_BLOCK_SCALES (reusing the native-fp8 path) so the excluded attention / shared-expert projections build as FP8 with a weight_scale, keep the per-layer NVFP4 experts, and keep the genuinely-unquantized modules (head, mtp) excluded.

   Result on 4× B300, TP4, moe_config.backend=TRTLLM:

   • weights load 100%, Application startup complete, chat completions generate correctly ("我是DeepSeek…");
   • q_b_proj / shared experts end up FP8 with weight_scale present — so the runtime _q_branch fused-FP8-q-quant assert I hit earlier is gone, exactly as you predicted.

   So both halves (fused-A dequant guard, and construction-time FP8 resolution) are now confirmed on the actual 851 GB checkpoint, not just synthetic replicas.

   The MTP piece: dense experts are NVFP4, MTP experts are MXFP4 (same checkpoint)

   You were right that MTP needed separate handling — the reason turned out to be a per-region layout split inside one checkpoint:

   • dense routed experts layers.0..60.ffn.experts → dtype U8 = NVFP4 (ModelOpt re-quantized these);
   • MTP routed experts mtp.0.ffn.experts → dtype I8 = MXFP4 (left at the base model's precision).

   The routed-MoE layout resolver detects a single layout (from layers.0.ffn.experts.0.w1.weight = NVFP4) and applies it to all MoE layers including the MTP layer, so the MTP experts get NVFP4 and die in fused_moe/quantization.py load_quant_scales. Detecting the MTP expert dtype separately (mtp.0.ffn.experts.0.w1.weight → I8 → W4A8_MXFP4_*) and covering the MTP layer indices with that config fixes it.

   With that, on 4× B300:

   • MTP=1 (checkpoint-native num_nextn_predict_layers): loads + serves + generates. Log: MTP routed experts layout=mxfp4 over 1 layer(s) / dense routed experts NVFP4.
   • MTP=3 (NVIDIA's B300 config — the checkpoint's single MTP layer is duplicated ×3): also loads + serves + generates, once the duplicated MTP layer indices model.layers.{num_hidden+i} are covered too. (One wrinkle: spec_config is None in from_pretrained in our path, so _get_mtp_num_layers returns 0 there — I currently cover a small generous range of MTP indices as a stopgap; the clean version would thread the model-side num_nextn_predict_layers in, or extend coverage where the MTP layers are duplicated.)

   Our model_config.py diff (rc15.post1) is small — happy to share it or fold the MTP-layout piece into your branch, whichever you prefer. We can re-run any revision of your branch against the real weights + B300 (both MTP off and MTP=1/3). For reference, env: 1.3.0rc15.post1, nvidia/DeepSeek-V4-Pro-NVFP4, TP4 on 4× B300, moe_config.backend=TRTLLM.
8. d3nb commented on Jul 11, 2026

   

   d3nb

   on Jul 11, 2026

   Author

   More actions

   @waynehacking8 I put the MTP-layout piece up as a PR against your branch so it stays consolidated: waynehacking8#1 (single change to _set_deepseek_v4_routed_moe_quant_config, +38/-6, on top of your 3972f5e). It detects the MTP routed-expert dtype separately (mtp.0.ffn.experts.0.w1.weight → I8/MXFP4) so the MTP layers stop inheriting the dense NVFP4 config. Validated on the real nvidia/DeepSeek-V4-Pro-NVFP4 + 4× B300 (MTP=1 and MTP=3, accept_len ≈ 2.86). Feel free to squash/reword or fold it in however you like.
9. waynehacking8

   added 2 commits that reference this issue on Jul 18, 2026

   [None][fix] Dequantize DeepSeek-V4 fused A when module is built unqua…

   ...

   a45ae7d

   [None][fix] Resolve MIXED_PRECISION construction for experts-only NVF…

   ...

   b3f55d7
10. waynehacking8

    added a commit that references this issue on Jul 27, 2026

    [None][fix] Dequantize DeepSeek-V4 fused A when module is built unqua…

    ...

    d2074da
11. waynehacking8

    added a commit that references this issue on Oct 17, 2026

    [NVIDIA#16196][fix] Describe the MTP layout case without its history

    ...

    1471ed4