w4a8_mxfp4_fp8 build failure

Currently, the quantization is failed when you build w4a8_mxfp4_fp8 engine using trtllm-build.

RuntimeError: Encounter error 'The value updated is not the same shape as the original. Updated: (5120, 1536), original: (5120, 3072)' for parameter 'transformer.layers.0.attention.qkv.weight'

TensorRT-LLM/tensorrt_llm/models/modeling_utils.py

Lines 784 to 789 in 751d5f1

if not from_pruned:

try:

param.value = weights[name]

except Exception as e:

raise RuntimeError(

f"Encounter error '{e}' for parameter '{name}'")

We tested on following environment. We haven't tried on B200, since we don't have it.

RTX 5090 and RTX PRO 6000
tensorrt_llm==0.21.0
nvidia_modelopt==0.33.0

@Barry-Delaney

@Barry-Delaney Could you please reply this issue? Thanks!

@youki-sada

@youki-sada ,
Thanks for reporting this issue! To help us reproduce and debug the w4a8_mxfp4_fp8 build failure, could you please provide more detailed reproduction steps?

@karljang

hf_ptq.py

python llm_ptq/hf_ptq.py --pyt_ckpt_path=meta-llama/Llama-3.2-3B-Instruct --export_path=/trt_engines/saved_models_3b_Wf4Af8KVf8 --sparsity_fmt=dense --qformat=w4a8_mxfp4_fp8 --calib_size=32 --batch_size=1 --inference_tensor_parallel=1 --inference_pipeline_parallel=1 --export_fmt=tensorrt_llm

The checkpoint is successfully exported.

~~~ omitted ~~~
model.layers.27.mlp.down_proj.input_quantizer    TensorQuantizer((4, 3) bit fake per-tensor amax=366.0000 calibrator=MaxCalibrator quant)
model.layers.27.mlp.down_proj.output_quantizer   TensorQuantizer(disabled)
model.layers.27.mlp.down_proj.weight_quantizer   TensorQuantizer((2, 1) bit fake block_sizes={-1: 32, 'type': 'dynamic', 'scale_bits': (8, 0)}, amax=None calibrator=MaxCalibrator quant)
lm_head.input_quantizer                          TensorQuantizer(disabled)
lm_head.output_quantizer                         TensorQuantizer(disabled)
lm_head.weight_quantizer                         TensorQuantizer(disabled)                                                                                                                                                                         675 TensorQuantizers found in model
~~~ omitted ~~~
Quantized model exported to :/trt_engines/saved_models_3b_Wf4Af8KVf8. Total time used 7.445543527603149s
########
GPU 0: Peak memory usage = 11.96 GB for all processes on the GPU
GPU 1: Peak memory usage = 0.64 GB for all processes on the GPU
########

/trt_engines/saved_models_3b_Wf4Af8KVf8/config.json

    "quantization": {
        "quant_algo": "W4A8_MXFP4_FP8",
        "kv_cache_quant_algo": "FP8",
        "group_size": 32,
        "smoothquant_val": 0.5,
        "clamp_val": null,
        "use_meta_recipe": false,
        "has_zero_point": false,
        "pre_quant_scale": false,
        "exclude_modules": [
            "lm_head"
        ]
    },

trtllm-build

toks=4096; maxb=16
trtllm-build --checkpoint_dir /trt_engines/saved_models_3b_Wf4Af8KVf8 --output_dir /trt_engines/tllm_3b_Wf4Af8KVf8--max_input_len $(($toks)) --max_seq_len $((2*$maxb*$toks)) --max_num_tokens $((2*$maxb*$toks)) --max_batch_size $maxb --gemm_plugin auto

Then, I got following error.

[2025-08-26 05:50:54] INFO config.py:54: PyTorch version 2.8.0a0+5228986c39.nv25.5 available.
[2025-08-26 05:50:54] INFO config.py:66: Polars version 1.25.2 available.
2025-08-26 05:51:00,721 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
[TensorRT-LLM] TensorRT-LLM version: 0.21.0
[08/26/2025-05:51:01] [TRT-LLM] [I] Set bert_attention_plugin to auto.
[08/26/2025-05:51:01] [TRT-LLM] [I] Set gpt_attention_plugin to auto.
[08/26/2025-05:51:01] [TRT-LLM] [I] Set gemm_plugin to auto.
[08/26/2025-05:51:01] [TRT-LLM] [I] Set gemm_swiglu_plugin to None.
[08/26/2025-05:51:01] [TRT-LLM] [I] Set fp8_rowwise_gemm_plugin to None.
[08/26/2025-05:51:01] [TRT-LLM] [I] Set nccl_plugin to auto.
[08/26/2025-05:51:01] [TRT-LLM] [I] Set lora_plugin to None.
[08/26/2025-05:51:01] [TRT-LLM] [I] Set dora_plugin to False.
[08/26/2025-05:51:01] [TRT-LLM] [I] Set moe_plugin to auto.
[08/26/2025-05:51:01] [TRT-LLM] [I] Set mamba_conv1d_plugin to auto.
[08/26/2025-05:51:01] [TRT-LLM] [I] Set low_latency_gemm_plugin to None.
[08/26/2025-05:51:01] [TRT-LLM] [I] Set low_latency_gemm_swiglu_plugin to None.
[08/26/2025-05:51:01] [TRT-LLM] [I] Set gemm_allreduce_plugin to None.
[08/26/2025-05:51:01] [TRT-LLM] [I] Set context_fmha to True.
[08/26/2025-05:51:01] [TRT-LLM] [I] Set bert_context_fmha_fp32_acc to False.
[08/26/2025-05:51:01] [TRT-LLM] [I] Set remove_input_padding to True.
[08/26/2025-05:51:01] [TRT-LLM] [I] Set norm_quant_fusion to False.
[08/26/2025-05:51:01] [TRT-LLM] [I] Set reduce_fusion to False.
[08/26/2025-05:51:01] [TRT-LLM] [I] Set user_buffer to False.
[08/26/2025-05:51:01] [TRT-LLM] [I] Set tokens_per_block to 32.
[08/26/2025-05:51:01] [TRT-LLM] [I] Set use_paged_context_fmha to True.
[08/26/2025-05:51:01] [TRT-LLM] [I] Set use_fp8_context_fmha to True.
[08/26/2025-05:51:01] [TRT-LLM] [I] Set fuse_fp4_quant to False.
[08/26/2025-05:51:01] [TRT-LLM] [I] Set multiple_profiles to False.
[08/26/2025-05:51:01] [TRT-LLM] [I] Set paged_state to True.
[08/26/2025-05:51:01] [TRT-LLM] [I] Set streamingllm to False.
[08/26/2025-05:51:01] [TRT-LLM] [I] Set use_fused_mlp to True.
[08/26/2025-05:51:01] [TRT-LLM] [I] Set pp_reduce_scatter to False.
[08/26/2025-05:51:01] [TRT-LLM] [W] Implicitly setting LLaMAConfig.producer = {'name': 'modelopt', 'version': '0.33.0'}
[08/26/2025-05:51:01] [TRT-LLM] [W] Implicitly setting LLaMAConfig.share_embedding_table = False
[08/26/2025-05:51:01] [TRT-LLM] [W] Implicitly setting LLaMAConfig.bias = False
[08/26/2025-05:51:01] [TRT-LLM] [W] Implicitly setting LLaMAConfig.rotary_pct = 1.0
[08/26/2025-05:51:01] [TRT-LLM] [W] Implicitly setting LLaMAConfig.rank = 0
[08/26/2025-05:51:01] [TRT-LLM] [W] Implicitly setting LLaMAConfig.decoder = llama
[08/26/2025-05:51:01] [TRT-LLM] [W] Implicitly setting LLaMAConfig.rmsnorm = True
[08/26/2025-05:51:01] [TRT-LLM] [W] Implicitly setting LLaMAConfig.lm_head_bias = False
[08/26/2025-05:51:01] [TRT-LLM] [W] Implicitly setting LLaMAConfig.fc_after_embed = False
[08/26/2025-05:51:01] [TRT-LLM] [W] Implicitly setting LLaMAConfig.use_input_layernorm_in_first_layer = True
[08/26/2025-05:51:01] [TRT-LLM] [W] Implicitly setting LLaMAConfig.use_last_layernorm = True
[08/26/2025-05:51:01] [TRT-LLM] [W] Implicitly setting LLaMAConfig.layer_idx_offset = 0
[08/26/2025-05:51:01] [TRT-LLM] [W] Implicitly setting LLaMAConfig.has_partial_lora_mask = False
[08/26/2025-05:51:01] [TRT-LLM] [W] Implicitly setting LLaMAConfig.tie_word_embeddings = True
[08/26/2025-05:51:01] [TRT-LLM] [I] Compute capability: (12, 0)
[08/26/2025-05:51:01] [TRT-LLM] [I] SM count: 188
[08/26/2025-05:51:01] [TRT-LLM] [I] SM clock: 3090 MHz
[08/26/2025-05:51:01] [TRT-LLM] [I] int4 TFLOPS: 0
[08/26/2025-05:51:01] [TRT-LLM] [I] int8 TFLOPS: 0
[08/26/2025-05:51:01] [TRT-LLM] [I] fp8 TFLOPS: 0
[08/26/2025-05:51:01] [TRT-LLM] [I] float16 TFLOPS: 0
[08/26/2025-05:51:01] [TRT-LLM] [I] bfloat16 TFLOPS: 0
[08/26/2025-05:51:01] [TRT-LLM] [I] float32 TFLOPS: 0
[08/26/2025-05:51:01] [TRT-LLM] [I] Total Memory: 95 GiB
[08/26/2025-05:51:01] [TRT-LLM] [I] Memory clock: 14001 MHz
[08/26/2025-05:51:01] [TRT-LLM] [I] Memory bus width: 512
[08/26/2025-05:51:01] [TRT-LLM] [I] Memory bandwidth: 1792 GB/s
[08/26/2025-05:51:01] [TRT-LLM] [I] PCIe speed: 2500 Mbps
[08/26/2025-05:51:01] [TRT-LLM] [I] PCIe link width: 16
[08/26/2025-05:51:01] [TRT-LLM] [I] PCIe bandwidth: 5 GB/s
[08/26/2025-05:51:01] [TRT-LLM] [W] Provided but not required tensors: {'transformer.layers.9.mlp.gate.weights_scaling_factor', 'transformer.layers.13.attention.qkv.weights_scaling_factor', 'transformer.layers.26.mlp.proj.weights_scaling_factor', 'transformer.layers.14.attention.qkv.weights_scaling_factor', 'transformer.layers.0.attention.qkv.activation_scaling_factor', 'transformer.layers.15.attention.qkv.activation_scaling_factor', 'transformer.layers.3.mlp.proj.activation_scaling_factor', 'transformer.layers.5.mlp.fc.weights_scaling_factor', 'transformer.layers.25.mlp.fc.weights_scaling_factor', 'transformer.layers.5.mlp.proj.activation_scaling_factor', 'transformer.layers.16.attention.dense.weights_scaling_factor', 'transformer.layers.10.mlp.proj.activation_scaling_factor', 'transformer.layers.15.mlp.gate.weights_scaling_factor', 'transformer.layers.12.attention.dense.weights_scaling_factor', 'transformer.layers.21.mlp.proj.activation_scaling_factor', 'transformer.layers.10.attention.qkv.weights_scaling_factor', 'transformer.layers.6.mlp.proj.weights_scaling_factor', 'transformer.layers.16.mlp.gate.activation_scaling_factor', 'transformer.layers.22.mlp.proj.activation_scaling_factor', 'transformer.layers.2.mlp.proj.weights_scaling_factor', 'transformer.layers.10.attention.dense.weights_scaling_factor', 'transformer.layers.20.mlp.fc.activation_scaling_factor', 'transformer.layers.23.mlp.fc.weights_scaling_factor', 'transformer.layers.1.attention.dense.weights_scaling_factor', 'transformer.layers.11.mlp.fc.activation_scaling_factor', 'transformer.layers.17.mlp.fc.weights_scaling_factor', 'transformer.layers.19.attention.qkv.activation_scaling_factor', 'transformer.layers.20.mlp.gate.weights_scaling_factor', 'transformer.layers.11.mlp.proj.activation_scaling_factor', 'transformer.layers.9.attention.qkv.weights_scaling_factor', 'transformer.layers.14.mlp.proj.activation_scaling_factor', 'transformer.layers.19.mlp.fc.activation_scaling_factor', 'transformer.layers.3.attention.qkv.weights_scaling_factor', 'transformer.layers.8.mlp.fc.weights_scaling_factor', 'transformer.layers.5.attention.qkv.activation_scaling_factor', 'transformer.layers.9.attention.qkv.activation_scaling_factor', 'transformer.layers.16.mlp.gate.weights_scaling_factor', 'transformer.layers.13.mlp.gate.weights_scaling_factor', 'transformer.layers.8.mlp.proj.weights_scaling_factor', 'transformer.layers.20.mlp.proj.activation_scaling_factor', 'transformer.layers.4.mlp.fc.weights_scaling_factor', 'transformer.layers.0.attention.dense.weights_scaling_factor', 'transformer.layers.13.attention.dense.weights_scaling_factor', 'transformer.layers.27.mlp.gate.weights_scaling_factor', 'transformer.layers.16.mlp.fc.activation_scaling_factor', 'transformer.layers.23.mlp.proj.weights_scaling_factor', 'transformer.layers.26.attention.dense.activation_scaling_factor', 'transformer.layers.25.attention.dense.activation_scaling_factor', 'transformer.layers.23.attention.dense.weights_scaling_factor', 'transformer.layers.2.attention.qkv.weights_scaling_factor', 'transformer.layers.24.attention.qkv.activation_scaling_factor', 'transformer.layers.16.attention.qkv.activation_scaling_factor', 'transformer.layers.12.mlp.fc.activation_scaling_factor', 'transformer.layers.19.attention.dense.weights_scaling_factor', 'transformer.layers.1.attention.qkv.activation_scaling_factor', 'transformer.layers.0.mlp.fc.weights_scaling_factor', 'transformer.layers.19.mlp.proj.activation_scaling_factor', 'transformer.layers.13.attention.qkv.activation_scaling_factor', 'transformer.layers.0.attention.qkv.weights_scaling_factor', 'transformer.layers.26.attention.dense.weights_scaling_factor', 'transformer.layers.19.mlp.gate.activation_scaling_factor', 'transformer.layers.22.attention.dense.weights_scaling_factor', 'transformer.layers.23.attention.qkv.weights_scaling_factor', 'transformer.layers.9.attention.dense.activation_scaling_factor', 'transformer.layers.26.attention.qkv.weights_scaling_factor', 'transformer.layers.23.mlp.gate.weights_scaling_factor', 'transformer.layers.14.mlp.fc.activation_scaling_factor', 'transformer.layers.13.mlp.fc.weights_scaling_factor', 'transformer.layers.6.attention.dense.weights_scaling_factor', 'transformer.layers.17.mlp.gate.activation_scaling_factor', 'transformer.layers.8.attention.qkv.weights_scaling_factor', 'transformer.layers.17.mlp.gate.weights_scaling_factor', 'transformer.layers.24.mlp.proj.activation_scaling_factor', 'transformer.layers.3.mlp.gate.activation_scaling_factor', 'transformer.layers.10.mlp.proj.weights_scaling_factor', 'transformer.layers.10.mlp.fc.weights_scaling_factor', 'transformer.layers.13.mlp.proj.activation_scaling_factor', 'transformer.layers.24.mlp.gate.weights_scaling_factor', 'transformer.layers.3.attention.qkv.activation_scaling_factor', 'transformer.layers.14.mlp.fc.weights_scaling_factor', 'transformer.layers.11.attention.dense.activation_scaling_factor', 'transformer.layers.27.attention.dense.activation_scaling_factor', 'transformer.layers.22.mlp.fc.weights_scaling_factor', 'transformer.layers.22.attention.qkv.activation_scaling_factor', 'transformer.layers.0.mlp.proj.weights_scaling_factor', 'transformer.layers.12.mlp.proj.weights_scaling_factor', 'transformer.layers.20.attention.dense.activation_scaling_factor', 'transformer.layers.13.mlp.fc.activation_scaling_factor', 'transformer.layers.15.mlp.proj.weights_scaling_factor', 'transformer.layers.19.mlp.proj.weights_scaling_factor', 'transformer.layers.5.mlp.gate.weights_scaling_factor', 'transformer.layers.17.attention.dense.weights_scaling_factor', 'transformer.layers.11.mlp.gate.weights_scaling_factor', 'transformer.layers.15.attention.dense.activation_scaling_factor', 'transformer.layers.11.mlp.proj.weights_scaling_factor', 'transformer.layers.14.attention.dense.activation_scaling_factor', 'transformer.layers.22.mlp.proj.weights_scaling_factor', 'transformer.layers.15.mlp.fc.weights_scaling_factor', 'transformer.layers.22.attention.dense.activation_scaling_factor', 'transformer.layers.12.attention.dense.activation_scaling_factor', 'transformer.layers.10.attention.qkv.activation_scaling_factor', 'transformer.layers.21.mlp.fc.activation_scaling_factor', 'transformer.layers.4.mlp.gate.weights_scaling_factor', 'transformer.layers.9.mlp.fc.activation_scaling_factor', 'transformer.layers.26.mlp.gate.weights_scaling_factor', 'transformer.layers.24.mlp.proj.weights_scaling_factor', 'transformer.layers.27.attention.qkv.activation_scaling_factor', 'transformer.layers.9.mlp.fc.weights_scaling_factor', 'transformer.layers.13.mlp.proj.weights_scaling_factor', 'transformer.layers.6.mlp.gate.activation_scaling_factor', 'transformer.layers.13.attention.dense.activation_scaling_factor', 'transformer.layers.24.mlp.gate.activation_scaling_factor', 'transformer.layers.27.mlp.proj.weights_scaling_factor', 'transformer.layers.6.mlp.fc.activation_scaling_factor', 'transformer.layers.26.mlp.proj.activation_scaling_factor', 'transformer.layers.20.mlp.gate.activation_scaling_factor', 'transformer.layers.8.mlp.fc.activation_scaling_factor', 'transformer.layers.8.attention.dense.activation_scaling_factor', 'transformer.layers.9.mlp.proj.activation_scaling_factor', 'transformer.layers.21.mlp.gate.weights_scaling_factor', 'transformer.layers.17.attention.dense.activation_scaling_factor', 'transformer.layers.3.attention.dense.weights_scaling_factor', 'transformer.layers.18.attention.dense.weights_scaling_factor', 'transformer.layers.3.mlp.fc.weights_scaling_factor', 'transformer.layers.24.attention.qkv.weights_scaling_factor', 'transformer.layers.4.mlp.proj.activation_scaling_factor', 'transformer.layers.25.mlp.proj.activation_scaling_factor', 'transformer.layers.18.mlp.gate.activation_scaling_factor', 'transformer.layers.16.attention.dense.activation_scaling_factor', 'transformer.layers.15.attention.qkv.weights_scaling_factor', 'transformer.layers.11.mlp.gate.activation_scaling_factor', 'transformer.layers.26.mlp.fc.weights_scaling_factor', 'transformer.layers.5.mlp.gate.activation_scaling_factor', 'transformer.layers.6.attention.qkv.weights_scaling_factor', 'transformer.layers.2.attention.qkv.activation_scaling_factor', 'transformer.layers.1.mlp.proj.weights_scaling_factor', 'transformer.layers.12.mlp.proj.activation_scaling_factor', 'transformer.layers.26.attention.qkv.activation_scaling_factor', 'transformer.layers.1.attention.dense.activation_scaling_factor', 'transformer.layers.11.mlp.fc.weights_scaling_factor', 'transformer.layers.2.mlp.fc.weights_scaling_factor', 'transformer.layers.4.attention.qkv.weights_scaling_factor', 'transformer.layers.0.mlp.gate.activation_scaling_factor', 'transformer.layers.12.mlp.gate.weights_scaling_factor', 'transformer.layers.18.mlp.proj.weights_scaling_factor', 'transformer.layers.24.mlp.fc.weights_scaling_factor', 'transformer.layers.18.mlp.fc.weights_scaling_factor', 'transformer.layers.20.attention.dense.weights_scaling_factor', 'transformer.layers.3.attention.dense.activation_scaling_factor', 'transformer.layers.6.mlp.gate.weights_scaling_factor', 'transformer.layers.21.mlp.proj.weights_scaling_factor', 'transformer.layers.18.mlp.fc.activation_scaling_factor', 'transformer.layers.19.mlp.gate.weights_scaling_factor', 'transformer.layers.16.mlp.fc.weights_scaling_factor', 'transformer.layers.18.attention.dense.activation_scaling_factor', 'transformer.layers.6.attention.qkv.activation_scaling_factor', 'transformer.layers.27.mlp.proj.activation_scaling_factor', 'transformer.layers.23.attention.dense.activation_scaling_factor', 'transformer.layers.21.attention.dense.weights_scaling_factor', 'transformer.layers.23.attention.qkv.activation_scaling_factor', 'transformer.layers.6.attention.dense.activation_scaling_factor', 'transformer.layers.25.attention.qkv.activation_scaling_factor', 'transformer.layers.12.mlp.gate.activation_scaling_factor', 'transformer.layers.16.mlp.proj.weights_scaling_factor', 'transformer.layers.5.attention.qkv.weights_scaling_factor', 'transformer.layers.18.attention.qkv.activation_scaling_factor', 'transformer.layers.20.attention.qkv.weights_scaling_factor', 'transformer.layers.17.mlp.fc.activation_scaling_factor', 'transformer.layers.7.mlp.gate.weights_scaling_factor', 'transformer.layers.17.mlp.proj.weights_scaling_factor', 'transformer.layers.27.mlp.fc.activation_scaling_factor', 'transformer.layers.25.mlp.gate.activation_scaling_factor', 'transformer.layers.17.mlp.proj.activation_scaling_factor', 'transformer.layers.22.mlp.gate.weights_scaling_factor', 'transformer.layers.27.attention.qkv.weights_scaling_factor', 'transformer.layers.20.mlp.fc.weights_scaling_factor', 'transformer.layers.20.mlp.proj.weights_scaling_factor', 'transformer.layers.2.mlp.gate.activation_scaling_factor', 'transformer.layers.15.mlp.fc.activation_scaling_factor', 'transformer.layers.7.attention.qkv.weights_scaling_factor', 'transformer.layers.8.attention.dense.weights_scaling_factor', 'transformer.layers.21.mlp.gate.activation_scaling_factor', 'transformer.layers.18.mlp.proj.activation_scaling_factor', 'transformer.layers.11.attention.qkv.weights_scaling_factor', 'transformer.layers.10.mlp.gate.weights_scaling_factor', 'transformer.layers.17.attention.qkv.activation_scaling_factor', 'transformer.layers.24.attention.dense.activation_scaling_factor', 'transformer.layers.27.mlp.fc.weights_scaling_factor', 'transformer.layers.1.mlp.gate.weights_scaling_factor', 'transformer.layers.4.attention.qkv.activation_scaling_factor', 'transformer.layers.7.attention.dense.weights_scaling_factor', 'transformer.layers.8.mlp.gate.weights_scaling_factor', 'transformer.layers.8.attention.qkv.activation_scaling_factor', 'transformer.layers.11.attention.qkv.activation_scaling_factor', 'transformer.layers.7.mlp.fc.weights_scaling_factor', 'transformer.layers.7.attention.qkv.activation_scaling_factor', 'transformer.layers.12.mlp.fc.weights_scaling_factor', 'transformer.layers.25.mlp.gate.weights_scaling_factor', 'transformer.layers.15.mlp.proj.activation_scaling_factor', 'transformer.layers.4.mlp.proj.weights_scaling_factor', 'transformer.layers.5.attention.dense.activation_scaling_factor', 'transformer.layers.11.attention.dense.weights_scaling_factor', 'transformer.layers.1.mlp.fc.activation_scaling_factor', 'transformer.layers.12.attention.qkv.activation_scaling_factor', 'transformer.layers.18.mlp.gate.weights_scaling_factor', 'transformer.layers.14.attention.qkv.activation_scaling_factor', 'transformer.layers.6.mlp.fc.weights_scaling_factor', 'transformer.layers.24.mlp.fc.activation_scaling_factor', 'transformer.layers.15.mlp.gate.activation_scaling_factor', 'transformer.layers.25.mlp.fc.activation_scaling_factor', 'transformer.layers.19.mlp.fc.weights_scaling_factor', 'transformer.layers.23.mlp.proj.activation_scaling_factor', 'transformer.layers.2.mlp.proj.activation_scaling_factor', 'transformer.layers.5.mlp.proj.weights_scaling_factor', 'transformer.layers.18.attention.qkv.weights_scaling_factor', 'transformer.layers.2.mlp.gate.weights_scaling_factor', 'transformer.layers.2.mlp.fc.activation_scaling_factor', 'transformer.layers.3.mlp.fc.activation_scaling_factor', 'transformer.layers.10.mlp.gate.activation_scaling_factor', 'transformer.layers.25.mlp.proj.weights_scaling_factor', 'transformer.layers.21.mlp.fc.weights_scaling_factor', 'transformer.layers.16.mlp.proj.activation_scaling_factor', 'transformer.layers.4.mlp.gate.activation_scaling_factor', 'transformer.layers.4.attention.dense.activation_scaling_factor', 'transformer.layers.21.attention.qkv.activation_scaling_factor', 'transformer.layers.9.mlp.gate.activation_scaling_factor', 'transformer.layers.3.mlp.proj.weights_scaling_factor', 'transformer.layers.14.mlp.gate.weights_scaling_factor', 'transformer.layers.9.mlp.proj.weights_scaling_factor', 'transformer.layers.27.mlp.gate.activation_scaling_factor', 'transformer.layers.7.mlp.proj.activation_scaling_factor', 'transformer.layers.10.attention.dense.activation_scaling_factor', 'transformer.layers.13.mlp.gate.activation_scaling_factor', 'transformer.layers.23.mlp.gate.activation_scaling_factor', 'transformer.layers.14.mlp.gate.activation_scaling_factor', 'transformer.layers.22.mlp.fc.activation_scaling_factor', 'transformer.layers.5.mlp.fc.activation_scaling_factor', 'transformer.layers.1.mlp.gate.activation_scaling_factor', 'transformer.layers.12.attention.qkv.weights_scaling_factor', 'transformer.layers.10.mlp.fc.activation_scaling_factor', 'transformer.layers.19.attention.dense.activation_scaling_factor', 'transformer.layers.1.mlp.fc.weights_scaling_factor', 'transformer.layers.16.attention.qkv.weights_scaling_factor', 'transformer.layers.21.attention.dense.activation_scaling_factor', 'transformer.layers.7.mlp.fc.activation_scaling_factor', 'transformer.layers.8.mlp.gate.activation_scaling_factor', 'transformer.layers.14.mlp.proj.weights_scaling_factor', 'transformer.layers.19.attention.qkv.weights_scaling_factor', 'transformer.layers.4.mlp.fc.activation_scaling_factor', 'transformer.layers.21.attention.qkv.weights_scaling_factor', 'transformer.layers.22.attention.qkv.weights_scaling_factor', 'transformer.layers.14.attention.dense.weights_scaling_factor', 'transformer.layers.24.attention.dense.weights_scaling_factor', 'transformer.layers.7.attention.dense.activation_scaling_factor', 'transformer.layers.3.mlp.gate.weights_scaling_factor', 'transformer.layers.5.attention.dense.weights_scaling_factor', 'transformer.layers.22.mlp.gate.activation_scaling_factor', 'transformer.layers.8.mlp.proj.activation_scaling_factor', 'transformer.layers.0.mlp.fc.activation_scaling_factor', 'transformer.layers.15.attention.dense.weights_scaling_factor', 'transformer.layers.1.attention.qkv.weights_scaling_factor', 'transformer.layers.2.attention.dense.activation_scaling_factor', 'transformer.layers.4.attention.dense.weights_scaling_factor', 'transformer.layers.27.attention.dense.weights_scaling_factor', 'transformer.layers.17.attention.qkv.weights_scaling_factor', 'transformer.layers.23.mlp.fc.activation_scaling_factor', 'transformer.layers.9.attention.dense.weights_scaling_factor', 'transformer.layers.7.mlp.proj.weights_scaling_factor', 'transformer.layers.2.attention.dense.weights_scaling_factor', 'transformer.layers.20.attention.qkv.activation_scaling_factor', 'transformer.layers.0.mlp.gate.weights_scaling_factor', 'transformer.layers.26.mlp.gate.activation_scaling_factor', 'transformer.layers.0.mlp.proj.activation_scaling_factor', 'transformer.layers.25.attention.qkv.weights_scaling_factor', 'transformer.layers.25.attention.dense.weights_scaling_factor', 'transformer.layers.0.attention.dense.activation_scaling_factor', 'transformer.layers.6.mlp.proj.activation_scaling_factor', 'transformer.layers.26.mlp.fc.activation_scaling_factor', 'transformer.layers.7.mlp.gate.activation_scaling_factor', 'transformer.layers.1.mlp.proj.activation_scaling_factor'}
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/modeling_utils.py", line 786, in load
    param.value = weights[name]
    ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/parameter.py", line 222, in value
    assert v.shape == self.shape, \
           ^^^^^^^^^^^^^^^^^^^^^
AssertionError: The value updated is not the same shape as the original. Updated: (5120, 1536), original: (5120, 3072)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/trtllm-build", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/commands/build.py", line 626, in main
    parallel_build(model_config, ckpt_dir, build_config, args.output_dir,
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/commands/build.py", line 419, in parallel_build
    passed = build_and_save(rank, rank % workers, ckpt_dir,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/commands/build.py", line 384, in build_and_save
    engine = build_model(build_config,
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/commands/build.py", line 361, in build_model
    model = model_cls.from_checkpoint(ckpt_dir, config=rank_config)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/modeling_utils.py", line 751, in from_checkpoint
    model.load(weights, from_pruned=is_checkpoint_pruned)
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/models/modeling_utils.py", line 788, in load
    raise RuntimeError(
RuntimeError: Encounter error 'The value updated is not the same shape as the original. Updated: (5120, 1536), original: (5120, 3072)' for parameter 'transformer.layers.0.attention.qkv.weight'

@karljang Any advice would be applicated.

Issue has not received an update in over 14 days. Adding stale label.

v1.1.0rc4 still have this issue, although it said Add NVFP4 x FP8 in their feature. We tested on following environment.

RTX PRO 6000
tensorrt_llm 1.1.0rc4
modelopt 35.0

I successfully built nvfp4 engine of unified HF using v1.2.0rc4, however execution of the engine is not supported yet.

@youki-sada

@youki-sada , I apologize for the delayed response. I’m trying to understand the current status.

If I’ve understood correctly, you’ve built an engine using NVFP4 x FP8, but it’s not serving because something is not supported yet.

However, the #6809 PR implements NVFP4 x FP8 for the PyTorch backend, not the TensorRT backend.
Therefore, if you were using the old TensorRT workflow, it makes sense that the engine’s execution isn’t supported.

Would you be able to try the PyTorch workflow instead?
Since the 1.0 release, TensorRT LLM has transitioned to the PyTorch workflow, which no longer relies on trtllm-build engines. The team is currently focused on developing this workflow.

Thank you!

@karljang

@karljang Thank you for your reply. Yes, I've already tried the Pytorch backend.

RTX PRO 6000 Max-Q
TensorRT-LLM v0.12.0rc5

>>> from tensorrt_llm import LLM
>>> model=LLM(model='/trt_engines/HF_3b_w4a8_nvfp4_fp8')
[12/11/2025-07:17:58] [TRT-LLM] [I] Using LLM with PyTorch backend
[12/11/2025-07:17:58] [TRT-LLM] [W] Using default gpus_per_node: 1
[12/11/2025-07:17:58] [TRT-LLM] [I] neither checkpoint_format nor checkpoint_loader were provided, checkpoint_format will be set to HF.
[TensorRT-LLM] TensorRT LLM version: 1.2.0rc5
[TensorRT-LLM][INFO] Refreshed the MPI local session
Loading safetensors weights in parallel: 100%|█████████████████████████████████| 1/1 [00:00<00:00, 88.71it/s]
Loading weights: 100%|████████████████████████████████████████████| 596/596 [00:03<00:00, 193.18it/s]
Model init total -- 4.37s
[TensorRT-LLM][INFO] Max KV cache blocks per sequence: 4097 [window size=8224], tokens per block=32, primary blocks=257, secondary blocks=0, max sequence length=131073
[TensorRT-LLM][INFO] Number of tokens per block: 32.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 0.44 GiB for max tokens in paged KV cache (8224).
[12/11/2025-07:18:28] [TRT-LLM] [E] Failed to initialize executor on rank 0: [TensorRT-LLM][ERROR] Assertion failed: Error occurred when running GEMM! (../tensorrt_llm/kernels/trtllmGenKernels/gemm/KernelRunner.cpp:143)
1       0x70bb6509eb89 tensorrt_llm::common::throwRuntimeError(char const*, int, char const*) + 97
2       0x70bb2c8a8265 tensorrt_llm::kernels::TrtllmGenGemmRunner::run(int, int, int, void const*, float const*, void const*, float const*, void*, float*, float*, void*, CUstream_st*, int) + 677
3       0x70bb6565a0fb /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libth_common.so(+0xc4b0fb) [0x70bb6565a0fb

@QiJune

Hi @QiJune, could you please take a look at the error log above and assist us or assign someone to handle it?
Thank you!

@youki-sada

@youki-sada ,
Could you please share the command you used to generate the checkpoint '/trt_engines/HF_3b_w4a8_nvfp4_fp8', so that we can easily reproduce the issue?

python llm_ptq/hf_ptq.py --pyt_ckpt_path=meta-llama/Llama-3.2-3B-Instruct --export_path=/trt_engines/HF_3b_w4a8_nvfp4_fp8 --sparsity_fmt=dense --qformat=w4a8_nvfp4_fp8 --calib_size=32 --batch_size=1 --inference_tensor_parallel=1 --inference_pipeline_parallel=1

I used nvidia-modelopt==0.39.0.

Issue has not received an update in over 14 days. Adding stale label.

This issue was closed because it has been 14 days without activity since it has been marked as stale.

Navigation Menu

Repository navigation

w4a8_mxfp4_fp8 build failure #6680

hf_ptq.py

trtllm-build

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	if not from_pruned:
	try:
	param.value = weights[name]
	except Exception as e:
	raise RuntimeError(
	f"Encounter error '{e}' for parameter '{name}'")

w4a8_mxfp4_fp8 build failure #6680

Description

Activity

Shixiaowei02 commented on Aug 11, 2025

karljang commented on Aug 25, 2025

youki-sada commented on Aug 26, 2025

hf_ptq.py

trtllm-build

github-actions commented on Sep 9, 2025

youki-sada commented on Sep 11, 2025

youki-sada commented on Dec 11, 2025

karljang commented on Dec 11, 2025

youki-sada commented on Dec 11, 2025

karljang commented on Dec 11, 2025

karljang commented on Dec 11, 2025

youki-sada commented on Dec 11, 2025

github-actions commented on Dec 26, 2025

github-actions commented on Jan 9, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions