Repository navigation
[Bug] ft checkpoint fails on Qwen3-Coder-30B-A3B modelopt NVFP4: qwen3_moe loader lacks NVFP4 expert-source wiring (0-dim scale scalars crash shard_tensor) #425
Description
Environment
- Windows 11, single RTX 5090 (32GB, sm_120)
- FreeToken v0.1.2+g141c31a8d, torch 2.11 / CUDA 13
Goal
Serve Qwen3-Coder-30B-A3B-Instruct (arch Qwen3MoeForCausalLM) from a native modelopt NVFP4 checkpoint via the proven FTW route (ft checkpoint → ft serve, the same path that works for the Qwen3.6-35B-A3B qwen3_5_moe model on this box).
Source tried: NVFP4/Qwen3-Coder-30B-A3B-Instruct-FP4 (modelopt, producer confirmed in hf_quant_config.json).
Repro
ft checkpoint --model <modelopt-NVFP4-dir> --out <out> --dtype bfloat16 --moe-backend offload --gpu 0
Failure
File "freetoken/models/loader.py", line 122, in iter_merged_tensors
File "freetoken/models/qwen3_moe/weight.py", line 57, in sharded_tensors
tensor = shard_tensor(...)
RuntimeError: chunk expects at least a 1-dimensional tensor
(crashes inside the compiled loader.shard_tensor when fed a 0-dim tensor)
Root cause (diagnosed)
qwen3_moe (registered for Qwen3MoeForCausalLM, added in this build) is a generic dense-MoE loader. Its generic experts.<idx>.<name> pattern feeds every per-expert tensor — including the modelopt quant 0-dim scale scalars input_scale and weight_scale_2 — through iter_stacked_experts → shard_tensor, which cannot chunk() a scalar.
By contrast every other modelopt-NVFP4 model here (gemma4, glm5_next, minimax_m2/m3, qwen4_exp) declares an Nvfp4ExpertSourceSpec and routes routed-experts to the offload-bank loader (load_moe_expert_sources), skipping input_scale and carrying weight_scale_2 as the per-row global scale (see glm5_next/weight.py kind_map). qwen3_moe has none of that wiring (only a comment referencing load_moe_expert_sources), so it was evidently validated on bf16 only and cannot ingest a modelopt-NVFP4 checkpoint.
Per-expert tensor shapes from the source:
[] down_proj.input_scale # 0-dim scalar
[2048,384] down_proj.weight
[2048,48] down_proj.weight_scale
[] down_proj.weight_scale_2 # 0-dim scalar
Ask
Wire NVFP4 expert-source support into qwen3_moe — qwen4_exp is the closest template (identical gate_proj/up_proj/down_proj + weight_scale/weight_scale_2/input_scale layout, and it also handles the shared_expert). Or confirm whether Qwen3MoeForCausalLM (Qwen3-Coder-30B-A3B / Qwen3-30B-A3B) NVFP4 is on the roadmap and which source layout it should take.
Happy to help test on sm_120/Windows once a nightly carries it.
Activity
ljrirene commentedon Sep 17, 2026
Same gap on a much bigger export —
nvidia/Qwen3-235B-A22B-Instruct-2507-NVFP4(94 layers, 128 experts, top-8) — so it's not specific to the 30B Coder checkpoint. Dropping the layout here in case it's useful.Two loader bugs fire before you can even reach this one; patches for both are in #496. Worth having those first, otherwise loading dies at 0% and never reaches
load_state_dict. One of them is thechunk expects at least a 1-dimensional tensoryou already hit here — your diagnosis is right that the expert scalars shouldn't be going throughshard_tensorat all, but note the attention-side scalars (o_proj.input_scale,o_proj.weight_scale_2) aren't expert tensors, so they'd keep crashing on theSPLIT_DIM_1arm even after the expert-source wiring lands. Cheap to guard separately.What the failure looks like once you're past them (with the bare assert in
python/freetoken/layers/base.pymade informative):Why. Mixed-precision export.
hf_quant_config.jsoncarriesquant_algo: NVFP4,kv_cache_quant_algo: FP8,group_size: 16, and anexclude_moduleslist that keepslm_head, every layer'sq_proj/k_proj/v_projand the MoE routergatein BF16 — whileo_projand all routed experts are NVFP4-packed. Layer-0 inventory straight from the safetensors:The qwen3_moe model definition declares every parameter full-width BF16 (
o_projexpects(4096, 8192)), so the mismatch is structural — it needs quant-aware module construction driven byhf_quant_config.json: packed-weight params, scale params, and honouringexclude_modules. Presumably along the lines of #438 for qwen3_5_moe and #426 for qwen4_exp, as you say.Environment: single RTX PRO 6000 Blackwell Max-Q 96 GB, driver 595.58.03, CUDA 13.0, torch 2.11.0+cu130, dual-socket EPYC / 1.1 TiB RAM, FreeToken 0.1.2, hybrid MoE backend. Happy to test a fix on the 235B export.