Repository navigation
Conversation
…ckpoints Accept a static NVFP4 or per-tensor FP8 quantization_config in the MiniMax-H3 transformer so a ModelOpt export loads as exported: quantized weights, scales, calibrated input_scale and the ignore list. Split only the row-indexed ff.net.0.proj tensors (weight, bias, a 2-D weight_scale) into gate/up and share the per-tensor scalars and input-side tensors. Cast each media projection's input from its own quant state instead of the global algorithm so excluded FP32 projections keep FP32 inputs. Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>
- Test helper passed quant_config twice; it now takes the override directly. - A one-row ff.net.0.proj tensor (a [1]-shaped per-tensor FP8 weight_scale) is shared between gate and up instead of being split. - Checkpoint-backed test: a ModelOpt-style NVFP4 and FP8 export of the tiny model goes through load_weights and one forward on the static path. Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>
_TINY_CONFIG has no transformer blocks, so transformer_blocks[0] raised IndexError on the GPU run. Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>
Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>
…d name the first non-finite Linear Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>
Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>
Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>
…tic-checkpoint Linear amplitudes The media-projection input cast reads has_any_quant on every projection, so the recording stub in the dynamic-FP8 input-dtype test needs the attribute. The static-checkpoint forward hooks now record each quantized Linear's output amplitude and the test rejects implausibly large values alongside non-finite ones, so a GEMM that returns garbage of finite magnitude is named too. Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>
…exporting it TRT-LLM Linears allocate their parameters with torch.empty, so the BF16 reference model's non-quantized tensors, the FP32 media projections among them, carried uninitialised memory into the synthetic checkpoint. The test now gives the reference the deterministic golden initialisation used by the Diffusers parity tests. Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>
…in case With the reference model initialised, the FP8 forward is finite on SM 107 as well, so the expected failure no longer applies. Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>
…GLU producers The static-NVFP4 block wrote three BF16 activations that the next Linear re-read only to quantize them. The AdaLN RMSNorm + modulation kernels and the SiLU-and-mul kernel now emit the packed E2M1 codes and swizzled E4M3 block scales directly, with the same rounding points as quantize_with_block_size, and hand them to the Linear as Fp4QuantizedTensor. H3_FUSED_QUANT=0 keeps the unfused path. Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>
…m a second table A block that folds its MLP residual into the next block's norm1 kernel reads the gate from its own AdaLN table while the scale and shift come from the next block's. Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>
… bandwidth-bound kernel The static-NVFP4 to_out Linear re-reads the BF16 attention output through quantize_with_block_size, one row per block with one 16-byte load per thread, which runs at a fifth of the HBM bandwidth. Quantize it instead with a tiled Triton kernel that reproduces fp4_quantize byte for byte (codes and swizzled scales) and hand the Linear the Fp4QuantizedTensor. H3_OUT_QUANT=0 keeps the Linear's own quantize. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Kanghwan Jang <kanghwanj@users.noreply.github.com> Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>
…aunch shape 16 rows x 512 columns per program with 4 warps was the fastest of the sweep at 19k, 38k and 104k rows (measured internally on SM 107; a clear win over the stock kernel at every shape). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Kanghwan Jang <kanghwanj@users.noreply.github.com> Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>
…antize Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>
Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>
karljang
force-pushed
the
feat/minimax-h3-fused-nvfp4-quant
branch
from
November 5, 2026 20:33
814cf6c to
19235a6
Compare
karljang
added a commit
to karljang/TensorRT-LLM
that referenced
this pull request
Nov 6, 2026
…DIA#21275 NVIDIA#19738 NVIDIA#21485 NVIDIA#21483 NVIDIA#21484 One branch that is the MiniMax-H3 recipe for SM 107, so it can be checked out and built instead of composed from per-lever patches at job time. Base: NVIDIA/TensorRT-LLM main aea80a0 (2026-11-06). The PR branches are merged in this order, every original commit kept: NVIDIA#21089 feat/minimax-h3-static-nvfp4-load static NVFP4 / FP8 checkpoint loader NVIDIA#21275 feat/minimax-h3-fa4-2cta-default FA4 2-CTA default on SM 103 / SM 107 NVIDIA#19738 wu6u3tw:fix/cudnn-fp8-attn-sm107 cuDNN FP8 attention on SM 107, LSE layout NVIDIA#21485 feat/minimax-h3-cudnn-fp8-ulysses cuDNN padding mask, fused FP8 QK cast, prefix output in place, async-Ulysses FP8 pairing NVIDIA#21483 feat/minimax-h3-fused-nvfp4-quant producer quantize fusion, residual fold, attention-output quantize NVIDIA#21484 feat/minimax-h3-vae-decoder-fp8-tiles schedule, FP8 tiled VAE decoder This commit applies the pre-commit fixes on the merged tree (ruff import order in pipeline_minimax_h3.py, ruff-format of vae_decoder_fp8.py), drops a measured kernel share from the vae_decoder_fp8 docstring, and routes the fused FP8 QK attention path's output through _out_proj: NVIDIA#21485 added that return after NVIDIA#21483 was cut, so the merge left it calling to_out directly and the attention-output quantize of NVIDIA#21483 never ran on the cuDNN FP8 row. How to turn the recipe on (NVFP4 checkpoint, request flags --attention-backend CUDNN --qk-dtype fp8 --v-dtype fp8): # the composed arm, every layer on (U1 and U4; drop H3_CUDNN_PREFIX_INPLACE only if the shape divides by U) export H3_FUSED_QUANT=1 H3_FUSED_QUANT_WARPS=4 H3_LEVER10_FOLD=1 export H3_FP8_SCALE_FUSION=dynamic H3_FP8_FUSION_SPLIT=1 export H3_OUT_QUANT=1 H3_CUDNN_PREFIX_INPLACE=1 export H3_L14_SCHED=1 H3_L14_VAE_FP8=1 H3_L14_VAE_TILE=288x272x32 # cuDNN FP8 attention: apply PR NVIDIA#19738 (it allows FP8 on SM 107 and fixes the LSE layout) to backends/cudnn.py first; # the measurements carried it as a shim ahead on PYTHONPATH, switched by H3_UNGATE_CUDNN=1; that shim is not in this folder # do NOT set H3_FP8_FUSION_BLOCK_H / H3_FP8_FUSION_WARPS: the module default (4 / 2) is the tuned launch # attention selection on the request: --attention-backend CUDNN --qk-dtype fp8 --v-dtype fp8 # U8 at 104k tokens: static FP8 scales instead of dynamic (table of 12 held-out prompts, in this folder) # (H3_FP8_STATIC_MARGIN defaults to 1.0) export H3_FP8_SCALE_FUSION=static H3_FP8_STATIC_SCALES=patches/fp8-static-scales-heldout.json # U8 at 19k tokens: all ten patches applied, step fusions off export H3_FUSED_QUANT=0 H3_OUT_QUANT=0; unset H3_LEVER10_FOLD H3_FP8_SCALE_FUSION H3_FP8_FUSION_SPLIT # fallback row (BF16 FA4 2-CTA): skip the NVIDIA#19738 step, unset H3_FP8_SCALE_FUSION H3_FP8_FUSION_SPLIT H3_CUDNN_PREFIX_INPLACE, # request FA4 and let the 2-CTA default of patch 2 apply (section 6 for the worker spawn) # the FP8 option (section 2.2): point the model at the static FP8 export (composite over the BF16 checkpoint), # same ten patches and the NVIDIA#19738 step; the NVFP4 fusions off, the static scale table never export H3_FUSED_QUANT=0 H3_OUT_QUANT=0; unset H3_LEVER10_FOLD export H3_FP8_SCALE_FUSION=dynamic H3_FP8_FUSION_SPLIT=1 H3_CUDNN_PREFIX_INPLACE=1 export H3_L14_SCHED=1 H3_L14_VAE_FP8=1 H3_L14_VAE_TILE=288x272x32 # request: --attention-backend CUDNN --qk-dtype fp8 --v-dtype fp8 The block above is the recipe report's "how to turn it on" verbatim; on this branch three of its lines are already satisfied by the code: NVIDIA#19738 is merged (no shim, no H3_UNGATE_CUDNN), H3_LEVER10_FOLD has no switch (the residual fold is part of H3_FUSED_QUANT, which defaults to on, as does H3_OUT_QUANT), and "patch 2" is the FA4 2-CTA default of NVIDIA#21275. The package ships one static FP8 descale table (calibrated at 544x960x124) under tensorrt_llm/_torch/visual_gen/models/minimax_h3/fp8_scale_tables/; the held-out table the report names for the U8 104k row stays beside the report. Signed-off-by: Kanghwan Jang <kanghwanj@nvidia.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Collaborator
Author
|
Closing this draft: the work continues on the integration branch |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fuse the NVFP4 activation quantize of MiniMax-H3 into the kernels that produce the activations, so the four
quantize_with_block_sizelaunches per transformer block disappear:fused_quant.py, Triton);H3_LEVER10_FOLD=1);fused_out_quant.py) instead of the stock launch.Every kernel reproduces the stock quantize bit for bit (video and audio outputs identical on 12 ship prompts at three shapes and at Ulysses degree 4). The fusions apply only to Linears loaded with static NVFP4 scales; a Linear carrying a pre-quant scale falls back to the unfused path.
Measured internally on SM 107 with the static NVFP4 checkpoint and cuDNN FP8 attention, one GPU, 544×960×124 frames: the producer quantize is the largest of the three gains, the residual fold is within noise, the attention-output quantize a small gain; timings are not published for this part. Switches:
H3_FUSED_QUANT(default on),H3_FUSED_QUANT_WARPS(4),H3_LEVER10_FOLD(off),H3_OUT_QUANT(default on).Stacked on #21089 (the static NVFP4 / FP8 checkpoint loader): review only the last five commits. Opened as a draft while the loader is in review; the env switches will become config fields before this leaves draft.
Test Coverage
tests/unittest/_torch/visual_gen/test_minimax_h3_static_checkpoint.pyfrom [None][feat] Load pre-quantized ModelOpt NVFP4 and FP8 MiniMax-H3 checkpoints #21089 covers the loader the fusions depend on.fused_quantandfused_out_quantagainstquantize_with_block_sizewill be added before the PR leaves draft.PR Checklist
🤖 Generated with Claude Code