[None][feat] Mistral Large 4: carry feat/mistral-large-4 onto main

3rdparty/fetch_content.json

@@ -62,7 +62,8 @@
       "git_repository": "https://github.com/deepseek-ai/FlashMLA.git",
       "git_tag": "15f13e5030374295491c5ce31b02d7e63a7772c6",
       "git_submodules_recurse": true,
-      "source_subdir": "dont-add-this-project-with-add-subdirectory"
+      "source_subdir": "dont-add-this-project-with-add-subdirectory",
+      "patch_file": "patches/flashmla_release_gil.patch"
     },
     {
       "name": "fmt",

3rdparty/patches/flashmla_release_gil.patch

@@ -0,0 +1,17 @@
+--- a/csrc/api/api.cpp
++++ b/csrc/api/api.cpp
+@@ -7,9 +7,9 @@
+
+ PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
+     m.doc() = "FlashMLA";
+-    m.def("sparse_decode_fwd", &sparse_attn_decode_interface);
+-    m.def("dense_decode_fwd", &dense_attn_decode_interface);
+-    m.def("sparse_prefill_fwd", &sparse_attn_prefill_interface);
+-    m.def("dense_prefill_fwd", &FMHACutlassSM100FwdRun);
+-    m.def("dense_prefill_bwd", &FMHACutlassSM100BwdRun);
++    m.def("sparse_decode_fwd", &sparse_attn_decode_interface, pybind11::call_guard<pybind11::gil_scoped_release>());
++    m.def("dense_decode_fwd", &dense_attn_decode_interface, pybind11::call_guard<pybind11::gil_scoped_release>());
++    m.def("sparse_prefill_fwd", &sparse_attn_prefill_interface, pybind11::call_guard<pybind11::gil_scoped_release>());
++    m.def("dense_prefill_fwd", &FMHACutlassSM100FwdRun, pybind11::call_guard<pybind11::gil_scoped_release>());
++    m.def("dense_prefill_bwd", &FMHACutlassSM100BwdRun, pybind11::call_guard<pybind11::gil_scoped_release>());
+ }

docs/source/deployment-guide/index.rst

@@ -39,3 +39,4 @@ The deployment guides below provide more detailed instructions for serving speci
    deployment-guide-for-glm-5-on-trtllm.md
    deployment-guide-for-glm-5.3-flash-on-trtllm.md
    deployment-guide-for-minimax-m3-on-trtllm.md
+   deployment-guide-for-mistral-large-4-on-trtllm.md

docs/source/models/supported-models.md

@@ -38,6 +38,7 @@ The following is a table of supported models for the PyTorch backend:
 | `MiniCPMV4_6ForConditionalGeneration` [^14]| MiniCPM-V 4.6                    | `openbmb/MiniCPM-V-4.6`                      |
 | `MiniMaxM3SparseForConditionalGeneration` [^12]| MiniMax-M3                       | `MiniMaxAI/MiniMax-M3`                      |
 | `MistralForCausalLM`                 | Mistral                            | `mistralai/Mistral-7B-v0.1`                  |
+| `MistralLarge4ForCausalLM` [^24]     | Mistral Large 4                    | `mistralai/Mistral-Large-4-1T-A52B-NVFP4`    |
 | `MixtralForCausalLM`                 | Mixtral                            | `mistralai/Mixtral-8x7B-v0.1`                |
 | `MllamaForConditionalGeneration`     | Llama 3.2                          | `meta-llama/Llama-3.2-11B-Vision`            |
 | `NemotronForCausalLM`                | Nemotron-3, Nemotron-4, Minitron   | `nvidia/Minitron-8B-Base`                    |
@@ -107,6 +108,7 @@ statuses for the same architecture in the two matrices.
 | `Gemma4UnifiedForConditionalGeneration` | Yes                | Yes        | Yes                        | No                    | Yes             | No                         | Yes           | No             | Yes                      | Yes                   | Yes             |
 | `Step3p7ForConditionalGeneration`| Yes               | Yes        | Yes                        | Untested              | Untested        | MTP                        | Yes           | No             | Yes                      | Yes                   | Yes             |
 | `MiniMaxM3SparseForConditionalGeneration` [^12] | Yes               | Yes        | Yes                        | Untested              | Untested        | EAGLE-3 (Linear), DSpark [^21] | Yes           | No             | N/A                      | Yes                   | Yes             |
+| `MistralLarge4ForCausalLM` [^24] | Yes               | Yes        | Yes                        | Yes [^25]             | Yes             | DSpark                     | Yes           | Yes            | N/A                      | Untested              | Yes             |
 
 [^1]: Chunked Prefill for MLA can only be enabled on SM90/SM100/SM103/SM107/SM120.
 [^2]: KV cache reuse for MLA can only be enabled on SM90/SM100/SM103/SM107/SM120/SM121 and in BF16/FP8 KV cache dtype.
@@ -127,6 +129,8 @@ statuses for the same architecture in the two matrices.
 [^21]: MiniMax-M3.1-Flash ships a standalone DSpark drafter in its `dspark/` directory (architecture `DSparkMiniMaxDraftModel`). Load it with `speculative_config: {decoding_type: DSpark, speculative_model: <checkpoint>/dspark, max_draft_len: 7}` (the drafter's block size is 7 and `max_draft_len` must match it). Keep KV cache block reuse enabled (the default): with `kv_cache_config.enable_block_reuse: false`, or on a disaggregated generation server, prompts of about 131k tokens or more fail with `Draft KV cache context resize failed`.
 [^22]: Text input only. The image and audio tokenizers of the checkpoint are not run, and the model generates only text tokens.
 [^23]: Text-only decision model. `trtllm-serve embeddings` detects the native checkpoint and selects the FlashInfer attention backend automatically. Python `LLM.encode` callers must set `model_kwargs.architectures` to `[PplxDeciderForDecision]`. The embedding vector contains the 255 raw decision logits; apply the checkpoint's calibration and candidate normalization in the caller.
+[^24]: Mistral Large 4 runs only on NVIDIA GB300 (`SM103`) from the NVFP4 checkpoint in the Mistral format (`checkpoint_format: mistral`); the FP8 checkpoint and the Hugging Face format are not supported. The architecture name is the one TensorRT LLM assigns when it reads the checkpoint's `params.json`. It is served by three modeling_v2 targets, tensor-parallel attention with expert-parallel experts, attention data parallelism with expert-parallel experts, and tensor parallelism, each on 4 GPUs or on 8 GPUs over two nodes, with contexts of up to 1,048,576 tokens. Speculative decoding uses the DSpark draft in the checkpoint (3 draft tokens). See the [Mistral Large 4 deployment guide](../deployment-guide/deployment-guide-for-mistral-large-4-on-trtllm.md).
+[^25]: Validated for one TEP4 context server and one TEP4 generation server at a 32K-token context, without speculative decoding.
 
 # Encoder-Decoder Feature Support Matrix (PyTorch Backend)
 
@@ -176,6 +180,7 @@ not used to mark a feature as supported here.
 | `Qwen3_5ForConditionalGeneration`    | Yes               | Yes        | Untested        | Yes           | No             | Untested              | Yes                       | L + I + V |
 | `Qwen3_5MoeForConditionalGeneration` | Yes               | Yes        | Untested        | Yes           | No             | Untested              | Yes                       | L + I + V |
 | `Qwen4ExpForConditionalGeneration`   | Untested          | Untested   | Untested        | Yes           | Untested       | Yes                   | No                        | L + I     |
+| `MistralLarge4ForCausalLM` [^24]     | Yes               | Yes        | Yes             | Yes           | Yes            | Untested              | Untested                  | L + I     |
 
 Note:
 - L: Language

examples/serve/compatibility/README.md

@@ -15,6 +15,21 @@ This directory contains individual, self-contained examples demonstrating Tensor
    trtllm-serve Qwen/Qwen3-8B --reasoning_parser "qwen3" --tool_parser "qwen3"
    ```
 
+   Native `mistral_large_4` required and named tool choices need the `llguidance`
+   guided-decoding backend. Override an xgrammar serving configuration with:
+
+   ```bash
+   trtllm-serve /path/to/checkpoint --config serve.yaml \
+       --tool_parser mistral_large_4 --reasoning_parser mistral_large_4 \
+       --set guided_decoding_backend=llguidance
+   ```
+
+   Required tool choices allow parallel calls unless the request explicitly sets
+   `parallel_tool_calls` to `false`. Named choices constrain generation to one
+   call, including when `parallel_tool_calls` is `true`, matching the single-call
+   forced-choice post-processing. Native required and named choices return HTTP
+   400 when the server uses xgrammar.
+
 
 ## Running Examples
 

jenkins/L0_Test.groovy

@@ -7640,12 +7640,15 @@ def launchTestJobs(pipeline, testFilter, globalVars)
         "GB300-4_GPUs-PyTorch-Post-Merge-2": ["auto:gb300-x4", "l0_gb300_multi_gpus", 2, 3, 4, 1, true, false],
         "GB300-4_GPUs-PyTorch-Post-Merge-3": ["auto:gb300-x4", "l0_gb300_multi_gpus", 3, 3, 4, 1, true, false],
         // PerfSanity post-merge tests
-        "GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-1": ["auto:gb300-x4", "l0_gb300_multi_gpus_perf_sanity", 1, 6, 4, 1, true, false],
-        "GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-2": ["auto:gb300-x4", "l0_gb300_multi_gpus_perf_sanity", 2, 6, 4, 1, true, false],
-        "GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-3": ["auto:gb300-x4", "l0_gb300_multi_gpus_perf_sanity", 3, 6, 4, 1, true, false],
-        "GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-4": ["auto:gb300-x4", "l0_gb300_multi_gpus_perf_sanity", 4, 6, 4, 1, true, false],
-        "GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-5": ["auto:gb300-x4", "l0_gb300_multi_gpus_perf_sanity", 5, 6, 4, 1, true, false],
-        "GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-6": ["auto:gb300-x4", "l0_gb300_multi_gpus_perf_sanity", 6, 6, 4, 1, true, false],
+        "GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-1": ["auto:gb300-x4", "l0_gb300_multi_gpus_perf_sanity", 1, 9, 4, 1, true, false],
+        "GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-2": ["auto:gb300-x4", "l0_gb300_multi_gpus_perf_sanity", 2, 9, 4, 1, true, false],
+        "GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-3": ["auto:gb300-x4", "l0_gb300_multi_gpus_perf_sanity", 3, 9, 4, 1, true, false],
+        "GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-4": ["auto:gb300-x4", "l0_gb300_multi_gpus_perf_sanity", 4, 9, 4, 1, true, false],
+        "GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-5": ["auto:gb300-x4", "l0_gb300_multi_gpus_perf_sanity", 5, 9, 4, 1, true, false],
+        "GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-6": ["auto:gb300-x4", "l0_gb300_multi_gpus_perf_sanity", 6, 9, 4, 1, true, false],
+        "GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-7": ["auto:gb300-x4", "l0_gb300_multi_gpus_perf_sanity", 7, 9, 4, 1, true, false],
+        "GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-8": ["auto:gb300-x4", "l0_gb300_multi_gpus_perf_sanity", 8, 9, 4, 1, true, false],
+        "GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-9": ["auto:gb300-x4", "l0_gb300_multi_gpus_perf_sanity", 9, 9, 4, 1, true, false],
     ]
     SBSASlurmTestConfigs = cbtsResizeSplits(SBSASlurmTestConfigs)
     fullSet += SBSASlurmTestConfigs.keySet()
@@ -7663,6 +7666,13 @@ def launchTestJobs(pipeline, testFilter, globalVars)
         // output breaks the mapping. For SingleNvlinkDomain see singleNvlinkDomainMode.
         "GB300-8_GPUs-2_Nodes-PyTorch-SingleNvlinkDomain-Post-Merge-1": ["auto:gb300-flex", "l0_gb300_multi_nodes_node2_gpu8", 1, 1, 8, 2],
         "GB300-16_GPUs-4_Nodes-PyTorch-SingleNvlinkDomain-Post-Merge-1": ["auto:gb300-flex", "l0_gb300_multi_nodes_node4_gpu16", 1, 1, 16, 4],
+        // 1P1D perf: one 4-GPU context node and one 4-GPU generation node.
+        // Explicit stages are discoverable by test_to_stage_mapping.py.
+        "GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-1": ["auto:gb300-flex", "l0_gb300_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node1_gpu4", 1, 5, 8, 2],
+        "GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-2": ["auto:gb300-flex", "l0_gb300_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node1_gpu4", 2, 5, 8, 2],
+        "GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-3": ["auto:gb300-flex", "l0_gb300_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node1_gpu4", 3, 5, 8, 2],
+        "GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-4": ["auto:gb300-flex", "l0_gb300_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node1_gpu4", 4, 5, 8, 2],
+        "GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-5": ["auto:gb300-flex", "l0_gb300_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node1_gpu4", 5, 5, 8, 2],
     ]
     // PerfSanity post-merge aggregated
     // 4 Nodes: Kimi K3 MXFP4 modeling_v2, TP16 attention on 16 GPUs in one NVLink domain
@@ -7783,15 +7793,6 @@ def launchTestJobs(pipeline, testFilter, globalVars)
         32,
         8
     )
-    // Nemotron-Ultra-V3 8k64k con1 and GLM-5.3-Flash FP8 8k1k con16: ctx1 (1 node, 4 GPUs) + gen1 tep4 (1 node, 4 GPUs) = 8 GPUs
-    multiNodesSBSAConfigs += buildStageConfigs(
-        "GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge",
-        "auto:gb300-flex",
-        "l0_gb300_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node1_gpu4",
-        2,
-        8,
-        2
-    )
     // Nemotron-Ultra-V3 50k2k con12: ctx1 (1 node, 4 GPUs) + gen6 (6 nodes, 4 GPUs each) = 28 GPUs
     multiNodesSBSAConfigs += buildStageConfigs(
         "GB300-28_GPUs-7_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN6-NODE1-GPU4-Post-Merge",

requirements.txt

@@ -76,7 +76,7 @@ einops
 fla-core==0.5.2 # the Kimi Linear / Kimi K3 KDA layers import fla.modules and fla.ops when they load
 flashinfer-python==0.7.2
 xgrammar==0.2.8
-llguidance==0.7.29
+llguidance==1.7.6
 jsonschema
 backoff
 nvtx
@@ -106,7 +106,8 @@ mcp>=1.23.0,<2.0.0 # 2.0.0 removed mcp.server.fastmcp; scaffolding still uses th
 apache-tvm-ffi==0.1.13.post2 # required by the CUTLASS DSL TVM-FFI provider
 torch-c-dlpack-ext==0.1.3 # used for reduce nvidia-cutlass-dsl host overhead, optional package for improved torch tensor calling perf
 flash-attn-4==4.0.0b19
-mistral-common>=1.10.0
+mistral-common[opencv]>=1.13.0; python_version >= "3.11"
+mistral-common>=1.10.0; python_version < "3.11" # 1.13.0 requires Python 3.11+
 torchao>=0.14.1,<0.16.0
 cuda-core
 llist

security_scanning/pyproject.toml

@@ -59,7 +59,7 @@ dependencies = [
     "fla-core (==0.5.2)",
     "flashinfer-python (==0.7.2)",
     "xgrammar (==0.2.8)",
-    "llguidance (==0.7.29)",
+    "llguidance (==1.7.6)",
     "jsonschema (>=4.26.0,<5.0.0)",
     "backoff (>=2.2.1,<3.0.0)",
     "nvtx (>=0.2.17,<0.3.0)",
@@ -79,7 +79,7 @@ dependencies = [
     "mcp (>=1.23.0,<2.0.0)",
     "torch-c-dlpack-ext (==0.1.3)",
     "flash-attn-4 (==4.0.0b19)",
-    "mistral-common (>=1.10.0)",
+    "mistral-common[opencv] (>=1.13.0) ; python_version >= \"3.11\"",
     "torchao (>=0.14.1,<0.16.0)",
     "cuda-tile (>=1.0.1)",
     "nvidia-cuda-tileiras (>=13.1,<13.2)",

tensorrt_llm/_torch/_experimental/modeling_v2/README.md

@@ -176,6 +176,10 @@ a Modeling V1 model class, and can override any of these steps:
 * **Constructor:** builds the model structure and parameters.
 * **`load_weights`:** loads checkpoint weights into the model. Put weight
   mapping, transformation and sharding in `weights.py`.
+* **`load_checkpoint`** (optional): reads the checkpoint directory itself. When
+  a target defines it, the engine calls it in place of `load_weights` and opens
+  no checkpoint file of its own. Use it for a checkpoint the target streams
+  from its files, in its on-disk tensor names.
 * **`post_load_weights`:** prepares the loaded weights, for example by
   transforming them or precomputing derived state. The default implementation
   in `DecoderModelForCausalLM` runs alias setup, weight transforms and derived

tensorrt_llm/_torch/_experimental/modeling_v2/catalog/_op.py

@@ -45,6 +45,7 @@
 
 import contextlib
 import enum
+import zlib
 from abc import ABC, abstractmethod
 from dataclasses import dataclass, field
 from typing import Any, Mapping
@@ -110,6 +111,14 @@ class Cell:
 
     why: str
     spec: Mapping[str, Any] = field(default_factory=dict)
+    #: The base of the cell's random inputs, for an entry whose test draws them.
+    #: None derives it from `why` (`seed_base`); set it when the inputs must not
+    #: follow the wording.
+    seed: int | None = None
+
+    def seed_base(self) -> int:
+        """The base an entry's test derives this cell's random inputs from."""
+        return zlib.crc32(self.why.encode()) if self.seed is None else self.seed
 
 
 class OpWrapper(ABC):