[None][modeling_v2] Multimodal mixin for modeling_v2

.github/CODEOWNERS

@@ -174,6 +174,7 @@
 /scripts/generate_config_database_tests.py @NVIDIA/trt-llm-models-devs @NVIDIA/trt-llm-doc-owners
 /scripts/generate_config_table.py @NVIDIA/trt-llm-models-devs @NVIDIA/trt-llm-doc-owners
 /tensorrt_llm/_torch/models @NVIDIA/trt-llm-models-devs
+/tensorrt_llm/_torch/multimodal @NVIDIA/trt-llm-models-devs @NVIDIA/trt-llm-runtime-devs
 /tensorrt_llm/_torch/modules @NVIDIA/trt-llm-models-devs
 /tensorrt_llm/_torch/modules/dwdp @NVIDIA/trt-llm-runtime-devs
 /tensorrt_llm/_torch/modules/linear.py @NVIDIA/trt-llm-runtime-devs

docs/source/developer-guide/multimodal.md

@@ -168,6 +168,46 @@ override selection explicitly; the compatibility selector cannot describe an
 arbitrary model layout. An empty partition dict means every field is shared,
 so it is appropriate only when selecting an item requires no payload changes.
 
+### Encoder Rows and Token Fusion
+
+`MultimodalRuntimeMixin` and `MultimodalModelMixin` live in
+[_torch/multimodal/model.py](source:tensorrt_llm/_torch/multimodal/model.py).
+`MultimodalRuntimeMixin` is the executor's multimodal capability contract and
+the encoder runtime: encoding, request-local reuse across prefill chunks, the
+persistent encoder-output cache, side-stream prefetch and item scheduling. A
+model that runs its own decoder, such as a Modeling V2 target, inherits it
+directly. `MultimodalModelMixin` extends it for composite models whose language
+model is a separate module: it delegates to that module and adds a default
+`forward` and the default text embedding fusion. The previous
+`_torch/models/modeling_multimodal_mixin.py` path re-exports the same classes
+for compatibility.
+
+Every `MultimodalRuntimeMixin` subclass implements `encode_multimodal_inputs`,
+`embedding_dim` and `embedding_dtype`. It can also declare its encoder's limits:
+
+* `max_encoder_tokens_per_item` bounds the dummy items that encoder memory
+  profiling builds and the largest atomic item that item scheduling budgets for.
+* `supports_concurrent_encoder` permits side-stream prefetch, which runs the
+  encoder on an auxiliary CUDA stream. The engine rejects
+  `encoder_side_stream_max_ahead > 0` for a model that does not set it.
+  Composite models set it by default.
+* `allows_breakable_prefill_cuda_graph` admits breakable prefill CUDA graphs
+  for a model that encodes outside the captured forward.
+
+`prepare_multimodal_embeddings` performs cache lookup and encoding, waits for
+prefetched results, selects current-chunk rows, and runs both model hooks.
+Its result contains the input token IDs returned by the full-embedding hook,
+active multimodal rows, and any auxiliary embedding tensors returned by the
+active-embedding hook. It does not look up text embeddings. When the caller
+passes the executor's `mm_token_indices`, the result carries them with the
+rows, after checking that both count the same rows, so a model with its own
+embedding splice gets rows and positions together.
+
+`MultimodalModelMixin.prepare_multimodal_inputs` uses the same preparation and
+then applies the default text embedding fusion. Model-specific encoder
+implementations and token embedding operations remain in their respective
+model packages.
+
 ### Item Boundaries and Model Hooks
 
 An item is the smallest independently encodable unit, not necessarily one

tensorrt_llm/_torch/_experimental/modeling_v2/README.md

@@ -125,6 +125,8 @@ it uses only runtime interfaces, for example:
 
 * `DecoderModel`, `DecoderModelForCausalLM`, `ModelConfig` and `Mapping`.
 * Attention and speculative metadata, and the KV cache managers.
+* `MultimodalRuntimeMixin` from `_torch/multimodal/model.py`, which provides
+  shared encoder execution and caching independently of model implementations.
 
 A target does not import, subclass or call Modeling V1 model code: the model
 classes in `_torch/models/`, the layers in `_torch/modules/`, MLA and the
@@ -134,6 +136,62 @@ path does not cover, and weight loading. The
 [DeepSeek R1 target](models/deepseek_v3/r1_0528_nvfp4__sm_10x__default/) is an
 example that uses no Modeling V1 model code.
 
+### Multimodal targets
+
+A target for a checkpoint whose input processor accepts media declares how it
+serves media. It inherits `MultimodalRuntimeMixin` from
+`_torch/multimodal/model.py` to serve media through the shared multimodal
+runtime, or `TextOnlyTarget` from [`_multimodal.py`](_multimodal.py) to serve
+the language model only. The marker refuses requests that carry media at
+admission, so each fails on its own rather than failing its batch in `forward`. `modeling_v2_resolve`
+checks this for every target a `route` returns, in `auto` as well as
+`require`, and `test_modeling_v2_multimodal_contract.py` checks it for every
+routed target in CI.
+
+A runtime target inherits `DecoderModelForCausalLM` and
+`MultimodalRuntimeMixin`. The mixin has no notion of a separate language model
+and defines nothing the decoder defines, so the decoder's speculative and
+runtime hooks are unaffected. Implement `encode_multimodal_inputs`,
+`embedding_dim` and `embedding_dtype`. The model loader reads the last two from
+the constructed target, so they must work with the checkpoint's real
+configuration. Expose `dtype` for encoder memory profiling.
+
+Before the core's forward, call `prepare_multimodal_embeddings` with the
+engine's context count, the request Params and the executor's
+`mm_token_indices`. It returns the active encoder rows after request-local
+reuse, persistent cache lookup, stream synchronization, chunk selection, and
+model hooks, paired with their placeholder positions. Feed both to the
+target's own token splice. Keep original token IDs available when other model
+operations need them.
+
+Filter position-only Params in `select_multimodal_params`: the processor emits
+them for text requests, and the executor adds them for generation requests.
+Enable `supports_encoder_cache` only when the processor supplies stable item
+keys and exact output row counts. Declare packed input partitions or implement
+item selection so partial cache hits encode only missing items. Encoder outputs
+and cached tensors are read-only.
+
+Declare the encoder's limits on the target:
+
+* `max_encoder_tokens_per_item` bounds the dummy items that encoder memory
+  profiling builds. Return the largest item the encoder runs in one piece when
+  it splits or refuses larger ones.
+* `supports_concurrent_encoder` permits side-stream prefetch. Set it only when
+  mutable encoder state, such as kernel workspaces, is separate per CUDA
+  stream.
+* `allows_breakable_prefill_cuda_graph` admits breakable prefill CUDA graphs.
+  Return True only when encoding happens outside the captured forward.
+
+Item scheduling is a separate capability. Set
+`supports_mm_encoder_item_scheduling` only after the complete cost, selection,
+and memory-profile contracts work for the target's certified encoder shapes.
+The Qwen3.8 Flash Next target uses inline encoding, chunk reuse, persistent
+encoder caching, prefetch and breakable prefill CUDA graphs; it leaves item
+scheduling disabled because the processor's maximum atomic image exceeds its
+certified per-frame patch bound. See the
+[multimodal developer guide](../../../../docs/source/developer-guide/multimodal.md)
+for the shared contracts.
+
 ## Adding a family or a target
 
 1. **New family only:** create `models/<family>/routing.py` with a `route`

tensorrt_llm/_torch/_experimental/modeling_v2/_multimodal.py

@@ -0,0 +1,78 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+"""The multimodal contract every target of a multimodal checkpoint meets.
+
+A checkpoint is multimodal when its input processor accepts media
+(``BaseMultimodalInputProcessor``). A target for such a checkpoint does one of two things:
+
+* It serves media through the shared multimodal runtime: it inherits
+  ``MultimodalRuntimeMixin`` (``_torch/multimodal/model.py``) and implements its required
+  members. The executor then gives it encoder-output caching, chunk reuse, prefetch and the
+  other runtime features.
+* It serves the checkpoint's language model only: it inherits :class:`TextOnlyTarget`,
+  builds no encoder, and the marker refuses requests carrying media at admission.
+
+``modeling_v2_resolve`` checks this for every target a ``route`` returns, under ``auto`` as
+well as ``require``: a target that meets neither is a bug in the target, not a deployment it
+declines.
+
+Like the router, this module imports nothing at load time: the registries it reads are
+imported when a target is checked.
+"""
+
+#: ``multimodal_data`` entries that carry media for an encoder, or encoder output in place of it.
+_MEDIA_KEYS = frozenset({"audio", "image", "video", "multimodal_embedding"})
+
+
+class TextOnlyTarget:
+    """Marker for a target that serves only the language model of a multimodal checkpoint.
+
+    The target builds no multimodal encoder. Requests that carry media are refused at
+    admission, so each fails on its own instead of failing every request in its batch.
+    """
+
+    def validate_multimodal_request_data(self, mm_data: dict) -> None:
+        """Refuse a request whose multimodal payload carries media; accept position-only data."""
+        media = sorted(key for key in _MEDIA_KEYS if mm_data.get(key) is not None)
+        if media:
+            raise ValueError(
+                f"{type(self).__name__} serves this checkpoint's language model only and "
+                f"does not accept multimodal input; the request carries {', '.join(media)}"
+            )
+
+
+def checkpoint_accepts_media(architecture: str) -> bool:
+    """Whether the input processor registered for ``architecture`` accepts media."""
+    from tensorrt_llm._torch.models.modeling_utils import get_registered_model_class
+    from tensorrt_llm.inputs.registry import INPUT_PROCESSOR_REGISTRY, BaseMultimodalInputProcessor
+
+    model_cls = get_registered_model_class(architecture)
+    if model_cls is None:
+        return False
+    processor_cls = INPUT_PROCESSOR_REGISTRY._input_processors_cls_by_model_type.get(model_cls)
+    return processor_cls is not None and issubclass(processor_cls, BaseMultimodalInputProcessor)
+
+
+def check_target_multimodal_contract(architecture: str, target: type) -> None:
+    """Raise ``MultimodalModelContractError`` unless ``target`` meets the contract above."""
+    from tensorrt_llm._torch.multimodal.model import (
+        MultimodalModelContractError,
+        MultimodalRuntimeMixin,
+    )
+
+    runs_runtime = issubclass(target, MultimodalRuntimeMixin)
+    text_only = issubclass(target, TextOnlyTarget)
+    if runs_runtime and text_only:
+        raise MultimodalModelContractError(
+            f"{target.__qualname__} inherits both MultimodalRuntimeMixin and TextOnlyTarget"
+        )
+    if runs_runtime:
+        target.check_multimodal_class_contract()
+        return
+    if text_only or not checkpoint_accepts_media(architecture):
+        return
+    raise MultimodalModelContractError(
+        f"{target.__qualname__} is a target for the multimodal architecture {architecture!r}; "
+        "it must inherit MultimodalRuntimeMixin to serve media, or TextOnlyTarget to serve "
+        "the language model only"
+    )

tensorrt_llm/_torch/_experimental/modeling_v2/_router_index.py

@@ -46,6 +46,8 @@
 
 from tensorrt_llm.logger import logger
 
+from ._multimodal import check_target_multimodal_contract
+
 if TYPE_CHECKING:
     from tensorrt_llm._torch.model_config import ModelConfig
     from tensorrt_llm.llmapi.llm_args import TorchLlmArgs
@@ -223,6 +225,7 @@ def modeling_v2_resolve(config: "ModelConfig", llm_args: "TorchLlmArgs") -> Opti
         )
         return None
     if target is not None:
+        check_target_multimodal_contract(arch, target)
         _RESOLVED_MODES[target] = mode
     return target
 

tensorrt_llm/_torch/_experimental/modeling_v2/models/kimi_k3_vl/kimi_k3_mxfp4__sm_100__tp16_moetp16ep1/modeling.py

@@ -95,6 +95,7 @@
 import torch
 from torch import nn
 
+from tensorrt_llm._torch._experimental.modeling_v2._multimodal import TextOnlyTarget
 from tensorrt_llm._torch._experimental.modeling_v2._native import load_family_ops
 from tensorrt_llm._torch._experimental.modeling_v2.models.kimi_k3_vl.kernels.k3_mla.decode_view import (
     k3_mla_decode_view,
@@ -3216,7 +3217,7 @@ def _check_construction(model_config: ModelConfig) -> None:
 
 
 # >>> route B: this target's name
-class ModelingV2KimiK3Mxfp4Sm100Tp16Moetp16ep1(KimiLinearForCausalLM):
+class ModelingV2KimiK3Mxfp4Sm100Tp16Moetp16ep1(KimiLinearForCausalLM, TextOnlyTarget):
     # <<< route B
     """The class ``route`` returns: this target's text model (`KimiLinearModel` above) behind its checks.
 

tensorrt_llm/_torch/_experimental/modeling_v2/models/kimi_k3_vl/kimi_k3_mxfp4__sm_100__tp16_moetp4ep4/modeling.py

@@ -93,6 +93,7 @@
 import torch
 from torch import nn
 
+from tensorrt_llm._torch._experimental.modeling_v2._multimodal import TextOnlyTarget
 from tensorrt_llm._torch._experimental.modeling_v2._native import load_family_ops
 from tensorrt_llm._torch._experimental.modeling_v2.models.kimi_k3_vl.kernels.k3_mla.decode_view import (
     k3_mla_decode_view,
@@ -3294,7 +3295,7 @@ def _check_construction(model_config: ModelConfig) -> None:
     )
 
 
-class ModelingV2KimiK3Mxfp4Sm100Tp16Moetp4ep4(KimiLinearForCausalLM):
+class ModelingV2KimiK3Mxfp4Sm100Tp16Moetp4ep4(KimiLinearForCausalLM, TextOnlyTarget):
     """The class ``route`` returns: this target's text model (`KimiLinearModel` above) behind its checks.
 
     It inherits the built-in Kimi K3 causal LM for the checkpoint load and the engine hooks (`load_weights` through

tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_103__tp4/modeling.py

@@ -77,6 +77,7 @@
 from torch import nn
 from transformers import PretrainedConfig
 
+from tensorrt_llm._torch._experimental.modeling_v2._multimodal import TextOnlyTarget
 from tensorrt_llm._torch._experimental.modeling_v2._router_index import (
     ModelingV2Mode,
     NoTarget,
@@ -4498,7 +4499,7 @@ def _runtime_hooks(spec_fold: Optional[M3SpecStepFold]) -> RuntimeHooks:
 
 
 class ModelingV2MiniMaxM3Nvfp4Sm103Tp4(
-    DecoderModelForCausalLM[MiniMaxM3Tp4ModelingV2, PretrainedConfig]
+    DecoderModelForCausalLM[MiniMaxM3Tp4ModelingV2, PretrainedConfig], TextOnlyTarget
 ):
     """MiniMax-M3 NVFP4 / sm_103 / tp4, text only, with one-model Eagle3 or no speculation.
 

tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_103__tp8/modeling.py

@@ -79,6 +79,7 @@
 from torch import nn
 from transformers import PretrainedConfig
 
+from tensorrt_llm._torch._experimental.modeling_v2._multimodal import TextOnlyTarget
 from tensorrt_llm._torch._experimental.modeling_v2._router_index import (
     ModelingV2Mode,
     NoTarget,
@@ -4167,7 +4168,7 @@ def _runtime_hooks(spec_fold: Optional[M3SpecStepFold]) -> RuntimeHooks:
 
 
 class ModelingV2MiniMaxM3Nvfp4Sm103Tp8(
-    DecoderModelForCausalLM[MiniMaxM3Tp8ModelingV2, PretrainedConfig]
+    DecoderModelForCausalLM[MiniMaxM3Tp8ModelingV2, PretrainedConfig], TextOnlyTarget
 ):
     """MiniMax-M3 NVFP4 / sm_103 / tp8, text only, with one-model Eagle3 or no speculation.
 

tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp2/modeling.py

@@ -88,6 +88,7 @@
 from torch import nn
 from transformers import PretrainedConfig
 
+from tensorrt_llm._torch._experimental.modeling_v2._multimodal import TextOnlyTarget
 from tensorrt_llm._torch._experimental.modeling_v2._router_index import (
     ModelingV2Mode,
     resolved_mode,
@@ -6092,7 +6093,7 @@ def _runtime_hooks(spec_fold: Optional[M3SpecStepFold]) -> RuntimeHooks:
 
 
 class ModelingV2MiniMaxM3Nvfp4Sm107Tp2(
-    DecoderModelForCausalLM[MiniMaxM3Tp2ModelingV2, PretrainedConfig]
+    DecoderModelForCausalLM[MiniMaxM3Tp2ModelingV2, PretrainedConfig], TextOnlyTarget
 ):
     """MiniMax-M3 NVFP4 / sm_107 / tp2, text only, with one-model Eagle3 or no speculation.
 

tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp4/modeling.py

@@ -79,6 +79,7 @@
 from torch import nn
 from transformers import PretrainedConfig
 
+from tensorrt_llm._torch._experimental.modeling_v2._multimodal import TextOnlyTarget
 from tensorrt_llm._torch._experimental.modeling_v2._router_index import (
     ModelingV2Mode,
     resolved_mode,
@@ -5566,7 +5567,7 @@ def _runtime_hooks(spec_fold: Optional[M3SpecStepFold]) -> RuntimeHooks:
 
 
 class ModelingV2MiniMaxM3Nvfp4Sm107Tp4(
-    DecoderModelForCausalLM[MiniMaxM3Tp4ModelingV2, PretrainedConfig]
+    DecoderModelForCausalLM[MiniMaxM3Tp4ModelingV2, PretrainedConfig], TextOnlyTarget
 ):
     """MiniMax-M3 NVFP4 / sm_107 / tp4, text only, with one-model Eagle3 or no speculation.
 

tensorrt_llm/_torch/_experimental/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp8/modeling.py

@@ -86,6 +86,7 @@
 from torch import nn
 from transformers import PretrainedConfig
 
+from tensorrt_llm._torch._experimental.modeling_v2._multimodal import TextOnlyTarget
 from tensorrt_llm._torch._experimental.modeling_v2._router_index import (
     ModelingV2Mode,
     resolved_mode,
@@ -5784,7 +5785,7 @@ def _runtime_hooks(spec_fold: Optional[M3SpecStepFold]) -> RuntimeHooks:
 
 
 class ModelingV2MiniMaxM3Nvfp4Sm107Tp8(
-    DecoderModelForCausalLM[MiniMaxM3Tp8ModelingV2, PretrainedConfig]
+    DecoderModelForCausalLM[MiniMaxM3Tp8ModelingV2, PretrainedConfig], TextOnlyTarget
 ):
     """MiniMax-M3 NVFP4 / sm_107 / tp8, text only, with one-model Eagle3 or no speculation.