[yechank-nvidia]

Description

Why. The executor decides whether a model is multimodal with isinstance(model, MultimodalModelMixin), so only subclasses of the mixin reach the multimodal runtime paths (item scheduling, the encoder cache, startup memory profiling, encoder timing). The executor also reaches into mixin internals, such as the private encoder cache initializer and the private encoder timing hook. A model that does not use the mixin, for example a Modeling V2 target, which must not import Modeling V1 model classes, has no written contract to follow.

What. Add SupportsMultimodal in tensorrt_llm/_torch/models/multimodal.py: capability flags, encoder parallel modes and attention backends, language_model, embedding properties, the model-owned encoder cache, item scheduling, encoding, fusion, and the encoder timing hook encoder_forward_context. The module imports no model class. is_multimodal_model() reads the supports_multimodal class marker, and get_multimodal_data_device_paths() is the one reader of the optional multimodal_data_device_paths, for any model.

Every executor gate that checked isinstance(..., MultimodalModelMixin) now uses the protocol: item scheduling, the encoder cache, startup memory profiling, the breakable prefill graph check, spec-decode fast paths, the weight transfer scope, and the encoder timing setup. MultimodalModelMixin sets the marker and stays the shared implementation. A model that does not use the mixin opts in with supports_multimodal = True. Cross-iteration encoder prefetch stays a mixin feature, because it uses mixin internals.

Review note: until #20739 merges, this branch also carries #20739's seven commits. Please review only the last commit, 253e64679b ("define a shared multimodal model protocol"). The branch will be rebased onto main once #20739 lands.

Builds on #20739. #20739 (input and encoder timing metrics) adds the encoder timing hook as the private mixin attribute _encoder_forward_context and sets it in PyExecutor._setup_encoder_perf_metrics behind an isinstance check. This PR makes the hook the public protocol member encoder_forward_context and gates that setup on the protocol. #20739's metrics and tests are otherwise unchanged; its two tests that set the hook use the new name.

Leaves room for #21052. #21052 (encoder input batching, and copying encoder inputs to the GPU inside the encoder call) needs no protocol change. Its transfer helper request_device_paths(model) reads the model's device paths, and it can do that through get_multimodal_data_device_paths(model), so there is still one reader. Whichever of the two PRs lands second needs a small rebase (about ten lines: imports and the device-path readers).

Also. Align the encoder entry points of six models (Gemma4, GLM-5 Next vision, Nemotron-H multimodal, Qwen2-VL, Qwen3-VL, MuseSpark) with the mixin's Sequence input, so all 21 mixin subclasses conform statically. Document the contract in docs/source/torch/adding_new_model.md, and list SupportsMultimodal with DecoderModel and DecoderModelForCausalLM among the runtime interfaces in the Modeling V2 README.

Performance. No runtime change: a class marker read replaces isinstance, and no tensor copy or host sync is added. Not measured.

Test Coverage

• New CPU tests in tests/unittest/_torch/multimodal/test_multimodal_mixin.py: detection does not evaluate encoder properties, the defaults do not enable optional features, the cache keeps the existing initialization hook, and the mixin defines every protocol member.
• New test tests/unittest/_torch/pyexecutor/engine/test_multimodal.py::test_item_scheduler_accepts_model_contract_without_mixin_inheritance: the item scheduler accepts a model that implements the protocol without the mixin.
• CPU unit tests in 21 targets (35 test files: item scheduling, encoder cache and prefetch, startup memory profiling, decoder and encoder runners, model engine and loader, LoRA, public API stability, input and encoder timing metrics), this PR vs its parent: 1871 vs 1866 cases, 1740 vs 1735 passed, 0 changed outcomes; the 5 new tests pass. The other 90 cases fail the same way on both trees: 84 need a GPU, 1 hits a CPU shared-memory error in the test container, and 5 need the pytest mocker fixture, which the harness does not load.
• mypy 2.1.0: --strict passes on the protocol module, and all 21 MultimodalModelMixin subclasses conform to SupportsMultimodal.
• Pre-commit hooks (check-only), py_compile and git diff --check pass.

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.