[yibinl-nvidia]

Description

Video requests can combine text, image, video, and audio references, but repeated multipart reference fields previously kept only the last part. This silently dropped conditioning inputs and allowed requests to evade model reference-count limits.

Preserve every multipart reference in per-modality order, including repeated uploads and mixtures of uploads with JSON reference objects/lists. Extend the request schema to accept upload lists and keep model capability/limit validation in the shared serving path for both /v1/videos and /v1/videos/sync. Add a MiniMax H3 Ref2VA example; the endpoint handling is model-independent.

Currently, only MiniMax H3 in Ref2VA mode supports the full Omni input combination (text, image, video, and audio). The shared endpoint handling is model-independent, so other models can use it when they implement and declare the corresponding input capabilities.

Test Coverage

• 389 tests passed across test_trtllm_serve_endpoints.py, test_visual_gen_utils.py, and test_minimax_h3_pipeline.py.
• Added 27 cases covering combined modalities, both routes, JSON and multipart transport, repeated/mixed references, reference order, slot limits, unsupported audio, and malformed parts. Coverage uses H3 Ref2VA metadata and an independently declared generic model.
• SKIP=type-check,clang-format,ruff-legacy pre-commit run -a passed.
• Real H3 Ref2VA run through trtllm-serve: one multipart request with prompt + image + video + audio returned HTTP 202, completed, and downloaded through /v1/videos/{id}/content with HTTP 200. Output: H.264 960×544, 124 frames at 24 FPS, stereo AAC at 32 kHz. Full FFmpeg decode and all 124 video frames verified; audio was non-silent. Generation/encoding took approximately 491 seconds on one GPU.
• Runtime qualification: PyAV and FFmpeg were installed for the end-to-end run. Input audio and the reference video's soundtrack were resampled to 32 kHz with FFmpeg because torchaudio was unavailable. This run does not validate runtime resampling of other sample rates.

PR Checklist

• Description, regression tests, and serving example updated.
• DCO sign-off included; no new repository dependencies or ownership changes.
• Apply the appropriate API compatibility label during review (additive upload-list support).

Dev Engineer Review

The request schema now accepts a single reference or a list of uploads and JSON reference objects. Multipart parsing preserves part order within each modality and supports repeated and mixed parts. Shared validation checks model-declared reference slots and count limits for video-generation requests.

The change updates video-reference handling, serving documentation, and endpoint tests. Model capabilities determine supported modalities and reference limits. Test execution results are not established by the repository inspection.

QA Engineer Review

test_trtllm_serve_endpoints.py adds tests for JSON and multipart requests, both video routes, repeated and mixed references, ordering, reference limits, unsupported audio, and malformed parts. The test file is listed in the CI tests/integration/test_lists/test-db/l0_cpu.yml list. No manual-QA listing is established. Coverage verdict: sufficient for the described request-parsing and validation changes. Test execution status is unavailable.

Per-File QA Perspective

• examples/visual_gen/serve/README.md: Documents combined text, image, video, and audio inputs, including repeated and mixed multipart references. Verify the model-dependent support guidance against the served API.
• tensorrt_llm/serve/openai_protocol.py: Reference fields now accept lists containing uploads or JSON reference objects. Verify schema parsing for single values, lists, and multipart uploads.
• tensorrt_llm/serve/openai_video_routes.py: Both video routes parse repeated reference fields and preserve order within each modality. Verify empty parts, malformed JSON parts, and mixed upload/object values.
• tensorrt_llm/serve/visual_gen_utils.py: Video requests validate reference slots and counts before reference processing. Verify that unsupported modalities and excess references fail before upload materialization.
• tests/unittest/_torch/visual_gen/test_trtllm_serve_endpoints.py: Adds endpoint tests for reference transports, ordering, limits, and error cases. The file is listed in the CI l0_cpu.yml list; no manual-QA listing is established.

[coderabbitai]

No actionable comments were generated in the recent review.

Recent review info

Run configuration

• Source mode: Auto
• Repository: NVIDIA/TensorRT-LLM
• Branch: main
• File: .coderabbit.yaml
• Commit: 0de8b6f
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 448b3dc7-3883-411f-8e1c-dcc85faf65fb

Commits

Reviewing files that changed from the base of the PR and between eaa226b and 9609154.

Files selected for processing: 1

• tests/unittest/_torch/visual_gen/test_trtllm_serve_endpoints.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

Video-generation requests can combine uploaded files and structured references for image, video, and audio inputs. Multipart parsing retains repeated values and their order. Parameter parsing checks reference support and count limits against pipeline slot metadata.

Changes

Video-generation media references

Layer / File(s)	Summary
Reference contract and multipart parsing tensorrt_llm/serve/openai_protocol.py, tensorrt_llm/serve/openai_video_routes.py, examples/visual_gen/serve/README.md	Reference fields accept lists that mix uploads and structured references. Multipart parsing handles repeated fields and parses nonempty text parts as JSON. Endpoint documentation includes audio references, and the README describes combined media inputs.
Slot validation and endpoint coverage tensorrt_llm/serve/visual_gen_utils.py, tests/unittest/_torch/visual_gen/test_trtllm_serve_endpoints.py	Parameter parsing checks explicit references against pipeline slot metadata. Tests cover JSON and multipart inputs, reference ordering, unsupported slots, count limits, and invalid JSON parts.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant openai_video_routes
  participant parse_visual_gen_params
  participant LoadedPipeline
  Client->>openai_video_routes: Send video-generation request with media references
  openai_video_routes->>parse_visual_gen_params: Pass parsed request
  parse_visual_gen_params->>LoadedPipeline: Check reference slot metadata
  parse_visual_gen_params-->>openai_video_routes: Accept request or return validation error

Loading

Suggested reviewers: schetlur-nv

Merge Risk: Minimal · up to 96091

The reference-handling change matches the supported contracts, with no established user-impacting regression. Normal CI follow-up remains appropriate.

Pre-merge checks: 5

Check name	Status	Explanation
Title check		The title follows the required ticket/type format and clearly identifies the main change: support for combined Omni references in video serving.
Description check		The description includes the required Description, Test Coverage, and PR Checklist sections. It explains the problem, solution, affected routes, model scope, tests, runtime validation, and API compati…
Docstring Coverage		Docstring coverage is 87.50% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentati…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Description check

Explanation

The description includes the required Description, Test Coverage, and PR Checklist sections. It explains the problem, solution, affected routes, model scope, tests, runtime validation, and API compatibility consideration.

Full details: Docstring Coverage

Explanation

Docstring coverage is 87.50% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 8 functions across 4 files.

Finishing Touches

Generate unit tests (beta)

• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

[yibinl-nvidia]

/bot run

[tensorrt-cicd]

PR_Github #81764 [ run ] triggered by Bot. Commit: eaa226b Link to invocation

[tensorrt-cicd]

PR_Github #81764 [ run ] completed with state SUCCESS. Commit: eaa226b
/LLM/main/L0_MergeRequest_PR pipeline #67422 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[yibinl-nvidia]

/bot run

[tensorrt-cicd]

PR_Github #82074 [ run ] triggered by Bot. Commit: 9609154 Link to invocation

[tensorrt-cicd]

PR_Github #82074 [ run ] completed with state SUCCESS. Commit: 9609154
/LLM/main/L0_MergeRequest_PR pipeline #67709 completed with status: 'SUCCESS'

CI Report

Link to invocation

[mikeiovine]

Stamp on behalf of runtime devs, delegating proper review to @NVIDIA/trt-llm-torch-visual-gen-devs

[luyiyun1021]

LGTM

[github-actions]

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.