[kaiyux]

No description provided.

[yorkeehuang]

@kaiyux Dear expert, I have two questions about this commit:

1. Can this modification improve ViT performance (it seems that it currently only supports llava_next), and what are the main optimizations introduced?
2. Are there plans to implement Qwen2 VL/Qwen2.5 VL visual encoder in multimodal_encoders, and if so, what is the approximate timeline?
   Thank you very much!

[symphonylyh]

@yorkeehuang
Edit: the following answer is for pytorch workflow. for TRT workflow, see answers in the next paragraph

1. no, at this point, the vision encoder ViT part is still just running huggingface pytorch, so not accelerated by TRT-LLM. We have plan to define ViT using TRT-LLM API, but no timeline yet.
2. are you asking about Qwen2-VL or its vision encoder specifically? qwen2-vl model is already supported in https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/_torch/models/modeling_qwen2vl.py, and can try https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/pytorch/quickstart_multimodal.py. but if you're asking about just its vision encoder, the answer is same as (1)

ah...are you asking about the TRT workflow? since there are two workflows in TRT-LLM (the new Pytorch workflow, and the old TRT workflow), things can get confusing. I guess you are asking about "TensorRT workflow - Added vision encoders with tensor parallelism and context parallelism support. Refer to examples/vit/README.md."? If so, the answers is below:

1. it's more of a functionality unblocker, not expecting too much performance gain. In the past, ViT is built as a single TensorRT engine, and you cannot do tensor parallelism / context parallelism. but with this change, TP/CP can be enabled for relatively large ViTs
2. it's not on the roadmap, because new model / features are likely to land in the Pytorch workflow instead of TRT workflow. but I think you can follow the ViT TP/CP changes to smoothly enable it for qwen's vision encoder on your own

[yorkeehuang]

@yorkeehuang Edit: the following answer is for pytorch workflow. for TRT workflow, see answers in the next paragraph

1. no, at this point, the vision encoder ViT part is still just running huggingface pytorch, so not accelerated by TRT-LLM. We have plan to define ViT using TRT-LLM API, but no timeline yet.
2. are you asking about Qwen2-VL or its vision encoder specifically? qwen2-vl model is already supported in https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/_torch/models/modeling_qwen2vl.py, and can try https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/pytorch/quickstart_multimodal.py. but if you're asking about just its vision encoder, the answer is same as (1)

ah...are you asking about the TRT workflow? since there are two workflows in TRT-LLM (the new Pytorch workflow, and the old TRT workflow), things can get confusing. I guess you are asking about "TensorRT workflow - Added vision encoders with tensor parallelism and context parallelism support. Refer to examples/vit/README.md."? If so, the answers is below:

1. it's more of a functionality unblocker, not expecting too much performance gain. In the past, ViT is built as a single TensorRT engine, and you cannot do tensor parallelism / context parallelism. but with this change, TP/CP can be enabled for relatively large ViTs
2. it's not on the roadmap, because new model / features are likely to land in the Pytorch workflow instead of TRT workflow. but I think you can follow the ViT TP/CP changes to smoothly enable it for qwen's vision encoder on your own

@symphonylyh
If I want to enhance the inference performance of Qwen2.5-VL's ViT by integrating FasterTransformer features in TensorRT-LLM, are there effective methods or similar examples? Looking forward to your reply, Thank you!

[symphonylyh]

@yorkeehuang what are "by integrating FasterTransformer features in TensorRT-LLM"? TRT-LLM should already included FastTransformer features