[Usage]: Best way to batch requests with different lora ids #10761

System Info

System Information:

• OS:
• Python version:
• CUDA version:
• GPU model(s):
• Driver version:
• TensorRT-LLM version:

Detailed output:

Paste the output of the above commands here

How would you like to use TensorRT-LLM

Assume model is llama 3.1 8B, and we use multi lora. What’s the best routing strategy in inference?

1. Can requests in 1 batch have different lora ids? Does it have negative impact compared to 1 batch having same lora ids?
2. Is it true that kv cache can’t be shared across different lora ids? Thus it is beneficial to route requests with same lora id to the same trtllm instance?
3. Are there any other reasons why we prefer routing requests with same lora id to the same trtllm instance?

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.