[suyoggupta]

1. This MR enables the integration of TRTLLM-bench with AutoDeploy.
2. Adds a feature to AutoDeploy inference optimizer to inflate the kv-caches to the available GPU memory. This helps improve the token throughput.

Next step is to close the perf gap between AutoDeploy and Pytorch backends.
Current results
Max throughput for llama3.1 8B ISL/OSL=128/128, fp16. H100.
AutoDeploy using FlashInfer attn backend:

Request Throughput (req/sec):                     98.6547                                                                                                    
Total Output Throughput (tokens/sec):             12627.7960                                                                                                 
Per User Output Throughput (tokens/sec/user):     14.1241                                                                                                    
Per GPU Output Throughput (tokens/sec/gpu):       12627.7960                                                                                                 
Total Latency (ms):                               10136.3690                                                                                                 
Average request latency (ms):                     9081.9387  

Pytorch using FlashInfer backend:

Request Throughput (req/sec):                     115.5171                                                                                                   
Total Output Throughput (tokens/sec):             14786.1895                                                                                                 
Per User Output Throughput (tokens/sec/user):     16.4321                                                                                                    
Per GPU Output Throughput (tokens/sec/gpu):       14786.1895                                                                                                 
Total Latency (ms):                               8656.7266
Average request latency (ms):                     7796.3203        

[juney-nvidia]

@kaiyux to make sure he is aware of the addition of AutoDeploy as another backend of trtllm-bench.

Thanks
June

[suyoggupta]

/bot run

[niukuo]

PR_Github #355 [ run ] triggered by Bot

[niukuo]

PR_Github #355 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #324 completed with status: 'FAILURE'

[suyoggupta]

/bot run

[niukuo]

PR_Github #376 [ run ] triggered by Bot

[niukuo]

PR_Github #376 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #339 completed with status: 'FAILURE'

[kaiyux]

The trtllm-bench part looks good to me.

@suyoggupta Is it possible to split the PR so that it only includes the changes of "enables the integration of TRTLLM-bench with AutoDeploy"? I see there are also a bunch of changes under AutoDeploy core tensorrt_llm/_torch/auto_deploy

[suyoggupta]

The trtllm-bench part looks good to me.

@suyoggupta Is it possible to split the PR so that it only includes the changes of "enables the integration of TRTLLM-bench with AutoDeploy"? I see there are also a bunch of changes under AutoDeploy core tensorrt_llm/_torch/auto_deploy

@kaiyux : The changes in autodeploy are needed to ensure the throughput benchmark in tensorrt-llm-bench can run without OOM. Those changes ensure that the autodeploy executor allocates a large enough kv-cache as required by the throughput test. So unfortunately, these changes have be bundled together.

[suyoggupta]

/bot run

[niukuo]

PR_Github #455 [ run ] triggered by Bot

[niukuo]

PR_Github #455 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #390 completed with status: 'FAILURE'

[lucaslie]

Looks great. Just a few minor improvements

[niukuo]

PR_Github #475 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #408 completed with status: 'FAILURE'

[suyoggupta]

/bot run

[niukuo]

PR_Github #496 [ run ] triggered by Bot

[niukuo]

PR_Github #496 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #428 completed with status: 'FAILURE'

[suyoggupta]

/bot run

[niukuo]

PR_Github #514 [ run ] triggered by Bot

[kaiyux]

The trtllm-bench part looks good to me.
@suyoggupta Is it possible to split the PR so that it only includes the changes of "enables the integration of TRTLLM-bench with AutoDeploy"? I see there are also a bunch of changes under AutoDeploy core tensorrt_llm/_torch/auto_deploy

@kaiyux : The changes in autodeploy are needed to ensure the throughput benchmark in tensorrt-llm-bench can run without OOM. Those changes ensure that the autodeploy executor allocates a large enough kv-cache as required by the throughput test. So unfortunately, these changes have be bundled together.

If that's the case, maybe we should merge those changes in core library firstly? There shouldn't be features "required by" the bench scripts, does that make sense?

Though I agree to save some overheads by avoid submitting another PR, just wanted to clarify and align the principle here. Thanks.

[kaiyux]

Approving on trtllm-bench changes. Thanks.

[niukuo]

PR_Github #514 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #439 completed with status: 'SUCCESS'

[kaiyux]

/bot run

[niukuo]

PR_Github #548 [ run ] triggered by Bot

[niukuo]

PR_Github #548 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #466 completed with status: 'SUCCESS'

[suyoggupta]

/bot run

[tensorrt-cicd]

PR_Github #612 [ run ] triggered by Bot

[suyoggupta]

If that's the case, maybe we should merge those changes in core library firstly? There shouldn't be features "required by" the bench scripts, does that make sense?

Though I agree to save some overheads by avoid submitting another PR, just wanted to clarify and align the principle here. Thanks.

I agree with the general principle and something we can follow for subsequent PRs.

[lucaslie]

lgtm

[tensorrt-cicd]

PR_Github #612 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #516 completed with status: 'SUCCESS'