From 3a35938313e33dcc283616a6d0860bbb4be88e2c Mon Sep 17 00:00:00 2001 From: Junyi Xu Date: Tue, 12 Aug 2025 13:07:44 +0000 Subject: [PATCH] [https://nvbugs/5437106][fix] Add L4 Scout benchmarking WAR option in deploy guide Signed-off-by: Junyi Xu --- .../quick-start-recipe-for-llama4-scout-on-trtllm.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/source/deployment-guide/quick-start-recipe-for-llama4-scout-on-trtllm.md b/docs/source/deployment-guide/quick-start-recipe-for-llama4-scout-on-trtllm.md index 41bffb4bb29f..e3168deca1fc 100644 --- a/docs/source/deployment-guide/quick-start-recipe-for-llama4-scout-on-trtllm.md +++ b/docs/source/deployment-guide/quick-start-recipe-for-llama4-scout-on-trtllm.md @@ -65,9 +65,14 @@ cuda_graph_config: max_batch_size: 1024 kv_cache_config: dtype: fp8 +use_torch_sampler: true EOF ``` +> Here `use_torch_sampler: true` is added as a temporary WAR to solve illegal memory access issue when using trtllm native sampler. +> +> TODO: Remove this after the issue is resolved + ### Launch the TRT-LLM Server Below is an example command to launch the TRT-LLM server with the Llama-4-Scout-17B-16E-Instruct-FP8 model from within the container. The command is specifically configured for the 1024/1024 Input/Output Sequence Length test. The explanation of each flag is shown in the “Configs and Parameters” section.