[Bug]: Rejection sampling in Kimi K2.6 #15182

Skipping import of cpp extensions due to incompatible torch version 2.11.0a0+eb65b36914.nv26.02 for torchao version 0.15.0             Please see https://github.com/pytorch/ao/issues/2919 for more info
/usr/local/lib/python3.12/dist-packages/modelopt/torch/__init__.py:36: UserWarning: transformers version 5.5.4 is incompatible with nvidia-modelopt and may cause issues. Please install recommended version with `pip install nvidia-modelopt[hf]` if working with HF models.
  _warnings.warn(
[TensorRT-LLM] TensorRT LLM version: 1.3.0rc17
/usr/local/lib/python3.12/dist-packages/tensorrt_llm/serve/openai_protocol.py:137: UserWarning: Field name "schema" in "ResponseFormat" shadows an attribute in parent "OpenAIBaseModel"
  class ResponseFormat(OpenAIBaseModel):
[06/09/2026-23:08:14] [TRT-LLM] [W] [llmapi] Overriding moe_config 
[06/09/2026-23:08:14] [TRT-LLM] [W] [llmapi] Overriding kv_cache_config 
[06/09/2026-23:08:14] [TRT-LLM] [I] [llmapi] Using LLM with PyTorch backend
[06/09/2026-23:08:14] [TRT-LLM] [I] [llmapi] neither checkpoint_format nor checkpoint_loader were provided, checkpoint_format will be set to HF.
[06/09/2026-23:08:14] [TRT-LLM] [I] [llmapi] start MpiSession with 4 workers
[06/09/2026-23:08:14] [TRT-LLM] [I] [llmapi] Found /data/weights/pytrtllm-moonshotai--Kimi-K2.6-NVFP4/002/hf_quant_config.json, pre-quantized checkpoint is used.
[06/09/2026-23:08:14] [TRT-LLM] [W] [quantize] Inline 'config.json.quantization_config' diverges from 'hf_quant_config.json':
  - group_size: file=16, inline=None
[06/09/2026-23:08:14] [TRT-LLM] [I] [llmapi] Setting quant_algo=NVFP4 from HF quant config.
[06/09/2026-23:08:14] [TRT-LLM] [I] [llmapi] Setting kv_cache_quant_algo=FP8 from HF quant config.
[06/09/2026-23:08:14] [TRT-LLM] [I] [llmapi] Setting group_size=16 from HF quant config.
[06/09/2026-23:08:14] [TRT-LLM] [I] [llmapi] Setting exclude_modules=['language_model.lm_head', 'language_model.model.layers.0.self_attn*', 'language_model.model.layers.... from HF quant config.
[06/09/2026-23:08:14] [TRT-LLM] [W] [tokenizr] cloudpickle is not installed. Objects from trust_remote_code dynamic modules may not be serializable across nodes in multi-node setups. Install cloudpickle to fix this: pip install cloudpickle
`torch_dtype` is deprecated! Use `dtype` instead!
[06/09/2026-23:08:14] [TRT-LLM] [W] [quantize] Inline 'config.json.quantization_config' diverges from 'hf_quant_config.json':
  - group_size: file=16, inline=None
`KimiK25Processor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead.
The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
[06/09/2026-23:08:15] [TRT-LLM] [W] [executor] Orchestrator is creating IPC executor
�[33;20mrank 0 using MpiPoolSession to spawn MPI processes
�[0m[06/09/2026-23:08:15] [TRT-LLM] [I] [executor] Generating a new HMAC key for server proxy_request_queue
[06/09/2026-23:08:15] [TRT-LLM] [I] [executor] Generating a new HMAC key for server worker_init_status_queue
[06/09/2026-23:08:15] [TRT-LLM] [I] [executor] Generating a new HMAC key for server proxy_result_queue
Skipping import of cpp extensions due to incompatible torch version 2.11.0a0+eb65b36914.nv26.02 for torchao version 0.15.0             Please see https://github.com/pytorch/ao/issues/2919 for more info
Skipping import of cpp extensions due to incompatible torch version 2.11.0a0+eb65b36914.nv26.02 for torchao version 0.15.0             Please see https://github.com/pytorch/ao/issues/2919 for more info
Skipping import of cpp extensions due to incompatible torch version 2.11.0a0+eb65b36914.nv26.02 for torchao version 0.15.0             Please see https://github.com/pytorch/ao/issues/2919 for more info
Skipping import of cpp extensions due to incompatible torch version 2.11.0a0+eb65b36914.nv26.02 for torchao version 0.15.0             Please see https://github.com/pytorch/ao/issues/2919 for more info
Multiple distributions found for package optimum. Picked distribution: optimum
Multiple distributions found for package modelopt. Picked distribution: nvidia-modelopt
/usr/local/lib/python3.12/dist-packages/modelopt/torch/__init__.py:36: UserWarning: transformers version 5.5.4 is incompatible with nvidia-modelopt and may cause issues. Please install recommended version with `pip install nvidia-modelopt[hf]` if working with HF models.
  _warnings.warn(
Multiple distributions found for package optimum. Picked distribution: optimum
Multiple distributions found for package modelopt. Picked distribution: nvidia-modelopt
Multiple distributions found for package optimum. Picked distribution: optimum
Multiple distributions found for package modelopt. Picked distribution: nvidia-modelopt
Multiple distributions found for package optimum. Picked distribution: optimum
Multiple distributions found for package modelopt. Picked distribution: nvidia-modelopt
/usr/local/lib/python3.12/dist-packages/modelopt/torch/__init__.py:36: UserWarning: transformers version 5.5.4 is incompatible with nvidia-modelopt and may cause issues. Please install recommended version with `pip install nvidia-modelopt[hf]` if working with HF models.
  _warnings.warn(
/usr/local/lib/python3.12/dist-packages/modelopt/torch/__init__.py:36: UserWarning: transformers version 5.5.4 is incompatible with nvidia-modelopt and may cause issues. Please install recommended version with `pip install nvidia-modelopt[hf]` if working with HF models.
  _warnings.warn(
/usr/local/lib/python3.12/dist-packages/modelopt/torch/__init__.py:36: UserWarning: transformers version 5.5.4 is incompatible with nvidia-modelopt and may cause issues. Please install recommended version with `pip install nvidia-modelopt[hf]` if working with HF models.
  _warnings.warn(
[TensorRT-LLM] TensorRT LLM version: 1.3.0rc17
[TensorRT-LLM] TensorRT LLM version: 1.3.0rc17
[TensorRT-LLM] TensorRT LLM version: 1.3.0rc17
[TensorRT-LLM] TensorRT LLM version: 1.3.0rc17
/usr/local/lib/python3.12/dist-packages/tensorrt_llm/serve/openai_protocol.py:137: UserWarning: Field name "schema" in "ResponseFormat" shadows an attribute in parent "OpenAIBaseModel"
  class ResponseFormat(OpenAIBaseModel):
/usr/local/lib/python3.12/dist-packages/tensorrt_llm/serve/openai_protocol.py:137: UserWarning: Field name "schema" in "ResponseFormat" shadows an attribute in parent "OpenAIBaseModel"
  class ResponseFormat(OpenAIBaseModel):
/usr/local/lib/python3.12/dist-packages/tensorrt_llm/serve/openai_protocol.py:137: UserWarning: Field name "schema" in "ResponseFormat" shadows an attribute in parent "OpenAIBaseModel"
  class ResponseFormat(OpenAIBaseModel):
/usr/local/lib/python3.12/dist-packages/tensorrt_llm/serve/openai_protocol.py:137: UserWarning: Field name "schema" in "ResponseFormat" shadows an attribute in parent "OpenAIBaseModel"
  class ResponseFormat(OpenAIBaseModel):
[TRT-LLM] [I] [runtime][RANK 1] Refreshed the MPI local session
[TRT-LLM] [I] [runtime][RANK 3] Refreshed the MPI local session
[TRT-LLM] [I] [runtime][RANK 0] Refreshed the MPI local session
[TRT-LLM] [I] [runtime][RANK 2] Refreshed the MPI local session
[06/09/2026-23:08:35] [TRT-LLM] [W] [executor][RANK 0] Worker process 28030 is affined to run on the following CPUs: [0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, 70, 72, 74, 76, 78, 80, 82, 84, 86, 88, 90, 92, 94, 96, 98, 100, 102, 104, 106, 108, 110, 112, 114, 116, 118, 120, 122, 124, 126, 128, 130, 132, 134, 136, 138, 140, 142, 144, 146, 148, 150, 152, 154, 156, 158, 160, 162, 164, 166, 168, 170, 172, 174, 176, 178, 180, 182, 184, 186, 188, 190, 192, 194, 196, 198, 200, 202, 204, 206, 208, 210, 212, 214, 216, 218, 220, 222, 224, 226, 228, 230, 232, 234, 236, 238, 240, 242, 244, 246, 248, 250, 252, 254] (subset of all logical CPUs). This may harm performance if set incorrectly.
[06/09/2026-23:08:35] [TRT-LLM] [W] [executor][RANK 0] Worker process 28030 has constrained CPU affinity but `TLLM_NUMA_AWARE_WORKER_AFFINITY` is not set. Removing CPU affinity constraints.
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!
[06/09/2026-23:08:35] [TRT-LLM] [W] [quantize][RANK 0] Inline 'config.json.quantization_config' diverges from 'hf_quant_config.json':
  - group_size: file=16, inline=None
[06/09/2026-23:08:35] [TRT-LLM] [I] [_torch][RANK 0] ATTENTION RUNTIME FEATURES:  AttentionRuntimeFeatures(chunked_prefill=True, cache_reuse=True, has_speculative_draft_tokens=False, chunk_size=16384, chunked_prefill_buffer_batch_size=4)
[06/09/2026-23:08:35] [TRT-LLM] [W] [quantize][RANK 0] Inline 'config.json.quantization_config' diverges from 'hf_quant_config.json':
  - group_size: file=16, inline=None
`torch_dtype` is deprecated! Use `dtype` instead!
`KimiK25Processor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead.
The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
`KimiK25Processor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead.
The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
`KimiK25Processor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead.
The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
`KimiK25Processor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead.
The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
[06/09/2026-23:08:37] [TRT-LLM] [W] [quantize][RANK 0] Inline 'config.json.quantization_config' diverges from 'hf_quant_config.json':
  - group_size: file=16, inline=None
[06/09/2026-23:08:37] [TRT-LLM] [I] [_torch][RANK 0] Validating KV Cache config against kv_cache_dtype="auto"
[06/09/2026-23:08:37] [TRT-LLM] [I] [_torch][RANK 0] KV cache quantization set to "auto". Using checkpoint KV quantization.
You are using a model of type `kimi_k2` to instantiate a model of type `deepseek_v3`. This may be expected if you are loading a checkpoint that shares a subset of the architecture (e.g., loading a `sam2_video` checkpoint into `Sam2Model`), but is otherwise not supported and can yield errors. Please verify that the checkpoint is compatible with the model you are instantiating.
You are using a model of type `kimi_k2` to instantiate a model of type `deepseek_v3`. This may be expected if you are loading a checkpoint that shares a subset of the architecture (e.g., loading a `sam2_video` checkpoint into `Sam2Model`), but is otherwise not supported and can yield errors. Please verify that the checkpoint is compatible with the model you are instantiating.
You are using a model of type `kimi_k2` to instantiate a model of type `deepseek_v3`. This may be expected if you are loading a checkpoint that shares a subset of the architecture (e.g., loading a `sam2_video` checkpoint into `Sam2Model`), but is otherwise not supported and can yield errors. Please verify that the checkpoint is compatible with the model you are instantiating.
You are using a model of type `kimi_k2` to instantiate a model of type `deepseek_v3`. This may be expected if you are loading a checkpoint that shares a subset of the architecture (e.g., loading a `sam2_video` checkpoint into `Sam2Model`), but is otherwise not supported and can yield errors. Please verify that the checkpoint is compatible with the model you are instantiating.
[06/09/2026-23:08:49] [TRT-LLM] [I] [_torch][RANK 0] Use 141.85 GB for model weights.
[06/09/2026-23:08:49] [TRT-LLM] [I] [_torch][RANK 0] Prefetching 550.24GB checkpoint files.

Loading weights: 100%|██████████| 29/29 [00:00<00:00, 105.01it/s]
Model init total -- 127.98s

Loading weights: 100%|██████████| 29/29 [00:00<00:00, 93.80it/s] 
Model init total -- 128.62s
[TRT-LLM] [I] [batchmgr][RANK 3] Max KV cache blocks per sequence: 8193 [window size=262146], tokens per block=32, primary blocks=8256, secondary blocks=0, max sequence length=262146
[TRT-LLM] [I] [batchmgr][RANK 3] Number of tokens per block: 32.
[TRT-LLM] [I] [batchmgr][RANK 3] [MemUsageChange] Allocated 8.65 GiB for max tokens in paged KV cache (264192).
[TRT-LLM] [I] [batchmgr][RANK 3] Max KV cache blocks per sequence: 8193 [window size=262146], tokens per block=32, primary blocks=8256, secondary blocks=0, max sequence length=262146
[TRT-LLM] [I] [batchmgr][RANK 3] Number of tokens per block: 32.
[TRT-LLM] [I] [batchmgr][RANK 3] [MemUsageChange] Allocated 0.14 GiB for max tokens in paged KV cache (264192).
[TRT-LLM] [I] [batchmgr][RANK 1] Max KV cache blocks per sequence: 8193 [window size=262146], tokens per block=32, primary blocks=8256, secondary blocks=0, max sequence length=262146
[TRT-LLM] [I] [batchmgr][RANK 1] Number of tokens per block: 32.
[TRT-LLM] [I] [batchmgr][RANK 1] [MemUsageChange] Allocated 8.65 GiB for max tokens in paged KV cache (264192).
[TRT-LLM] [I] [batchmgr][RANK 1] Max KV cache blocks per sequence: 8193 [window size=262146], tokens per block=32, primary blocks=8256, secondary blocks=0, max sequence length=262146
[TRT-LLM] [I] [batchmgr][RANK 1] Number of tokens per block: 32.
[TRT-LLM] [I] [batchmgr][RANK 1] [MemUsageChange] Allocated 0.14 GiB for max tokens in paged KV cache (264192).

Loading weights:   0%|          | 0/29 [00:00<?, ?it/s]
Loading weights:   0%|          | 0/29 [00:00<?, ?it/s]
Loading weights:  24%|██▍       | 7/29 [00:00<00:00, 58.22it/s]
Loading weights:  52%|█████▏    | 15/29 [00:00<00:00, 129.98it/s]
Loading weights:  69%|██████▉   | 20/29 [00:00<00:00, 90.83it/s]
Loading weights: 100%|██████████| 29/29 [00:00<00:00, 78.81it/s]

Loading weights:  97%|█████████▋| 28/29 [00:00<00:00, 78.35it/s] 
Loading weights: 100%|██████████| 29/29 [00:00<00:00, 86.63it/s]
Model init total -- 128.41s
Model init total -- 128.46s
[06/09/2026-23:10:45] [TRT-LLM] [I] [_torch][RANK 0] Using Sampler: Eagle3OneModelSampler
[06/09/2026-23:10:45] [TRT-LLM] [I] [_torch][RANK 0] [create_py_executor] Created execution_stream: <torch.cuda.Stream device=cuda:0 cuda_stream=0x42e63080>
[06/09/2026-23:10:45] [TRT-LLM] [I] [_torch][RANK 0] Memory used at after loading weights (inside torch) in memory usage profiling: 142.22 GiB
[06/09/2026-23:10:45] [TRT-LLM] [I] [_torch][RANK 0] Memory used at after loading weights (outside torch) in memory usage profiling: 3.73 GiB
[06/09/2026-23:10:45] [TRT-LLM] [I] [_torch][RANK 0] [kv cache manager] Primary/secondary blocks for window sizes set to {262146: (8256, 0)} for estimation dry run
[06/09/2026-23:10:45] [TRT-LLM] [I] [_torch][RANK 0] Adjusted attention window size to 262146 in blocks_per_window
[06/09/2026-23:10:45] [TRT-LLM] [I] [_torch][RANK 0] [KVCacheManager] execution_stream: <torch.cuda.Stream device=cuda:0 cuda_stream=0x42e63080>
[06/09/2026-23:10:45] [TRT-LLM] [I] [_torch][RANK 0] [KVCacheManager] blocks_per_window: {262146: (8256, 0)}
[TRT-LLM] [I] [batchmgr][RANK 2] Max KV cache blocks per sequence: 8193 [window size=262146], tokens per block=32, primary blocks=8256, secondary blocks=0, max sequence length=262146
[TRT-LLM] [I] [batchmgr][RANK 2] Number of tokens per block: 32.
[TRT-LLM] [I] [batchmgr][RANK 2] [MemUsageChange] Allocated 8.65 GiB for max tokens in paged KV cache (264192).
[TRT-LLM] [I] [batchmgr][RANK 0] Max KV cache blocks per sequence: 8193 [window size=262146], tokens per block=32, primary blocks=8256, secondary blocks=0, max sequence length=262146
[TRT-LLM] [I] [batchmgr][RANK 0] Number of tokens per block: 32.
[TRT-LLM] [I] [batchmgr][RANK 0] [MemUsageChange] Allocated 8.65 GiB for max tokens in paged KV cache (264192).
[TRT-LLM] [I] [batchmgr][RANK 2] Max KV cache blocks per sequence: 8193 [window size=262146], tokens per block=32, primary blocks=8256, secondary blocks=0, max sequence length=262146
[TRT-LLM] [I] [batchmgr][RANK 2] Number of tokens per block: 32.
[TRT-LLM] [I] [batchmgr][RANK 2] [MemUsageChange] Allocated 0.14 GiB for max tokens in paged KV cache (264192).
[06/09/2026-23:10:45] [TRT-LLM] [I] [_torch][RANK 0] [kv cache manager] Primary/secondary blocks for window sizes set to {262146: (8256, 0)} for estimation dry run
[06/09/2026-23:10:45] [TRT-LLM] [I] [_torch][RANK 0] Adjusted attention window size to 262146 in blocks_per_window
[06/09/2026-23:10:45] [TRT-LLM] [I] [_torch][RANK 0] [KVCacheManager] execution_stream: <torch.cuda.Stream device=cuda:0 cuda_stream=0x42e63080>
[06/09/2026-23:10:45] [TRT-LLM] [I] [_torch][RANK 0] [KVCacheManager] blocks_per_window: {262146: (8256, 0)}
[TRT-LLM] [I] [batchmgr][RANK 0] Max KV cache blocks per sequence: 8193 [window size=262146], tokens per block=32, primary blocks=8256, secondary blocks=0, max sequence length=262146
[TRT-LLM] [I] [batchmgr][RANK 0] Number of tokens per block: 32.
[TRT-LLM] [I] [batchmgr][RANK 0] [MemUsageChange] Allocated 0.14 GiB for max tokens in paged KV cache (264192).
[06/09/2026-23:10:46] [TRT-LLM] [I] [_torch][RANK 0] max_seq_len=262146, max_num_requests=64, max_num_tokens=16384, max_batch_size=64
[06/09/2026-23:10:46] [TRT-LLM] [I] [_torch][RANK 0] cache_transceiver is disabled
[06/09/2026-23:10:46] [TRT-LLM] [I] [_torch][RANK 0] [PyExecutor] execution_stream initialized: <torch.cuda.Stream device=cuda:0 cuda_stream=0x42e63080>. 
[06/09/2026-23:10:46] [TRT-LLM] [I] [_torch][RANK 0] Run warmup with 1 tokens, include 0 generation tokens
[06/09/2026-23:10:46] [TRT-LLM] [I] [_torch][RANK 0] PDL enabled
[TRT-LLM] [W] [thop][RANK 1] Attention workspace size is not enough, increase the size from 0 bytes to 33591808 bytes
[TRT-LLM] [W] [thop][RANK 0] Attention workspace size is not enough, increase the size from 0 bytes to 33591808 bytes
[TRT-LLM] [W] [thop][RANK 2] Attention workspace size is not enough, increase the size from 0 bytes to 33591808 bytes
[06/09/2026-23:10:50] [TRT-LLM] [W] [_torch][RANK 0] [AutoTuner] trtllm::tunable_allreduce::allreduce using the fallback tactic, due to cache miss on input shapes=(torch.Size([1, 7168]), torch.Size([1, 7168]), torch.Size([7168]), torch.Size([1]), torch.Size([0]), torch.Size([14]))
[TRT-LLM] [W] [thop][RANK 3] Attention workspace size is not enough, increase the size from 0 bytes to 33591808 bytes
[TRT-LLM] [I] [thop][RANK 0] Detecting local TP group for rank 0
[TRT-LLM] [I] [thop][RANK 2] Detecting local TP group for rank 2
[TRT-LLM] [I] [thop][RANK 1] Detecting local TP group for rank 1
[TRT-LLM] [I] [thop][RANK 3] Detecting local TP group for rank 3
[TRT-LLM] [I] [thop][RANK 0] TP group is intra-node for rank 0
[TRT-LLM] [I] [thop][RANK 2] TP group is intra-node for rank 2
[TRT-LLM] [I] [thop][RANK 1] TP group is intra-node for rank 1
[TRT-LLM] [I] [thop][RANK 3] TP group is intra-node for rank 3
[06/09/2026-23:10:53] [TRT-LLM] [W] [_torch][RANK 0] [AutoTuner] trtllm::nvfp4_gemm::gemm using the fallback tactic, due to cache miss on input shapes=(torch.Size([1, 3584]), torch.Size([9216, 3584]), torch.Size([57344]), torch.Size([4128768]), torch.Size([1]))
[06/09/2026-23:10:53] [TRT-LLM] [W] [_torch][RANK 0] [AutoTuner] trtllm::fused_moe::gemm1 using the fallback tactic, due to cache miss on input shapes=(torch.Size([1, 3584]), torch.Size([96, 4096, 448]), torch.Size([0]), torch.Size([96, 7168, 128]), torch.Size([0]))
[06/09/2026-23:10:53] [TRT-LLM] [W] [_torch][RANK 0] [AutoTuner] trtllm::fused_moe::gemm2 using the fallback tactic, due to cache miss on input shapes=(torch.Size([1, 3584]), torch.Size([96, 4096, 448]), torch.Size([0]), torch.Size([96, 7168, 128]), torch.Size([0]))
/usr/local/lib/python3.12/dist-packages/torch/_dynamo/variables/functions.py:2152: UserWarning: Dynamo detected a call to a `functools.lru_cache`-wrapped function at 'sampling.py:1820'. Dynamo ignores the cache wrapper and directly traces the wrapped function. Silent incorrectness is only a *potential* risk, not something we have observed. Enable TORCH_LOGS=+dynamo for a DEBUG stack trace.

This call originates from:
  File "/usr/local/lib/python3.12/dist-packages/flashinfer/sampling.py", line 1820, in top_k_mask_logits
    return get_sampling_module().top_k_mask_logits(

  torch._dynamo.utils.warn_once(msg)

...

/usr/local/lib/python3.12/dist-packages/torch/_dynamo/variables/functions.py:2152: UserWarning: Dynamo detected a call to a `functools.lru_cache`-wrapped function at 'sampling.py:1067'. Dynamo ignores the cache wrapper and directly traces the wrapped function. Silent incorrectness is only a *potential* risk, not something we have observed. Enable TORCH_LOGS=+dynamo for a DEBUG stack trace.

This call originates from:
  File "/usr/local/lib/python3.12/dist-packages/flashinfer/sampling.py", line 1067, in top_p_sampling_from_probs
    return get_sampling_module().top_p_sampling_from_probs(

  torch._dynamo.utils.warn_once(msg)
[06/09/2026-23:10:58] [TRT-LLM] [I] [_torch][RANK 0] Run warmup with 1 tokens, include 1 generation tokens
[TRT-LLM] [W] [thop][RANK 0] Attention workspace size is not enough, increase the size from 33591808 bytes to 39100416 bytes
[TRT-LLM] [W] [thop][RANK 2] Attention workspace size is not enough, increase the size from 33591808 bytes to 39100416 bytes
[TRT-LLM] [W] [thop][RANK 1] Attention workspace size is not enough, increase the size from 33591808 bytes to 39100416 bytes
[TRT-LLM] [W] [thop][RANK 3] Attention workspace size is not enough, increase the size from 33591808 bytes to 39100416 bytes
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [30,0,0], thread: [96,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [30,0,0], thread: [97,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [30,0,0], thread: [98,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [30,0,0], thread: [99,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [30,0,0], thread: [100,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [30,0,0], thread: [101,0/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [54,0,0], thread: [64,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [54,0,0], thread: [65,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [54,0,0], thread: [66,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [54,0,0], thread: [67,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [54,0,0], thread: [68,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [54,0,0], thread: [69,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [54,0,0], thread: [70,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [54,0,0], thread: [71,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
...
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [10,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [10` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [30,0,0], thread: [120,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [30,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [10,0,0], thread: [5,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [10,0,0], thread: [6,0,0], thread: [121,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [30,0,0], thread: [122,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [10,0,0], thread: [7,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
...
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [55,0,0], thread: [109,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [55,0,0], thread: [110` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [20,0,0], thread: [18,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [20,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [55,0,0], thread: [111,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [55,0,0], thread: [112,0,0] Assertion `srcIndex < srcSelectDimSize,0,0], thread: [19,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [20,0,0], thread: [20,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [55,0,0], thread: [113,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [55,0,0], thread: [114,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [55,0,0` failed.
...
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [11,0,0], thread: [80,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [11,0,0], thread: [81,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515,0], thread: [71,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [21,0,0], thread: [72,0,0] Assertion `srcIndex < srcSelectDimSize: indexSelectSmallIndex: block: [11,0,0], thread: [82,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [11,0,0], thread: [83,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [11,0,0` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [21,0,0], thread: [73,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [21], thread: [84,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [11,0,0], thread: [85,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [11,0,0], thread: [86,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu,0,0], thread: [74,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [21,0,0], thread: [75,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
:1515: indexSelectSmallIndex: block: [11,0,0], thread: [87,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [11,0,0], thread: [88,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [11,0/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [21,0,0], thread: [76,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [21,0], thread: [89,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [11,0,0], thread: [90,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [11,0,0], thread: [91,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [11,0,0], thread: [92,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [11,0,0], thread: [93,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [11,0,0], thread: [94,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [11,0,0], thread: [95,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [42,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
,0,0], thread: [77,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [21,0,0], thread: [78,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [21,0,0], thread: [79,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [21,0,0], thread: [80,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
...
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [20,0,0], thread: [109,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [15,0,0], thread: [43,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [15,0,0], thread: [44,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [15:1515: indexSelectSmallIndex: block: [20,0,0], thread: [110,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [20,0,0], thread: [111,0,0], thread: [45,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [15,0,0], thread: [46,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [15,0,0], thread: [47,0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [20,0,0], thread: [112,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [15,0,0], thread: [48,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [15,0,0], thread: [49,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
:1515: indexSelectSmallIndex: block: [20,0,0], thread: [113,0,0] Assertion `srcIndex < srcSelectDimSize` failed.

], thread: [21,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [25,0,0], thread: [22,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [25,0,0], thread: [23,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [49,0,0], thread: [96,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu: block: [25,0,0], thread: [24,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [25,0,0], thread: [25,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [25,0,0], thread: [26:1515: indexSelectSmallIndex: block: [49,0,0], thread: [97,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [49,0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [42,0,0], thread: [126,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [42,0,0], thread: [127,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[06/09/2026-23:11:11] [TRT-LLM] [E] [executor][RANK 1] Failed to initialize executor on rank 1: CUDA error: device-side assert triggered
Search for `cudaErrorAssert' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

[06/09/2026-23:11:11] [TRT-LLM] [E] [executor][RANK 2] Failed to initialize executor on rank 2: CUDA error: device-side assert triggered
Search for `cudaErrorAssert' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

[06/09/2026-23:11:11] [TRT-LLM] [E] [executor][RANK 1] Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/worker.py", line 302, in worker_main
    worker: GenerationExecutorWorker = worker_cls(
                                       ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/worker.py", line 67, in __init__
    self.setup_engine()
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/base_worker.py", line 286, in setup_engine
    self.engine = _create_py_executor(
                  ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/base_worker.py", line 257, in _create_py_executor
    _executor = create_executor(**args)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor_creator.py", line 914, in create_py_executor
    py_executor = create_py_executor_instance(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/_util.py", line 1726, in create_py_executor_instance
    return PyExecutor(
           ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 517, in __init__
    self.model_engine.warmup(self.resource_manager)
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/model_engine.py", line 707, in wrapper
    return method(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/model_engine.py", line 738, in wrapper
    result = method(self, resource_manager, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/model_engine.py", line 839, in warmup
    self._general_warmup(resource_manager, warmup_requests_configs)
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/model_engine.py", line 871, in _general_warmup
    self._general_warmup_impl(resource_manager, warmup_requests_configs)
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/model_engine.py", line 922, in _general_warmup_impl
    torch.cuda.synchronize()
  File "/usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py", line 1164, in synchronize
    return torch._C._cuda_synchronize()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.AcceleratorError: CUDA error: device-side assert triggered
Search for `cudaErrorAssert' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.


[06/09/2026-23:11:11] [TRT-LLM] [E] [executor][RANK 2] Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/worker.py", line 302, in worker_main
    worker: GenerationExecutorWorker = worker_cls(
                                       ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/worker.py", line 67, in __init__
    self.setup_engine()
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/base_worker.py", line 286, in setup_engine
    self.engine = _create_py_executor(
                  ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/base_worker.py", line 257, in _create_py_executor
    _executor = create_executor(**args)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor_creator.py", line 914, in create_py_executor
    py_executor = create_py_executor_instance(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/_util.py", line 1726, in create_py_executor_instance
    return PyExecutor(
           ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 517, in __init__
    self.model_engine.warmup(self.resource_manager)
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/model_engine.py", line 707, in wrapper
    return method(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/model_engine.py", line 738, in wrapper
    result = method(self, resource_manager, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/model_engine.py", line 839, in warmup
    self._general_warmup(resource_manager, warmup_requests_configs)
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/model_engine.py", line 871, in _general_warmup
    self._general_warmup_impl(resource_manager, warmup_requests_configs)
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/model_engine.py", line 922, in _general_warmup_impl
    torch.cuda.synchronize()
  File "/usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py", line 1164, in synchronize
    return torch._C._cuda_synchronize()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.AcceleratorError: CUDA error: device-side assert triggered
Search for `cudaErrorAssert' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
...
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1515: indexSelectSmallIndex: block: [19,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
terminate called after throwing an instance of 'tensorrt_llm::_v1::common::TllmException'
  what():  [TensorRT-LLM][ERROR] CUDA runtime error in ::cudaStreamDestroy(stream): device-side assert triggered (../include/tensorrt_llm/runtime/cudaStream.h:128)
1       0x74290c98662b void tensorrt_llm::_v1::common::check<cudaError>(cudaError, char const*, char const*, int) + 139
2       0x74290ca43671 std::_Sp_counted_ptr_inplace<tensorrt_llm::runtime::CudaStream, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() + 113
3       0x74292154d1fa std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 44
4       0x7428c7faff80 std::_Sp_counted_ptr_inplace<tensorrt_llm::batch_manager::kv_cache_manager::KVCacheTransferManager, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() + 160
5       0x74292154d1fa std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 44
6       0x7428c7f80e44 tensorrt_llm::batch_manager::kv_cache_manager::WindowBlockManager::~WindowBlockManager() + 644
7       0x7428c7fa819b tensorrt_llm::batch_manager::kv_cache_manager::KVCacheManager::~KVCacheManager() + 315
8       0x7429216465f3 /usr/local/lib/python3.12/dist-packages/tensorrt_llm/bindings.cpython-312-x86_64-linux-gnu.so(+0x67a5f3) [0x7429216465f3]
9             0x575b9e /usr/bin/python3() [0x575b9e]
10            0x5758ec /usr/bin/python3() [0x5758ec]
11            0x59f325 /usr/bin/python3() [0x59f325]
12            0x575b9e /usr/bin/python3() [0x575b9e]
13            0x5758ec /usr/bin/python3() [0x5758ec]
14            0x59f325 /usr/bin/python3() [0x59f325]
15            0x575b9e /usr/bin/python3() [0x575b9e]
16            0x5758ec /usr/bin/python3() [0x5758ec]
17            0x59f325 /usr/bin/python3() [0x59f325]
18            0x579672 /usr/bin/python3() [0x579672]
19            0x59f149 /usr/bin/python3() [0x59f149]
20            0x558d21 /usr/bin/python3() [0x558d21]
21            0x6107d5 /usr/bin/python3() [0x6107d5]
22            0x6107e5 /usr/bin/python3() [0x6107e5]
23            0x6107e5 /usr/bin/python3() [0x6107e5]
24            0x6107e5 /usr/bin/python3() [0x6107e5]
25            0x6107e5 /usr/bin/python3() [0x6107e5]
26            0x5532db /usr/bin/python3() [0x5532db]
27            0x59f2a3 /usr/bin/python3() [0x59f2a3]
28            0x5dfb20 _PyEval_EvalFrameDefault + 37568
29            0x5d582b PyEval_EvalCode + 347
30            0x5d33bc /usr/bin/python3() [0x5d33bc]
31            0x581e4d /usr/bin/python3() [0x581e4d]
32            0x549935 PyObject_Vectorcall + 53
33            0x5d733f _PyEval_EvalFrameDefault + 2783
34            0x6bc772 /usr/bin/python3() [0x6bc772]
35            0x6bc3a2 Py_RunMain + 562
36            0x6bc00d Py_BytesMain + 45
37      0x742cca77d1ca /usr/lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca) [0x742cca77d1ca]
38      0x742cca77d28b __libc_start_main + 139
39            0x657445 _start + 37
[74e3743f0d7c:28032] *** Process received signal ***
[74e3743f0d7c:28032] Signal: Aborted (6)
[74e3743f0d7c:28032] Signal code:  (-6)
[74e3743f0d7c:28032] [ 0] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x45330)[0x742cca798330]
[74e3743f0d7c:28032] [ 1] /usr/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x11c)[0x742cca7f1b2c]
[74e3743f0d7c:28032] [ 2] /usr/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x1e)[0x742cca79827e]
[74e3743f0d7c:28032] [ 3] /usr/lib/x86_64-linux-gnu/libc.so.6(abort+0xdf)[0x742cca77b8ff]
[74e3743f0d7c:28032] [ 4] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xa5ff5)[0x742cc0253ff5]
[74e3743f0d7c:28032] [ 5] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xbb0da)[0x742cc02690da]
[74e3743f0d7c:28032] [ 6] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(__cxa_call_terminate+0x33)[0x742cc02538e6]
[74e3743f0d7c:28032] [ 7] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(__gxx_personality_v0+0x31a)[0x742cc02688ba]
[74e3743f0d7c:28032] [ 8] /usr/lib/x86_64-linux-gnu/libgcc_s.so.1(+0x22b06)[0x742cc01a2b06]
[74e3743f0d7c:28032] [ 9] /usr/lib/x86_64-linux-gnu/libgcc_s.so.1(_Unwind_RaiseException+0x311)[0x742cc01a31f1]
[74e3743f0d7c:28032] [10] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(__cxa_throw+0x44)[0x742cc0269384]
[74e3743f0d7c:28032] [11] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libth_common.so(_ZN12tensorrt_llm3_v16common5checkI9cudaErrorEEvT_PKcS6_i+0xc7)[0x74290c986667]
[74e3743f0d7c:28032] [12] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libth_common.so(_ZNSt23_Sp_counted_ptr_inplaceIN12tensorrt_llm7runtime10CudaStreamESaIvELN9__gnu_cxx12_Lock_policyE2EE10_M_disposeEv+0x71)[0x74290ca43671]
[74e3743f0d7c:28032] [13] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/bindings.cpython-312-x86_64-linux-gnu.so(_ZNSt16_Sp_counted_baseILN9__gnu_cxx12_Lock_policyE2EE10_M_releaseEv+0x2c)[0x74292154d1fa]
[74e3743f0d7c:28032] [14] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZNSt23_Sp_counted_ptr_inplaceIN12tensorrt_llm13batch_manager16kv_cache_manager22KVCacheTransferManagerESaIvELN9__gnu_cxx12_Lock_policyE2EE10_M_disposeEv+0xa0)[0x7428c7faff80]
[74e3743f0d7c:28032] [15] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/bindings.cpython-312-x86_64-linux-gnu.so(_ZNSt16_Sp_counted_baseILN9__gnu_cxx12_Lock_policyE2EE10_M_releaseEv+0x2c)[0x74292154d1fa]
[74e3743f0d7c:28032] [16] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm13batch_manager16kv_cache_manager18WindowBlockManagerD1Ev+0x284)[0x7428c7f80e44]
[74e3743f0d7c:28032] [17] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm13batch_manager16kv_cache_manager14KVCacheManagerD1Ev+0x13b)[0x7428c7fa819b]
[74e3743f0d7c:28032] [18] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/bindings.cpython-312-x86_64-linux-gnu.so(+0x67a5f3)[0x7429216465f3]
[74e3743f0d7c:28032] [19] /usr/bin/python3[0x575b9e]
[74e3743f0d7c:28032] [20] /usr/bin/python3[0x5758ec]
[74e3743f0d7c:28032] [21] /usr/bin/python3[0x59f325]
[74e3743f0d7c:28032] [22] /usr/bin/python3[0x575b9e]
[74e3743f0d7c:28032] [23] /usr/bin/python3[0x5758ec]
[74e3743f0d7c:28032] [24] /usr/bin/python3[0x59f325]
[74e3743f0d7c:28032] [25] /usr/bin/python3[0x575b9e]
[74e3743f0d7c:28032] [26] /usr/bin/python3[0x5758ec]
[74e3743f0d7c:28032] [27] /usr/bin/python3[0x59f325]
[74e3743f0d7c:28032] [28] /usr/bin/python3[0x579672]
[74e3743f0d7c:28032] [29] /usr/bin/python3[0x59f149]
[74e3743f0d7c:28032] *** End of error message ***
terminate called after throwing an instance of 'tensorrt_llm::_v1::common::TllmException'
  what():  [TensorRT-LLM][ERROR] CUDA runtime error in ::cudaStreamDestroy(stream): device-side assert triggered (../include/tensorrt_llm/runtime/cudaStream.h:128)
1       0x7d6350c8662b void tensorrt_llm::_v1::common::check<cudaError>(cudaError, char const*, char const*, int) + 139
2       0x7d6350d43671 std::_Sp_counted_ptr_inplace<tensorrt_llm::runtime::CudaStream, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() + 113
3       0x7d636584d1fa std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 44
4       0x7d630c2aff80 std::_Sp_counted_ptr_inplace<tensorrt_llm::batch_manager::kv_cache_manager::KVCacheTransferManager, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() + 160
5       0x7d636584d1fa std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 44
6       0x7d630c280e44 tensorrt_llm::batch_manager::kv_cache_manager::WindowBlockManager::~WindowBlockManager() + 644
7       0x7d630c2a819b tensorrt_llm::batch_manager::kv_cache_manager::KVCacheManager::~KVCacheManager() + 315
8       0x7d63659465f3 /usr/local/lib/python3.12/dist-packages/tensorrt_llm/bindings.cpython-312-x86_64-linux-gnu.so(+0x67a5f3) [0x7d63659465f3]
9             0x575b9e /usr/bin/python3() [0x575b9e]
10            0x5758ec /usr/bin/python3() [0x5758ec]
11            0x59f325 /usr/bin/python3() [0x59f325]
12            0x575b9e /usr/bin/python3() [0x575b9e]
13            0x5758ec /usr/bin/python3() [0x5758ec]
14            0x59f325 /usr/bin/python3() [0x59f325]
15            0x575b9e /usr/bin/python3() [0x575b9e]
16            0x5758ec /usr/bin/python3() [0x5758ec]
17            0x59f325 /usr/bin/python3() [0x59f325]
18            0x579672 /usr/bin/python3() [0x579672]
19            0x59f149 /usr/bin/python3() [0x59f149]
20            0x558d21 /usr/bin/python3() [0x558d21]
21            0x6107d5 /usr/bin/python3() [0x6107d5]
22            0x6107e5 /usr/bin/python3() [0x6107e5]
23            0x6107e5 /usr/bin/python3() [0x6107e5]
24            0x6107e5 /usr/bin/python3() [0x6107e5]
25            0x6107e5 /usr/bin/python3() [0x6107e5]
26            0x5532db /usr/bin/python3() [0x5532db]
27            0x59f2a3 /usr/bin/python3() [0x59f2a3]
28            0x5dfb20 _PyEval_EvalFrameDefault + 37568
29            0x5d582b PyEval_EvalCode + 347
30            0x5d33bc /usr/bin/python3() [0x5d33bc]
31            0x581e4d /usr/bin/python3() [0x581e4d]
32            0x549935 PyObject_Vectorcall + 53
33            0x5d733f _PyEval_EvalFrameDefault + 2783
34            0x6bc772 /usr/bin/python3() [0x6bc772]
35            0x6bc3a2 Py_RunMain + 562
36            0x6bc00d Py_BytesMain + 45
37      0x7d670ee1f1ca /usr/lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca) [0x7d670ee1f1ca]
38      0x7d670ee1f28b __libc_start_main + 139
39            0x657445 _start + 37
[74e3743f0d7c:28031] *** Process received signal ***
[74e3743f0d7c:28031] Signal: Aborted (6)
[74e3743f0d7c:28031] Signal code:  (-6)
[74e3743f0d7c:28031] [ 0] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x45330)[0x7d670ee3a330]
[74e3743f0d7c:28031] [ 1] /usr/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x11c)[0x7d670ee93b2c]
[74e3743f0d7c:28031] [ 2] /usr/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x1e)[0x7d670ee3a27e]
[74e3743f0d7c:28031] [ 3] /usr/lib/x86_64-linux-gnu/libc.so.6(abort+0xdf)[0x7d670ee1d8ff]
[74e3743f0d7c:28031] [ 4] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xa5ff5)[0x7d66f8227ff5]
[74e3743f0d7c:28031] [ 5] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xbb0da)[0x7d66f823d0da]
[74e3743f0d7c:28031] [ 6] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(__cxa_call_terminate+0x33)[0x7d66f82278e6]
[74e3743f0d7c:28031] [ 7] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(__gxx_personality_v0+0x31a)[0x7d66f823c8ba]
[74e3743f0d7c:28031] [ 8] /usr/lib/x86_64-linux-gnu/libgcc_s.so.1(+0x22b06)[0x7d6704096b06]
[74e3743f0d7c:28031] [ 9] /usr/lib/x86_64-linux-gnu/libgcc_s.so.1(_Unwind_RaiseException+0x311)[0x7d67040971f1]
[74e3743f0d7c:28031] [10] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(__cxa_throw+0x44)[0x7d66f823d384]
[74e3743f0d7c:28031] [11] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libth_common.so(_ZN12tensorrt_llm3_v16common5checkI9cudaErrorEEvT_PKcS6_i+0xc7)[0x7d6350c86667]
[74e3743f0d7c:28031] [12] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libth_common.so(_ZNSt23_Sp_counted_ptr_inplaceIN12tensorrt_llm7runtime10CudaStreamESaIvELN9__gnu_cxx12_Lock_policyE2EE10_M_disposeEv+0x71)[0x7d6350d43671]
[74e3743f0d7c:28031] [13] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/bindings.cpython-312-x86_64-linux-gnu.so(_ZNSt16_Sp_counted_baseILN9__gnu_cxx12_Lock_policyE2EE10_M_releaseEv+0x2c)[0x7d636584d1fa]
[74e3743f0d7c:28031] [14] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZNSt23_Sp_counted_ptr_inplaceIN12tensorrt_llm13batch_manager16kv_cache_manager22KVCacheTransferManagerESaIvELN9__gnu_cxx12_Lock_policyE2EE10_M_disposeEv+0xa0)[0x7d630c2aff80]
[74e3743f0d7c:28031] [15] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/bindings.cpython-312-x86_64-linux-gnu.so(_ZNSt16_Sp_counted_baseILN9__gnu_cxx12_Lock_policyE2EE10_M_releaseEv+0x2c)[0x7d636584d1fa]
[74e3743f0d7c:28031] [16] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm13batch_manager16kv_cache_manager18WindowBlockManagerD1Ev+0x284)[0x7d630c280e44]
[74e3743f0d7c:28031] [17] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm13batch_manager16kv_cache_manager14KVCacheManagerD1Ev+0x13b)[0x7d630c2a819b]
[74e3743f0d7c:28031] [18] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/bindings.cpython-312-x86_64-linux-gnu.so(+0x67a5f3)[0x7d63659465f3]
[74e3743f0d7c:28031] [19] /usr/bin/python3[0x575b9e]
[74e3743f0d7c:28031] [20] /usr/bin/python3[0x5758ec]
[74e3743f0d7c:28031] [21] /usr/bin/python3[0x59f325]
[74e3743f0d7c:28031] [22] /usr/bin/python3[0x575b9e]
[74e3743f0d7c:28031] [23] /usr/bin/python3[0x5758ec]
[74e3743f0d7c:28031] [24] /usr/bin/python3[0x59f325]
[74e3743f0d7c:28031] [25] /usr/bin/python3[0x575b9e]
[74e3743f0d7c:28031] [26] /usr/bin/python3[0x5758ec]
[74e3743f0d7c:28031] [27] /usr/bin/python3[0x59f325]
[74e3743f0d7c:28031] [28] /usr/bin/python3[0x579672]
[74e3743f0d7c:28031] [29] /usr/bin/python3[0x59f149]
[74e3743f0d7c:28031] *** End of error message ***
terminate called after throwing an instance of 'tensorrt_llm::_v1::common::TllmException'
  what():  [TensorRT-LLM][ERROR] CUDA runtime error in ::cudaStreamDestroy(stream): device-side assert triggered (../include/tensorrt_llm/runtime/cudaStream.h:128)
1       0x7201cf98662b void tensorrt_llm::_v1::common::check<cudaError>(cudaError, char const*, char const*, int) + 139
2       0x7201cfa43671 std::_Sp_counted_ptr_inplace<tensorrt_llm::runtime::CudaStream, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() + 113
3       0x7201e454d1fa std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 44
4       0x72018afaff80 std::_Sp_counted_ptr_inplace<tensorrt_llm::batch_manager::kv_cache_manager::KVCacheTransferManager, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() + 160
5       0x7201e454d1fa std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 44
6       0x72018af80e44 tensorrt_llm::batch_manager::kv_cache_manager::WindowBlockManager::~WindowBlockManager() + 644
7       0x72018afa819b tensorrt_llm::batch_manager::kv_cache_manager::KVCacheManager::~KVCacheManager() + 315
8       0x7201e46465f3 /usr/local/lib/python3.12/dist-packages/tensorrt_llm/bindings.cpython-312-x86_64-linux-gnu.so(+0x67a5f3) [0x7201e46465f3]
9             0x575b9e /usr/bin/python3() [0x575b9e]
10            0x5758ec /usr/bin/python3() [0x5758ec]
11            0x59f325 /usr/bin/python3() [0x59f325]
12            0x575b9e /usr/bin/python3() [0x575b9e]
13            0x5758ec /usr/bin/python3() [0x5758ec]
14            0x59f325 /usr/bin/python3() [0x59f325]
15            0x575b9e /usr/bin/python3() [0x575b9e]
16            0x5758ec /usr/bin/python3() [0x5758ec]
17            0x59f325 /usr/bin/python3() [0x59f325]
18            0x579672 /usr/bin/python3() [0x579672]
19            0x59f149 /usr/bin/python3() [0x59f149]
20            0x558d21 /usr/bin/python3() [0x558d21]
21            0x6107d5 /usr/bin/python3() [0x6107d5]
22            0x6107e5 /usr/bin/python3() [0x6107e5]
23            0x6107e5 /usr/bin/python3() [0x6107e5]
24            0x6107e5 /usr/bin/python3() [0x6107e5]
25            0x6107e5 /usr/bin/python3() [0x6107e5]
26            0x5532db /usr/bin/python3() [0x5532db]
27            0x59f2a3 /usr/bin/python3() [0x59f2a3]
28            0x5dfb20 _PyEval_EvalFrameDefault + 37568
29            0x5d582b PyEval_EvalCode + 347
30            0x5d33bc /usr/bin/python3() [0x5d33bc]
31            0x581e4d /usr/bin/python3() [0x581e4d]
32            0x549935 PyObject_Vectorcall + 53
33            0x5d733f _PyEval_EvalFrameDefault + 2783
34            0x6bc772 /usr/bin/python3() [0x6bc772]
35            0x6bc3a2 Py_RunMain + 562
36            0x6bc00d Py_BytesMain + 45
37      0x72058d69a1ca /usr/lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca) [0x72058d69a1ca]
38      0x72058d69a28b __libc_start_main + 139
39            0x657445 _start + 37
[74e3743f0d7c:28033] *** Process received signal ***
[74e3743f0d7c:28033] Signal: Aborted (6)
[74e3743f0d7c:28033] Signal code:  (-6)
[74e3743f0d7c:28033] [ 0] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x45330)[0x72058d6b5330]
[74e3743f0d7c:28033] [ 1] /usr/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x11c)[0x72058d70eb2c]
[74e3743f0d7c:28033] [ 2] /usr/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x1e)[0x72058d6b527e]
[74e3743f0d7c:28033] [ 3] /usr/lib/x86_64-linux-gnu/libc.so.6(abort+0xdf)[0x72058d6988ff]
[74e3743f0d7c:28033] [ 4] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xa5ff5)[0x72036d5a3ff5]
[74e3743f0d7c:28033] [ 5] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xbb0da)[0x72036d5b90da]
[74e3743f0d7c:28033] [ 6] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(__cxa_call_terminate+0x33)[0x72036d5a38e6]
[74e3743f0d7c:28033] [ 7] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(__gxx_personality_v0+0x31a)[0x72036d5b88ba]
[74e3743f0d7c:28033] [ 8] /usr/lib/x86_64-linux-gnu/libgcc_s.so.1(+0x22b06)[0x720588560b06]
[74e3743f0d7c:28033] [ 9] /usr/lib/x86_64-linux-gnu/libgcc_s.so.1(_Unwind_RaiseException+0x311)[0x7205885611f1]
[74e3743f0d7c:28033] [10] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(__cxa_throw+0x44)[0x72036d5b9384]
[74e3743f0d7c:28033] [11] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libth_common.so(_ZN12tensorrt_llm3_v16common5checkI9cudaErrorEEvT_PKcS6_i+0xc7)[0x7201cf986667]
[74e3743f0d7c:28033] [12] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libth_common.so(_ZNSt23_Sp_counted_ptr_inplaceIN12tensorrt_llm7runtime10CudaStreamESaIvELN9__gnu_cxx12_Lock_policyE2EE10_M_disposeEv+0x71)[0x7201cfa43671]
[74e3743f0d7c:28033] [13] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/bindings.cpython-312-x86_64-linux-gnu.so(_ZNSt16_Sp_counted_baseILN9__gnu_cxx12_Lock_policyE2EE10_M_releaseEv+0x2c)[0x7201e454d1fa]
[74e3743f0d7c:28033] [14] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZNSt23_Sp_counted_ptr_inplaceIN12tensorrt_llm13batch_manager16kv_cache_manager22KVCacheTransferManagerESaIvELN9__gnu_cxx12_Lock_policyE2EE10_M_disposeEv+0xa0)[0x72018afaff80]
[74e3743f0d7c:28033] [15] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/bindings.cpython-312-x86_64-linux-gnu.so(_ZNSt16_Sp_counted_baseILN9__gnu_cxx12_Lock_policyE2EE10_M_releaseEv+0x2c)[0x7201e454d1fa]
[74e3743f0d7c:28033] [16] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm13batch_manager16kv_cache_manager18WindowBlockManagerD1Ev+0x284)[0x72018af80e44]
[74e3743f0d7c:28033] [17] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm13batch_manager16kv_cache_manager14KVCacheManagerD1Ev+0x13b)[0x72018afa819b]
[74e3743f0d7c:28033] [18] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/bindings.cpython-312-x86_64-linux-gnu.so(+0x67a5f3)[0x7201e46465f3]
[74e3743f0d7c:28033] [19] /usr/bin/python3[0x575b9e]
[74e3743f0d7c:28033] [20] /usr/bin/python3[0x5758ec]
[74e3743f0d7c:28033] [21] /usr/bin/python3[0x59f325]
[74e3743f0d7c:28033] [22] /usr/bin/python3[0x575b9e]
[74e3743f0d7c:28033] [23] /usr/bin/python3[0x5758ec]
[74e3743f0d7c:28033] [24] /usr/bin/python3[0x59f325]
[74e3743f0d7c:28033] [25] /usr/bin/python3[0x575b9e]
[74e3743f0d7c:28033] [26] /usr/bin/python3[0x5758ec]
[74e3743f0d7c:28033] [27] /usr/bin/python3[0x59f325]
[74e3743f0d7c:28033] [28] /usr/bin/python3[0x579672]
[74e3743f0d7c:28033] [29] /usr/bin/python3[0x59f149]
[74e3743f0d7c:28033] *** End of error message ***
--------------------------------------------------------------------------
Child job 2 terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------