[Bug]: cross Kv sizing with whisper #19514

System Info

Cross-attention KV cache for encoder-decoder models is sized from max_input_len (default 1024), so Whisper's 1500 encoder positions never fit

Found while verifying #19450 on hardware (follow-up to #18609; details and the run matrix are in my comments on both). Once the scheduler no longer crashes, every
openai/whisper-large-v3 request on the PyTorch backend with default arguments fails with:

tensorrt_llm.executor.utils.RequestError: The max KV cache length of input sequences (1500) exceeds the KV cache manager's maximum supported length (1024).
  File ".../tensorrt_llm/_torch/attention_backend/trtllm.py", line 754, in prepare
    assert max_kv_len <= self.kv_cache_manager.max_seq_len
  (reached via model_engine.py prepare_cross_metadata -> _prepare_enc_dec_cross_attn_inputs)
[TRT-LLM] [E] [_torch] Error in event loop: Sampling failed

Cause

_get_cross_kv_cache_layout() in tensorrt_llm/_torch/pyexecutor/_util.py computes the cross pool's
max_seq_len as: engine max_seq_len, overridden by llm_args.max_input_len (default 1024,
llm_args.py:4675), then min()'d with an encoder limit read from the model config. The config
attribute list (max_encoder_input_len, encoder_max_input_length, max_encoder_position_embeddings,
encoder_max_position_embeddings, max_position_embeddings, n_positions) does not include
max_source_positions, which is how WhisperConfig declares its encoder length (1500). Net effect:
the cross pool is sized to a decoder-side prompt limit, and the encoder output of every 30 s clip is
larger than the pool.

Fix (tested)

As @100milliongold noted on #19450, adding max_source_positions to the lookup alone would not help:
max_input_len overwrites the length unconditionally and the encoder limit is then applied with
min(), so min(1024, 1500) is still 1024. The override is the defect. The change below therefore
treats the encoder's declared position limit as authoritative for the cross pool (adding
max_source_positions so Whisper declares one) and only falls back to max_input_len when the
model declares no encoder length. Diff against v1.3.0rc25 (applies cleanly to main):

--- a/tensorrt_llm/_torch/pyexecutor/_util.py	2026-09-20 08:34:45.206197754 -0500
+++ b/tensorrt_llm/_torch/pyexecutor/_util.py	2026-09-20 08:34:45.457201610 -0500
@@ -1798,18 +1798,26 @@
             head_dim = encoder_hidden_size // encoder_num_heads
 
         max_seq_len = fallback_max_seq_len or self._max_seq_len
-        max_input_len = getattr(self._llm_args, "max_input_len", None)
-        if isinstance(max_input_len, int) and max_input_len > 0:
-            max_seq_len = max_input_len
         encoder_limit = self._get_config_int_attr(
             config,
             ("max_encoder_input_len", "encoder_max_input_length",
              "max_encoder_position_embeddings",
-             "encoder_max_position_embeddings", "max_position_embeddings",
-             "n_positions"),
+             "encoder_max_position_embeddings", "max_source_positions",
+             "max_position_embeddings", "n_positions"),
         )
         if encoder_limit is not None:
-            max_seq_len = min(max_seq_len, encoder_limit)
+            # The cross pool holds the encoder output, so the encoder's own
+            # position limit is the authoritative length. Do not let
+            # max_input_len (a decoder-side prompt limit, default 1024)
+            # shrink it below that: Whisper's encoder emits
+            # max_source_positions=1500 frames for every clip, and the
+            # request would fail in prepare_cross_metadata with "max KV cache
+            # length of input sequences (1500) exceeds ... (1024)".
+            max_seq_len = encoder_limit
+        else:
+            max_input_len = getattr(self._llm_args, "max_input_len", None)
+            if isinstance(max_input_len, int) and max_input_len > 0:
+                max_seq_len = max_input_len
 
         return num_layers, num_kv_heads, head_dim, max_seq_len

Verification

RTX 5070 Ti (sm_120), nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc25 with #19450 applied, 30 s 16 kHz
clip, LLM API directly:

Workaround for anyone hitting this today: pass max_input_len=1500 to LLM(...).

Side note for other Whisper users: do not pass max_seq_len=448 (my original repro did). The model's infer_max_seq_len() correctly returns 1500 for the encoder pass; forcing 448 makes the engine's encoder-token limit max_batch_size * 448, which fails outright at batch 2 (microBatchScheduler.cpp:365: The number of encoder tokens (1500) exceeds the limit value (896)) and at batch 8 slips through but silently degrades the transcript (a phrase dropped, "chest discomfort ... relieved by rest" became "hypotension" on the same clip, reproducible across runs). A user-supplied max_seq_len below max_source_positions should probably be rejected for this model. Also, once the cross pool is sized correctly, free_gpu_memory_fraction=0.16 with max_batch_size=8 is too small on a 16 GB card (No free block found at executor init); 0.30 with batch 2, or 0.30 with batch 8, both work.

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

.

Expected behavior

.

actual behavior

.

additional notes

.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.