[Bug]: Whisper (PyTorch backend) fails with RuntimeError: bad optional access in the capacity scheduler — TensorRT-LLM 1.3.0rc25, RTX 5070 Ti (sm_120)

System Info

X86
Sm120
Whisper large v3

Who can help?

No response

Information

The official example scripts
My own modified scripts

Tasks

An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
My own task or dataset (give details below)

Reproduction

[Unable to encode/decode ](scheduler.py:462 schedule_request -> self.impl(...) -> RuntimeError: bad optional access)

Expected behavior

Active endpoint

actual behavior

HTTP 404

additional notes

Whisper support missing

Before submitting a new issue...

Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

Whisper (PyTorch backend) fails with RuntimeError: bad optional access in the capacity scheduler — TensorRT-LLM 1.3.0rc25, RTX 5070 Ti (sm_120)

Summary

openai/whisper-large-v3 loads successfully under the PyTorch backend in
TensorRT-LLM 1.3.0rc25, but every generation request fails with
RuntimeError: bad optional access raised from the C++ capacity scheduler,
before the model forward pass is reached.

The failure is identical whether the request arrives over trtllm-serve's
/v1/chat/completions or through the Python LLM API directly, which rules out
the OpenAI serving layer as the cause.

Environment


Container	`nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc25`
TensorRT-LLM	1.3.0rc25
PyTorch	2.12.0a0+5aff3928d8.nv26.05
CUDA (torch)	13.2
GPU	NVIDIA GeForce RTX 5070 Ti, 16303 MiB
Compute capability	12.0 (sm_120)
Driver	610.43.02
Host OS	Linux 7.0.0-30-generic x86_64
Model	`openai/whisper-large-v3` (HF safetensors, fp16)

torch.cuda.get_arch_list() reports sm_120 and compute_120 are compiled in,
so this does not appear to be a missing-architecture problem.

Reproduction

Minimal script (the __main__ guard is required — the LLM API re-executes the
module in MPI workers):

import time, wave
import numpy as np

def main():
    from tensorrt_llm import LLM, SamplingParams
    from tensorrt_llm.llmapi import KvCacheConfig

    with wave.open("clip30.wav", "rb") as w:          # 30 s, 16 kHz, mono
        sr, n = w.getframerate(), w.getnframes()
        raw = w.readframes(n)
    audio = np.frombuffer(raw, dtype=np.int16).astype(np.float32) / 32768.0

    llm = LLM(model="openai/whisper-large-v3", backend="pytorch",
              max_batch_size=8, max_seq_len=448,
              kv_cache_config=KvCacheConfig(free_gpu_memory_fraction=0.16,
                                            cross_kv_cache_fraction=0.5))
    print("LLM constructed OK")

    outs = llm.generate([{"prompt": "", "multi_modal_data": {"audio": (audio, sr)}}],
                        SamplingParams(max_tokens=220, temperature=0.0))
    print(outs[0].outputs[0].text)

if __name__ == "__main__":
    main()

Run:

docker run --rm --gpus '"device=0"' -v ~/.cache/huggingface:/root/.cache/huggingface \
  --entrypoint bash nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc25 \
  -c 'python -u /opt/direct_test.py'

Expected

A transcript of the 30-second audio clip.

Actual

The model constructs successfully, then generation fails:

LLM constructed OK
[TRT-LLM] [W] [thop][RANK 0] Attention workspace size is not enough,
    increase the size from 0 bytes to 33576192 bytes
[TRT-LLM] [E] [_torch] Error in event loop: bad optional access
[TRT-LLM] [E] [_torch] Traceback (most recent call last):
  File ".../tensorrt_llm/_torch/pyexecutor/py_executor.py", line 1159, in _event_loop_wrapper
    self.event_loop()
  File ".../tensorrt_llm/_torch/pyexecutor/py_executor.py", line 5007, in _executor_loop_overlap
    scheduled_batch, iter_stats = self._prepare_and_schedule_batch()
  File ".../tensorrt_llm/_torch/pyexecutor/py_executor.py", line 3797, in _prepare_and_schedule_batch
    scheduled_batch, ... = self._schedule(
  File ".../tensorrt_llm/_torch/pyexecutor/scheduler/scheduler.py", line 6247, in _schedule
    scheduler_output = self.scheduler.schedule_request(
  File ".../tensorrt_llm/_torch/pyexecutor/scheduler/scheduler.py", line 547, in schedule_request
    self.capacity_scheduler.schedule_request(active_requests)
  File ".../tensorrt_llm/_torch/pyexecutor/scheduler/scheduler.py", line 462, in schedule_request
    return self.impl(
RuntimeError: bad optional access

The exception originates in the C++ capacity scheduler (self.impl(...)), i.e.
during scheduling, before the model forward pass. This suggests an optional
field required for encoder-decoder requests (an encoder output length or similar)
is unset by the time the scheduler inspects the request.

The preceding Attention workspace size is not enough, increase the size from 0 bytes warning may be related — a workspace sized at 0 bytes looks like the same
class of uninitialised state.

What was ruled out

Not the serving layer. The same error occurs via the Python LLM API with no
HTTP, no chat template and no OpenAI content parsing involved.
Not the input format. Passing a bare ndarray produces a clear, different
error (Unsupported audio item for Whisper: expected a file path/URL, an (array, sample_rate) tuple, or a dict with 'array'/'sampling_rate'), so the
(array, sample_rate) form is being accepted by the input processor.
Not a missing GPU architecture. sm_120/compute_120 are in
torch.cuda.get_arch_list(), and the model loads and allocates KV cache normally
(self 0.90 GiB + cross 0.83 GiB at free_gpu_memory_fraction=0.16,
cross_kv_cache_fraction=0.5).
Not KV misconfiguration. Omitting cross_kv_cache_fraction fails earlier and
explicitly (ValueError: Encoder-decoder models require kv_cache_config.cross_kv_cache_fraction to be set), so that path is correct here.

Secondary observations

trtllm-serve exposes no /v1/audio/transcriptions or /v1/audio/translations
route (both return 404); Whisper is only reachable via /v1/chat/completions.
No Whisper chat template ships in examples/serve/chat_templates/, so an audio
chat request fails with No chat template found for the given tokenizer and tools
until the user supplies one via --chat_template.
The container ships no Whisper examples, tests or documentation — only
tensorrt_llm/_torch/models/modeling_whisper.py.
The legacy TensorRT-engine Whisper example was removed after v1.2.1, and the
PyTorch implementation is absent from v1.2.1, so no release currently provides a
working Whisper path.

References

Thank you for responding. I don't have write access to this repository, so GitHub won't let me assign the issue. Could you please self-assign or have another maintainer assign it to you? Thanks! Ravo42

…

On Fri, Sep 4, 2026 at 3:48 AM Chandan Kumar ***@***.***> wrote: *modelpath-dev* left a comment (NVIDIA/TensorRT-LLM#18609) <#18609 (comment)> I'll work on this. Could you assign it to me? I'll pin a regression test (or stabilize the flake) and keep the change small. — Reply to this email directly, view it on GitHub <#18609?email_source=notifications&email_token=CNJNBL7BVUCT25MGR3MSKXD5NJ6XTA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNJTHAYDGNBZGM2KM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-5538034934>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/CNJNBL5HTMQXDL5M5CGLOS35NJ6XTAVCNFSNUABFKJSXA33TNF2G64TZHM3DOOJTGY3DANJRHNEXG43VMU5TKMZSHAZTIOJWGY42C5QC> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/CNJNBL236HVSNC4Q2CGKGJT5NJ6XTA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNJTHAYDGNBZGM2KM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KUZTPN52GK4S7NFXXG> and Android <https://github.com/notifications/mobile/android/CNJNBLZILNYB3PTGTZIF5BD5NJ6XTA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNJTHAYDGNBZGM2KM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA>. Download it today! You are receiving this because you authored the thread.Message ID: ***@***.***>

I traced this from the stack you posted. It is not SM120-specific and not audio-specific: any encoder-decoder request whose encoder side is features rather than tokens hits it, with the default KV cache settings.

Where it throws

self.impl(...) at scheduler.py:462 reaches cpp/tensorrt_llm/batch_manager/capacityScheduler.cpp, which dereferences the encoder unique tokens at three places:

line	guard
64	`crossKvCacheManager && crossKvCacheManager->isEnableBlockReuse()`
347	the same, plus `!isVariableWindow()`
355	the same

auto uniqueTokens = *(req->getEncoderUniqueTokens().value());

None of the three checks getEncoderUniqueTokens().has_value(). They only check that a cross KV cache manager exists and that block reuse is on.

Why the optional is empty for Whisper

mEncoderUniqueTokens is derived from encoder tokens only (llmRequest.h:2272):

if (mEncoderTokens.has_value() && mEncoderTokens.value())
{
    ...
    mEncoderUniqueTokens = encoderUniqueTokens;
}

Whisper feeds the encoder features, not tokens. modeling_whisper.py:788 passes "encoder_input_features": input_features.contiguous(), and llm_request.py:928-934 keeps encoder_input_features on the Python side while encoder_input_tokens stays None. So mEncoderUniqueTokens is never populated and stays nullopt.

The header right below the getter already acknowledges both shapes of encoder input (llmRequest.h:663):

/// @return Encoder input length, or nullopt when this request has no encoder side.
[[nodiscard]] std::optional<SizeType32> tryGetEncoderInputLen() const
{
    if (mEncoderInputFeatures.has_value()) { ... }
    if (getEncoderTokens().has_value())    { ... }
    return std::nullopt;
}

So the codebase knows the encoder side can be features; the scheduler's prefix-reuse path is the place that assumes tokens.

Why every request fails

enable_block_reuse defaults to True (llm_args.py:4106), and an encoder-decoder run requires cross_kv_cache_fraction, which is what creates the cross manager. Both guard conditions are therefore satisfied on a default Whisper run, on any architecture. Your SM120 box is not what selects this path.

That also explains why it throws before the forward pass, and why the serving layer made no difference.

Two checks you could run

KvCacheConfig(enable_block_reuse=False, cross_kv_cache_fraction=0.5) should get you past the scheduler. If it does, that confirms the path and gives you a workaround meanwhile.
If it still throws with reuse off, my reading is incomplete and I would like to know.

Fix

The conservative shape is to skip cross prefix-reuse analysis when the encoder unique tokens are absent, rather than throwing: prefix reuse matches on token prefixes, and a features-based encoder has no tokens to match, so there is nothing for analyzePrefixReuse to do there.

I am happy to open that PR, but I want to say plainly what I can and cannot stand behind: the diagnosis above is from reading main, and I have no SM120 or Whisper setup, so I have not reproduced your failure or verified a patch against it. If you can run check 1 above, that would turn my reading into something confirmed.

Your report made this findable: the exact frame, self.impl(...), and ruling out the serving layer and the input format is what narrowed it to the C++ scheduler.

@ravo42-maker

@ravo42-maker I opened #19450 with the guard.

It adds && req->getEncoderUniqueTokens().has_value() at the three sites, so a features-based encoder leaves the cross prefix-reuse summary unset instead of dereferencing an empty optional. That unset state is what beneficialToSkip and registerContributedBlocks already handle, and what they already see when there is no cross manager or block reuse is off.

The check I asked about still matters, and now for two reasons. If KvCacheConfig(enable_block_reuse=False, cross_kv_cache_fraction=0.5) gets you past the scheduler, it confirms the diagnosis and gives you something to run today without waiting for the patch. If it does not, then #19450 is aimed at the wrong thing and I would rather know before it is reviewed.

I still have not run any of this: no SM120 and no Whisper setup here, so the PR says plainly that it is unverified on hardware.

Tested on the original hardware today (RTX 5070 Ti / sm_120, nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc25, openai/whisper-large-v3, 30 s 16 kHz mono clip, LLM API directly). Five runs, one variable at a time:

#	`enable_block_reuse`	`max_input_len`	`max_seq_len`	result
1	True (default)	1024 (default)	448	`RuntimeError: bad optional access` in the capacity scheduler (the original report)
2	False	1024 (default)	448	gets past the scheduler, then `RequestError: The max KV cache length of input sequences (1500) exceeds the KV cache manager's maximum supported length (1024)` from `attention_backend/trtllm.py:754` via `prepare_cross_metadata`, followed by `AssertionError: Sampling failed`
3	False	1024	2048	`No free block found. This shouldn't happen! (evictionPolicy.cpp:160)` at executor init
4	False	1500	unset (`infer_max_seq_len()` = 1500)	works — correct transcript, 30 s clip in 3.5 s (cold, 8.7x realtime)
5	True (default)	1500	unset	`bad optional access` again

So the diagnosis holds: with everything else sized correctly (run 5 vs run 4), block reuse on vs off is exactly the difference between the scheduler crash and a working transcription. #19450 is aimed at the right thing.

Run 2 exposes a second, independent problem that anyone will hit right after #19450 lands: _get_cross_kv_cache_layout() in _torch/pyexecutor/_util.py sizes the cross-attention KV pool from llm_args.max_input_len, whose default is 1024 (llm_args.py:4675), and Whisper's encoder emits max_source_positions = 1500 positions. The config-attribute fallback list there (max_encoder_input_len, encoder_max_input_length, max_encoder_position_embeddings, encoder_max_position_embeddings, max_position_embeddings, n_positions) does not include max_source_positions, so nothing corrects it. The cross pool should be sized from the encoder length, not a decoder input limit. Workaround until then: pass max_input_len=1500 (and don't pass max_seq_len=448 — the model's infer_max_seq_len() already returns 1500 for the encoder pass; forcing 448 is what my original repro did wrong). Happy to open that as a separate issue if you'd prefer it tracked apart from this one.

Update — built and tested the PR itself. I applied the capacityScheduler.cpp hunk of #19450 (head 3b82b78) onto the v1.3.0rc25 tag (785c948; it applies cleanly, all three sites are identical there), built a wheel in nvcr.io/nvidia/tensorrt-llm/devel:1.3.0rc25 with scripts/build_wheel.py --cuda_architectures "120-real" --fast_build, and installed it over the release container with pip install --no-deps --force-reinstall. Same GPU, same clip, same scripts:

#	`enable_block_reuse`	`max_input_len`	rc25 stock	rc25 + #19450
5	True (default)	1500	`bad optional access`	works — same transcript as run 4, 30 s clip in 3.95 s cold (7.6x realtime)
1	True (default)	1024 (default)	`bad optional access`	past the scheduler; then the cross-KV sizing error (`1500 exceeds 1024`) + `Sampling failed`

So the guard does what it says on sm_120: block reuse stays on, the request reaches the model, and Whisper transcribes. Run 1 is the second, independent bug described above, now the only thing between a stock config and a working transcription.

[None][fix] Size the cross-attention KV cache from the encoder's decl…

fix NVIDIA#18609: do not require encoder tokens for cross prefix reuse

Navigation Menu

Repository navigation

[Bug]: Whisper (PyTorch backend) fails with RuntimeError: bad optional access in the capacity scheduler — TensorRT-LLM 1.3.0rc25, RTX 5070 Ti (sm_120) #18609

System Info

Who can help?

Information

Tasks

Reproduction

Expected behavior

actual behavior

additional notes

Before submitting a new issue...

Summary

Environment

Reproduction

Expected

Actual

What was ruled out

Secondary observations

References

Where it throws

Why the optional is empty for Whisper

Why every request fails

Two checks you could run

Fix

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug]: Whisper (PyTorch backend) fails with RuntimeError: bad optional access in the capacity scheduler — TensorRT-LLM 1.3.0rc25, RTX 5070 Ti (sm_120) #18609

Description

System Info

Who can help?

Information

Tasks

Reproduction

Expected behavior

actual behavior

additional notes

Before submitting a new issue...

Activity

ravo42-maker commented on Sep 3, 2026

Summary

Environment

Reproduction

Expected

Actual

What was ruled out

Secondary observations

References

ravo42-maker commented on Sep 4, 2026

100milliongold commented on Sep 20, 2026

Where it throws

Why the optional is empty for Whisper

Why every request fails

Two checks you could run

Fix

100milliongold commented on Sep 20, 2026

ravo42-maker commented on Sep 20, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions