[codebreaker77]

Description

Add POST /v1/audio/transcriptions for short audio clips using the served Whisper model. The route accepts multipart file/model, returns JSON or plain text, and supports optional language and temperature through the existing Whisper preprocessing/generation path.

This is a bounded first part of #19994. Long-form windowing, timestamped/verbose output, translation, prompt conditioning and streaming remain outside this change; unsupported options return explicit errors. Upload size and the model's audio window are validated before waveform allocation.

Test Coverage

The branch now includes upstream main fec3d91919501d68a802aa616e8903c3d5ffcef0 through signed merge commit 80c01ba45c11f23e0bf2d054980adf15fcf3588f. Both transcription handlers and upstream's new iteration-stat recording method are retained.

• 31 direct CPU request-contract tests and 7 supplementary server-method checks pass (38 total) on the refreshed code. They cover multipart/options validation, decoding/resampling, language/input conversion, JSON/text output, backend errors, cancellation/disconnect cleanup and route registration.
• The supplementary local runner extracts the actual server methods, substitutes SamplingParams and a component factory, and bypasses native package initialization. These are component results, not normal-package constructor/inference validation.
• The shipped server tests now build through upstream's serve_env/AggConfig factory and inspect routes with effective_routes(). They no longer bypass the server constructor or register a duplicate test-only transcription route. Their native constructor path still needs a matching build.
• Upstream's unittest/serve CPU-list entry covers both test files; duplicate individual entries were removed.
• Applicable formatting/lint, legacy-config, vendor-source and version-record checks pass. The commit skips full-tree test-list validation because the sparse checkout omits integration files, and the advisory Pyrefly hook because its Windows invocation fails to resolve the shell-script path. This is not a full pre-commit pass.

The earlier reporter's SM120 validation and test kit apply to the earlier branch revision, not this refreshed merge. No new GPU inference is claimed.

Remaining validation before readiness

Normal-package tests and route-reference generation are blocked locally by the missing native TensorRT-LLM Windows libraries; the Docker Linux engine is unavailable. In a matching build, run the two transcription test files, regenerate tests/unittest/api_stability/references/trtllm_serve_routes.yaml with PYTHONPATH=tests python3 -m test_common.serve.routes --write, and run the route/API gates. The generated reference has not been edited by hand. The new endpoint requires the serving API review described in serve/AGENTS.md, as well as required project CI and refreshed model validation. This PR remains a draft.

PR Checklist

The description, scope, tests and documentation have been reviewed as applicable. No dependency, CODEOWNERS or architecture-diagram change is introduced. Validation limits and outstanding API/reference work are explicit above.

• Please check this after reviewing the above items as appropriate for this PR.

[ravo42-maker]

Attaching the sm_120 reproduction kit for d50e1d20 as requested in #19994. Everything below is what produced the numbers in the issue report. Hardware: RTX 5070 Ti (16 GB), driver for CUDA 13.x, Docker with the NVIDIA runtime.

1. Serving config (whisper.yaml). Same as the one in docs/source/commands/trtllm-serve/audio-transcriptions.md, with the two values a 16 GB card needs: max_batch_size: 2 and free_gpu_memory_fraction: 0.30 (from #19514 / #19302). On main the explicit enable_block_reuse: false is no longer required (#19302 auto-disables reuse for encoder-decoder), but it is harmless and the 1.3.0rc25 release image still needs it.

attn_backend: TRTLLM
dtype: float16
max_batch_size: 2
max_input_len: 1500
max_seq_len: 2048
max_num_tokens: 3000
enable_chunked_prefill: false
kv_cache_config:
  enable_block_reuse: false
  free_gpu_memory_fraction: 0.30
  cross_kv_cache_fraction: 0.5
scheduler_config:
  use_python_scheduler: true

2. Test audio. Synthetic, no real speech. espeak-ng output resampled to the Whisper rate, cut to exact lengths:

espeak-ng -v en-us -s 160 -w raw.wav "$(cat some_paragraph.txt)"        # >45 s of speech
ffmpeg -y -i raw.wav -ac 1 -ar 16000 -sample_fmt s16 -t 30 clip30.wav   # exactly one window
ffmpeg -y -i raw.wav -ac 1 -ar 16000 -sample_fmt s16 -t 16 clip16.wav
ffmpeg -y -i raw.wav -ac 1 -ar 16000 -sample_fmt s16 -t 45 clip45.wav   # must be rejected

3a. Serve from this branch built for sm_120 (what the main numbers came from):

# build host: nvcr.io/nvidia/tensorrt-llm/devel:1.4.0rc0 (torch 2.14 nv26.08), CPU only, ~2 h
git lfs pull
python3 scripts/build_wheel.py --cuda_architectures "120-real" --fast_build --job_count 16 --clean
# serve (same image), offline HF cache mounted at /root/.cache/huggingface
pip install build/tensorrt_llm-1.4.0rc1-cp312-cp312-linux_x86_64.whl
trtllm-serve openai/whisper-large-v3 --backend pytorch --config whisper.yaml --host 0.0.0.0 --port 8000

Two caveats, neither caused by this PR: (a) with a fully offline cache the newer huggingface_hub raised IncompleteSnapshotError on the model id, so I passed the snapshot directory as the model and --served_model_name openai/whisper-large-v3; (b) the link of libtensorrt_llm.so failed with undefined reference to c10::cuda::getCurrentCUDAStream from tinygemm2 under --no-undefined. Local workaround, identical tree otherwise:

--- a/cpp/tensorrt_llm/runtime/utils/CMakeLists.txt
+++ b/cpp/tensorrt_llm/runtime/utils/CMakeLists.txt
-target_link_libraries(pg_utils PUBLIC torch_cpu c10)
+target_link_libraries(pg_utils PUBLIC torch_cpu c10 c10_cuda)

3b. Serve from the 1.3.0rc25 release image with only this PR's tensorrt_llm/serve/ hunks overlaid (what the rc25 numbers came from):

DP=/usr/local/lib/python3.12/dist-packages
cd $DP && patch -p1 -F2 < pr20616-serve-only.diff      # audio_transcriptions.py + openai_server.py hunks
sed -i 's/^from fastapi import Body, FastAPI, Request$/from fastapi import Body, FastAPI, HTTPException, Request/' $DP/tensorrt_llm/serve/openai_server.py   # rc25 lacks this import; main has it
trtllm-serve openai/whisper-large-v3 --backend pytorch --config whisper.yaml --host 0.0.0.0 --port 8000

4. Client smoke script. Prints status, wall time and the first 600 bytes of each response.

#!/usr/bin/env bash
set -u
B=${B:-http://localhost:8000}; M=openai/whisper-large-v3; U="$B/v1/audio/transcriptions"
req() { local name="$1"; shift; local t0=$(date +%s.%N)
  local out; out=$(curl -s -m 300 -w '\n__HTTP__%{http_code}' "$@"); local t1=$(date +%s.%N)
  printf '\n### %s  -> HTTP %s  %.2fs\n%s\n' "$name" "${out##*__HTTP__}" "$(echo "$t1 - $t0" | bc)" "$(echo "${out%__HTTP__*}" | head -c 600)"; }
req models            "$B/v1/models"
req json_en_t0_cold   -F model=$M -F file=@clip30.wav -F language=en -F response_format=json -F temperature=0 "$U"
req json_en_t0_warm   -F model=$M -F file=@clip30.wav -F language=en -F response_format=json -F temperature=0 "$U"
req json_en_t0_repeat -F model=$M -F file=@clip30.wav -F language=en -F response_format=json -F temperature=0 "$U"
req text_en_t0        -F model=$M -F file=@clip30.wav -F language=en -F response_format=text "$U"
req json_nolang       -F model=$M -F file=@clip30.wav -F response_format=json "$U"
req json_en_t02       -F model=$M -F file=@clip30.wav -F language=en -F temperature=0.2 "$U"
req clip16s           -F model=$M -F file=@clip16.wav -F language=en "$U"
req err_45s           -F model=$M -F file=@clip45.wav -F language=en "$U"
req err_timestamp_granularities -F model=$M -F file=@clip30.wav -F 'timestamp_granularities[]=word' "$U"
req err_verbose_json  -F model=$M -F file=@clip30.wav -F response_format=verbose_json "$U"
req err_not_audio     -F model=$M -F file=@whisper.yaml "$U"
req err_unknown_model -F model=nope -F file=@clip30.wav "$U"
req err_unknown_field -F model=$M -F file=@clip30.wav -F hotwords=foo "$U"
req err_stream        -F model=$M -F file=@clip30.wav -F stream=true "$U"
t0=$(date +%s.%N)
curl -s -m 300 -F model=$M -F file=@clip30.wav -F language=en "$U" -o c1.json &
curl -s -m 300 -F model=$M -F file=@clip16.wav -F language=en "$U" -o c2.json &
wait; printf '\n### concurrent_2  %.2fs\n%s\n%s\n' "$(echo "$(date +%s.%N) - $t0" | bc)" "$(cat c1.json)" "$(cat c2.json)"

Expected on d50e1d20 (both 3a and 3b): the first eight requests return 200 with identical greedy text for json/text/nolang/warm/repeat, temperature=0.2 differs slightly, the next seven return 400/400/400/400/404/400/400 with the messages listed in the issue, and the concurrent pair both return 200. Timings on this card from the source build: cold 3.5 s, warm 0.37 s, 16 s clip 0.26 s, two concurrent 0.44 s wall.

5. CPU tests (same command as the PR description), run inside the serving container with the two test files and tensorrt_llm/serve/audio_transcriptions.py present at their repo-relative paths: 38 passed on rc25 + overlay.

Happy to re-run on any later commit once CI or maintainer feedback changes the branch.