You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug]: trtllm-serve /v1/chat/completions rejects the standard OpenAI field prompt_cache_key with 400 #20367
or with the OpenAI Python client: client.chat.completions.create(..., prompt_cache_key="session-1234").
Expected behavior
The request is served. prompt_cache_key is part of OpenAI's Chat Completions request body (a hint for prompt-cache routing), and clients built on it may set it on every request of a session.
Actual behavior
The request fails validation with extra_forbidden ("Extra inputs are not permitted") for prompt_cache_key, because ChatCompletionRequest inherits extra="forbid" from OpenAIBaseModel and has no such field. ResponsesRequest already accepts it (it switched to extra="allow", and the comment there names prompt_cache_key as one of the fields real clients send), so the same client works against /v1/responses but not against /v1/chat/completions.
Additional notes
KV cache reuse in TensorRT-LLM needs no client hint, and cache_salt is the field that isolates cached prefixes, so the field can be accepted and ignored, the same way parallel_tool_calls and user are. PR to follow.
Activity