[JunyiXu-nv]

Stacked on #21406 ([TRTLLM-17261][test] Add route, middleware-order and shim-identity snapshots and a CPU test seam for trtllm-serve) and on #21804 ([None][feat] Responses: serving-extension hooks for model-specific request handling): the first commits below belong to those PRs and disappear from this diff when they merge; this PR's own change is the last commit.

Description

A Responses request (/v1/responses) for a Kimi-K3 checkpoint was prepared like any other model's: the generic reasoning_effort / thinking chat-template kwargs, which the K3 template ignores, no tool_choice or response_format control messages, no sampling-parameter policy, and a replayed conversation the K3 renderer could not always bind. Seen in production agent traffic with Kimi-K3 behind a Responses client that replays its whole history every turn: a tool result returned in several items shifted the index of every later result in the turn, and a single result whose call was no longer in the history failed every later turn of the session with a 400.

KimiK3ServingExtension now implements the Responses hooks of ServingExtension, so every Kimi-K3 rule lives in the extension and the Responses path keeps no model-name branch:

• apply_responses_extensions: reasoning.effort maps onto K3's thinking_effort (minimal/low -> low, medium/high -> high, xhigh/max -> max, each the nearest K3 level at or above the one asked for; "none" turns thinking off; no effort leaves the template default). tool_choice "required" / "none" with tools declared and text.format json_object / json_schema (with its schema) become the kwargs the template reads. Explicit client chat_template_kwargs win over derived values, as on chat. The same kwargs steer the kimi_k3 reasoning parser's initial channel, so the parser starts in the channel the prompt opened. The TRTLLM_KIMI_PARAM_POLICY sampling policy (off by default) applies to the fields this endpoint has: top_p unset or 1.0 is pinned to 0.95, other values and an out-of-range temperature are rejected with a 400.
• prepare_responses_history: within a run of tool results each call keeps one result message (the segments of a split result are joined in order; the K3 renderer numbers results by counting messages), and a result no call before it claims is replayed as a user note carrying its output, after its run, so the rest of the run still binds by id. An assistant turn without calls ends the binding, as in the renderer. Nothing a client sent is dropped, and the input conversation is not mutated.

On the Responses surface medium therefore means thinking_effort=high (and minimal -> low, xhigh -> max), as the Kimi-K3 deployment this comes from did; the chat surface keeps leaving those values to the template default. No chat behavior changes in this PR.

The sampling policy is the single implementation chat already uses (enforce_kimi_param_policy), generalized to check only the sampling fields a request type declares: a chat request has them all, a Responses request only temperature and top_p. Chat behavior and messages are unchanged (the existing chat policy tests pass untouched, and the new tests assert the chat messages verbatim).

replay_stored_turn is left on its default for Kimi-K3: the store writer never puts reasoning and text in one stored message, so the "strip only the reasoning" rule and the default replay produce the same conversation on this branch; #21058 makes that rule the generic one.

The handler no longer reports tool_choice="required" as "accepted but not enforced" for a model whose extension enforces it through its template (required_tool_choice_mode() == "template") when tools are declared; other models, grammar-mode extensions (whose grammar the Responses path does not attach) and required without tools are reported as before.

The serve test factory can build a kimi_k3 server: its fake checkpoint is the composite (text + vision) config, which is the shape that resolves to the kimi_k3 model type.

Behavior change: for model_type == "kimi_k3" only. Responses requests now render K3's thinking-effort, tool-choice and response-format control messages from the request fields, replay split or orphan tool results as described above, and (only with TRTLLM_KIMI_PARAM_POLICY=1) pin top_p and reject out-of-policy sampling values. Other models are unchanged. The KV-aware router follows: with the hooks PR it runs the same Responses hooks on a copy of the request, so its routing ids equal the worker's prompt ids. Measured with a probe template mirroring the K3 renderer's control-message placement (32-token routing blocks; before = this change on a router that rendered without the hooks, after = this branch):

case	before: first divergent token / matching blocks	after
effort high	none / 13 of 13	none / 13 of 13
effort medium	1 / 0 of 13	none / 13 of 13
effort minimal	1 / 0 of 13	none / 13 of 13
effort xhigh, none, no effort	(not measured)	none / 13 of 13
tool_choice required (+tools)	418 / 13 of 14	none / 14 of 14
text.format json_schema	418 / 13 of 15	none / 15 of 15
split results	456 / 16 of 18	none / 18 of 18
orphan result	473 / 15 of 21	none / 21 of 21

Interactions with open PRs: #21058 changes the generic stored-turn rule (see above). #21479 edits serialize_tool, not touched here.

Test Coverage

New tests/unittest/serve/test_responses_kimi_k3.py (collected through the unittest/serve entry of l0_cpu.yml), CPU only. The handler tests build a kimi_k3 server with isolated_serve_env and give it a probe chat template on a byte-level tokenizer, so the prompt the fake engine receives shows every kwarg and message the renderer was handed:

• the six effort levels, no effort, and "none" (prompt and parser: the reply is a message, not reasoning; streaming and not);
• tool_choice required/none with tools, auto, and required without tools; text.format json_schema (schema in the prompt), json_object, text; explicit chat_template_kwargs winning;
• the policy pinning top_p (unset and 1.0), rejecting a temperature of 1.5 before any engine call, and staying off by default;
• split and orphan tool results on the K3 server versus a model without an extension;
• the tool_choice="required" warning: absent for K3 with tools, present for K3 without tools, for a generic model and for a grammar-mode extension (captured with a handler on the TRT-LLM logger);
• a stored turn replays its text and not its reasoning across previous_response_id;
• the KV-aware router's ids equal the worker's prompt ids for every case of the table above, and the routed request stays as sent;
• the hooks directly on inline conversations: the derived-kwargs table, the joining rules (separator, trailing newline, empty segment, content parts, later runs), orphan notes and their placement, an assistant turn without calls ending the binding, the input conversation left unmutated, and the policy on both request types.

Verification (CPU host; a test-only plugin declares the C++ op the compiled artifacts predate): the new file (69 passed); tests/unittest/serve (#21406's object.__new__ ratchet and route snapshot included), test_kimi_serve_extensions.py, test_serving_extensions.py, test_render_callers.py, test_responses_input_preprocess.py, test_responses_store_replay.py, test_responses_usage.py, test_responses_serving_extensions.py, test_responses_streaming_events.py, api_stability/test_serve_routes.py and the two Responses routing tests of disaggregated/test_router.py (690 passed together); the route snapshot regenerator reports no differences. Mutation gate: sixteen planted inverses of the change (a wrong effort level, "none" ignored, tool_choice derived without tools, the schema replaced, derived values beating explicit kwargs, the policy not applied, the policy reading undeclared fields, no merge, no orphan notes, a lost separator, a note before its run, the warning not suppressed, the warning suppressed without tools, call ids surviving an assistant turn without calls, the warning suppressed for grammar mode, the merge mutating its input) each turn the new tests red. Hypothesis properties over generated conversations and parameters (no result text lost, every remaining result binds by id within its run, one kept result per call then the run's notes between the same non-tool messages, idempotence, the policy agreeing between chat and Responses on the shared fields, effort never derived lower than asked, tool_choice derived only with tools, exactly one newline between joined segments) pass; the mutations on the paths they state break them, the handler-warning mutations and the schema value are caught by the example tests. Rendering through the real K3 checkpoint renderer needs the checkpoint files and is left to a GPU end-to-end run.

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

Reviewers by CODEOWNERS: trt-llm-runtime-devs (tensorrt_llm/serve/), trt-llm-devs (tests/unittest/serve/, tests/test_common/). tensorrt_llm/serve/openai_server.py and tensorrt_llm/serve/extensions/ are in the multi-GPU trigger list of jenkins/L0_MergeRequest.groovy, so this PR needs the ci: full pre-merge approved label to run the full pre-merge pipeline.

🤖 Generated with Claude Code