Repository navigation
[None][refactor] Move the Responses stream lifecycle into serve/_base/responses_stream.py - #22176
Draft
JunyiXu-nv wants to merge 29 commits into
Draft
JunyiXu-nv wants to merge 29 commits into
JunyiXu-nv wants to merge 29 commits into
Conversation
…am with context_length_exceeded A streamed /v1/responses request whose prompt exceeds the context window now gets the stream OpenAI sends for it: response.created, response.in_progress, error, then response.failed with error.code context_length_exceeded, after which the stream ends normally. Clients act on that event: the Codex CLI compacts its history on it, while it treats an HTTP 400 as an invalid request and an aborted stream as a transport error that it retries. Before, the answer depended on where the executor rejected the prompt: - in a worker process (raised while the stream runs): response.failed carried server_error, and the re-raise then aborted the connection; - in this process (raised by generate_async): HTTP 400; - disaggregated serving (the context worker's 400): HTTP 400. Non-streamed requests keep HTTP 400 with code context_length_exceeded. Every other stream failure keeps server_error and the re-raise; its response.failed frame is byte-identical to before. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…ses-context-length-in-stream Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com> # Conflicts: # tensorrt_llm/serve/openai_server.py # tensorrt_llm/serve/responses_utils.py # tests/unittest/llmapi/apps/test_responses_streaming_events.py
…e while the context phase runs On the disaggregated server, a streamed /v1/responses request receives no bytes at all, not even the HTTP status line, until the context phase is done: the route awaits the pipeline before it creates the streaming response, and the pipeline returns its stream only once the context worker has answered, in both schedule orders. Queueing plus prefill of a long agent prompt can take minutes, and a client or gateway that drops a stream without a first event loses the request (Codex behind a gateway gives up after about 120 s); its retry starts the prefill over. An aggregated server sends response.created at once and repeats response.in_progress while the stream is silent. A streamed request now waits for its pipeline for at most _RESPONSES_STREAM_START_WAIT_S (1 s). A pipeline that answers within the wait is answered as before, its failures with their HTTP status. For one still running, the stream starts with response.created and response.in_progress built from the request, and keep_responses_stream_alive repeats response.in_progress whenever the stream is silent. The worker's stream follows without its own opening events, numbered to continue the sequence the client has. The request id is forwarded to the workers, so that they answer under the id the opening named. A pipeline that fails after the stream started ends it with error and response.failed, which carry the message of the HTTP error the request would otherwise have received. A client that disconnects first cancels the pipeline, and a cancelled context phase releases a generation server reserved for the request. Non-streamed requests and the other routes are unchanged. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
Describe in the disaggregated serving guide how a streamed /v1/responses request whose context phase is still running gets its opening from the server, and how a later failure is reported. Import AsyncIterator from typing, with the module's other typing names. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
… its opening only Once the server had opened a stream, the relay still passed the generation worker's own keepalives, its repeats of response.in_progress, on to the client. They carry the worker's response object, created on the worker's clock, not the one the client was opened with, while an aggregated server repeats the stream's first response.in_progress exactly. They also came on top of the keepalives this server sends, so a silent worker had the client kept alive twice. The relay now drops every response.created and response.in_progress of the worker's, wherever it comes. That loses nothing, since a worker's keepalive repeats its own opening: the client is kept alive by this server alone, with the opening's response.in_progress and the next sequence number. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…er stream breaks, close it when abandoned A delayed stream guards the generation worker's stream with the processor's failed events, so a read failure before the terminal event ends the stream with one error and one response.failed for the response it opened with, and the stream then ends cleanly. A stream the pipeline answered with inside the start wait keeps the bare error event. The delayed response closes the stream its pipeline answered with however the response ends, also when the body never started. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…ext-length-in-stream Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…a late failure; always forward request_id A context-length rejection that arrives after an opened Responses stream has started now ends the stream with error and response.failed carrying context_length_exceeded, the error response's own code, instead of upstream_error and server_error, so agent clients can compact their history and retry. response.failed is built the same way the in-stream over-length rejection builds it, because the SDK's ResponseError does not list that code. The body forwarded to a worker now always includes a Responses request's request_id, replacing a self-assignment that only worked through pydantic's fields-set tracking. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…ext-length-in-stream Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…apshots and a CPU test seam for trtllm-serve Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
… baselines at the merge base The route reference is generated from the live apps; the object.__new__ baseline and the gRPC import allowance record what exists on main today and only shrink. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…n the snapshot matrix An attached frontend of a multi-frontend server registers ten launcher forwarding routes before the local table; seven share a method and path with local handlers and three use a path parameter that matches every batch id, so registration order decides which handler serves them. The new configuration records all of it, a live invariant pins the forwarding-before-local order independently of the reference, and the route-class failure message now says why the msgpack route class matters. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
- AGENTS.md: the CPU test command sets LLM_MODELS_ROOT, which the runner requires, and the stage-mapping note says the mapper matches list-entry text, so a file listed through its directory maps by the directory. - check_contracts matches each from-import as <module>.<name>, so 'from pkg import sub' and 'from .. import sub' count as imports of the submodule; new test for both forms and for the allowance entry. - The VisualGen /v1/models test asserts the 200 status. - The AGENTS.md command check also recognizes commands that start with LLM_MODELS_ROOT=. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…on the current main The disaggregated frontend now registers its routes with KvHintsRoute (NVIDIA#20269), which strips kv_hints from request bodies unless the server trusts its ingress. Regenerate the route reference: the six disaggregated configurations record KvHintsRoute where they recorded APIRoute; their route sets and order are unchanged. The route-class invariant, which a regenerated reference cannot hide, now requires KvHintsRoute on every API route of the disaggregated frontend and APIRoute on the coordinator app. Dropping the KV-hint policy from the frontend fails it even after a regeneration, and both apps still reject the aggregated server's msgpack route class. The object.__new__ baseline records the sites on main at the merge base: test_kv_hints.py builds one server that way (NVIDIA#20269), and test_harmony_parsing.py no longer builds any (NVIDIA#21176). The harmony parsing tests are now registered as a whole file (NVIDIA#21411), so the node-id entry that the CPU-selection test used as its example is gone. The node-id check now runs select_entries on a one-entry list of its own, so it no longer depends on which node-id entries l0_cpu.yml holds. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…ail on silent regressions The route snapshot now requires every configuration other than the three invalid ones to build, so a configuration that starts failing at construction cannot pass through a regenerated reference. The object.__new__ ratchet resolves 'from ... import <server class> as <name>' aliases and records each baseline site as its enclosing function or class plus the server class, so swapping one site for another in the same file no longer passes. The baseline keeps the same 52 sites in the same 26 files. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…xception handler for subclasses OpenAIDisaggServer._handle_exception(self, exception) keeps the signature it has on main. Subclasses override it with that signature (NeMo-RL's OpenAIDisaggServerAdaptor), so the extra request argument would have made every failed request raise TypeError there. The in-stream answer for a streamed Responses request that the context worker rejected for its length moves to the caller, _wrap_entry_point. The handler, base or override, rules on the failure first, as it does for every failure; only when it returns the worker's context_length_exceeded envelope is that envelope re-shaped into the stream. An override that answers the failure itself keeps its answer. Tests drive the wrapped route instead of the handler, and a subclass with a one-argument handler is checked for an ordinary failure and for a streamed over-length Responses request. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…lm/serve/_base/msgspec_route.py Generated move: _msgpack_decoder, _MsgspecRequest and _MsgspecRoute leave tensorrt_llm/serve/openai_server.py byte for byte; the module imports _MsgspecRoute back for its app router and keeps no re-export (the three names are private). The two disaggregated test importers now import the new module. OpenAIServer.health, .version and .get_model are removed here and replaced by the system router added in the following commits. tensorrt_llm/serve/_base/msgspec_route.py joins legacy-files.txt (its code comes from a legacy-formatted file), and the generated lint configs are regenerated with scripts/legacy_utils.py gen-configs. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…ve the system endpoints through it New structure: tensorrt_llm/serve/_base/ (shared serving infrastructure) and tensorrt_llm/serve/api/ (endpoint packages, one router plus one serving class each; the package __init__ imports nothing so routers load lazily, by role). serve/_base/routing.py provides make_router(), which always uses the msgpack-decoding route class, and attach_routers(), which imports and attaches only the routers whose RouterSpec matches the server role. The first endpoint package, serve/api/ops/system/, serves /health, /version and /v1/models for every role. OpenAIServer builds a SystemServing with the injected _health_probe (today's probe body, SIGINT side effect included) and attaches the routers before the role dispatch; the 12 add_api_route lines for the three paths leave the four role tables. Same routes, methods, route classes, bodies and conflict-relevant order: the recorded route reference is unchanged. The layering tests gain the new packages' contracts, the route-snapshot failure message points new endpoints at the router pattern, and the serve agent docs describe where a new endpoint goes. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…w serve packages tensorrt_llm/serve/_base/ and tensorrt_llm/serve/api/ops/ join the multi-GPU trigger list: the msgpack route class decodes the disaggregated frontend's worker requests, and /health is the disaggregated readiness probe, so changes under these directories keep running the multi-GPU stages that exercise that path. The three new modules join the strict mypy file set, and the type-check pre-commit hook now also triggers on changes under tensorrt_llm/serve/ and tensorrt_llm/commands/serve_cli/, so edits to typed serving modules get local type checking; the hook still checks only the declared file set. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
… tests that build routes by hand Two multi-frontend tests build an OpenAIServer without its constructor and call register_routes() themselves. Since /health, /version and /v1/models are attached through the system router by the constructor, those tests no longer saw the three routes: one asserted that /version stays local on an attached frontend (404), the other reported the three per-frontend-safe entries as removed. Both helpers now set the serving object and attach the routers before the hand registration, in the constructor's order, and the route enumeration walks included routers, which FastAPI 0.139 keeps as objects without a path in app.routes. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…n the OpenEngine metrics test The HTTP-enablement case builds a real OpenAIServer through its constructor and looked for /prometheus/metrics by reading .path off every entry of app.routes. Since the constructor attaches the system router, FastAPI 0.139 keeps an included-router object there that has no .path, and the child process failed with AttributeError. The test now matches a GET /prometheus/metrics request against the routes with the public route-matching API, which works on both ends of the pinned FastAPI range. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…d name The check that endpoint packages under tensorrt_llm/serve/api do not import each other read the package off the from-module alone, so `from tensorrt_llm.serve import api` inside an endpoint package passed and `from tensorrt_llm.serve.api.ops import system` inside ops/system failed. It now reads the package off <module>.<name> per imported name, as the layering contracts do, and a synthetic-edge test pins both forms. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…ency logger out of responses_utils tensorrt_llm/serve/responses_utils.py is the Responses API implementation, but the disaggregated client, service, coordinator and router imported it only for three small service types and a latency logger, and so loaded the tool parsers, web_search and the harmony adapter at import time. These definitions move, verbatim, into the shared serving package: tensorrt_llm.serve.responses_utils.ResponseHooks -> tensorrt_llm.serve._base.service_types tensorrt_llm.serve.responses_utils.done_generator -> tensorrt_llm.serve._base.service_types tensorrt_llm.serve.responses_utils.UCompletionResponseOrGenerator -> tensorrt_llm.serve._base.service_types tensorrt_llm.serve.responses_utils.PeriodicLatencyLogger -> tensorrt_llm.serve._base.latency_logger The old module keeps no re-export: no code outside the repository imports these names from it, and every importer in the repository is rewritten to the new modules (openai_client, openai_disagg_service, openai_service, the function-level imports in router and disagg_coordinator, and the two tests under tests/unittest/disaggregated/). The imports that responses_utils.py no longer needs (ABC, abstractmethod, UCompletionRequest, UCompletionResponse) go with them. The two new modules take the formatting group of the file their code comes from: they are registered in legacy-files.txt and the derived lint configs are regenerated with scripts/legacy_utils.py gen-configs. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…tils in the disaggregated client and server tensorrt_llm/serve/openai_client.py and tensorrt_llm/serve/openai_disagg_server.py took get_steady_clock_now_in_seconds from the re-export in tensorrt_llm/serve/responses_utils.py; they now import it from tensorrt_llm._utils, where it is defined, so that nothing under the disaggregated client needs the Responses implementation for the clock. The re-export in responses_utils.py itself is unchanged. openai_client still binds the clock as a module-level name, so the tests that patch tensorrt_llm.serve.openai_client.get_steady_clock_now_in_seconds keep intercepting it. The disaggregated server also imports ResponseHooks from tensorrt_llm.serve._base.service_types, the module that defines it, instead of through openai_disagg_service. The copyright year of tensorrt_llm/serve/openai_service.py, whose import this change edits, is brought to 2026. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
… the Responses implementation A layering contract in tests/unittest/serve/test_layering.py forbids tensorrt_llm.serve.responses_utils as an import of openai_client, openai_service, openai_disagg_service, disagg_coordinator and router, at module level and inside functions. Its one allowance is responses_done_generator in openai_disagg_service, which leaves when the Responses stream code shared with the disaggregated frontend moves to tensorrt_llm/serve/_base/responses_stream.py; the allowance only shrinks. tests/unittest/serve/test_disagg_imports.py imports each of those modules in a fresh interpreter and checks that the Responses implementation is not loaded at all, with a positive control that the probe reports it when it is imported. The openai_disagg_service case is a strict expected failure for the same responses_done_generator import. The file is collected through the existing unittest/serve entry of the CPU test list and carries the cpu_only marker. The expected failure is limited to the import-isolation assertion: an import crash of the module raises CalledProcessError and fails the test. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…requests in-stream with context_length_exceeded Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…eam and keep it alive while the context phase runs Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…m/serve/_base/responses_stream.py The terminal events and the response.failed frame, the stream termination codes with the classification of a stream-ending exception, the SSE frame markers, the guard that ends a failing stream with its terminal events, the keepalive that repeats response.in_progress and renumbers the events after it, the relay of a worker's stream, the SSE formatting helpers and the done generator move out of responses_utils.py into a module both the aggregated and the disaggregated server import. The moved definitions are unchanged; the new file joins the legacy formatting group of the file its code comes from. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…ads them and pin the new module's imports The keepalive tests set RESPONSES_STREAM_KEEPALIVE_INTERVAL_S on the module that now defines it, tensorrt_llm/serve/_base/responses_stream.py, and the disaggregated Responses tests call keep_responses_stream_alive and relay_after_opening there. The disaggregated service no longer imports responses_utils, so the import-isolation test loses its expected failure and the layering contract its allowance. A new contract keeps the shared stream module a leaf: it imports neither server, nor responses_utils, and of the executor only its error contract in tensorrt_llm.executor.utils, which the new module now names explicitly for RequestError and EngineDeadError too. The new module gets its docstring, and responses_utils.py loses the section header that introduced the keepalive code, all of which moved. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
This was referenced Nov 12, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@coderabbitai summary
Stacked on #21510 ([TRTLLM-17262][refactor] Move the disaggregated service types and latency logger out of responses_utils), #21060 ([None][fix] Responses: fail over-length streamed requests in-stream with context_length_exceeded) and #21452 ([None][fix] Disaggregated Responses: open the stream and keep it alive while the context phase runs): this PR's own change is the last 2 commits; the earlier commits and two integration merges belong to its dependencies. #21510 still carries #21448 and the historical commits of #21406. #21406 has since merged as
9ceb00a84e; this PR was verified against the unchanged integration basec1200008f5. The dependency history will leave this diff when the remaining prerequisites merge and this move is regenerated on main.Description
tensorrt_llm/serve/responses_utils.pyis the Responses API implementation, but both servers take their Responses stream lifecycle from it: the aggregated server (openai_server.py) guards a stream against an early failure, keeps it alive and closes it with the request, and the disaggregated server (openai_disagg_server.py,openai_disagg_service.py) opens a stream before the context phase answers, keeps it alive, relays the generation worker's stream behind that opening, guards the relay and sends an already-complete response as an SSE run. Importing all ofresponses_utilsfor that drags the tool parsers, web search and the harmony adapter into the disaggregated frontend.This PR moves that lifecycle into one module both servers import,
tensorrt_llm/serve/_base/responses_stream.py, generated as a verbatim move (check below): the terminal event matching a response's status and theresponse.failedframe; the codes a stream that stops early reports (engine_error,upstream_error,internal_error,context_length_exceeded) with the classification of the exception behind them and its client-facing description; the frame markers that tell a terminal frame apart; the bareerrorevent a relay ends with; the guard that ends a stream failing before its terminal event with the events saying so and answers an over-length rejection in-stream; the keepalive that repeatsresponse.in_progresswhile a stream is silent and renumbers the events after it, with its tick loop, JSON member finder and sequence-number stamping; the streaming response class that closes its body iterator however the request ends; the relay of a worker's stream to a client that already holds its opening; the SSE formatting helpers and the done generator. 29 definitions move;responses_utils.pyimports back the four its streaming processor still uses (_stream_terminal_event,_response_failed_frame,_format_sse_event,STREAM_TERMINATION_CONTEXT_LENGTH_EXCEEDED).What the new module imports: the protocol models (
tensorrt_llm.serve.openai_protocoland the OpenAI SDK event types they are built on), the logger, the standard library, the SSE/HTTP libraries the moved code already used (fastapi.responses.StreamingResponse,anyio,pydantic_core), and from the executor only its error contract,tensorrt_llm.executor.utils:RequestErrorandEngineDeadError, the exception types a stream can end with, andCONTEXT_LENGTH_EXCEEDED_CODEwithis_context_length_exceeded_message, the over-length rejection the guard answers in-stream. That one import is allowed because the classification of a stream-ending exception is part of the lifecycle, and because it adds no import weight to either frontend: the disaggregated server already importstensorrt_llm.executorandtensorrt_llm.executor.utilsfor the same names. The new module namestensorrt_llm.executor.utilsfor all four (the old file tookRequestErrorandEngineDeadErrorfrom the packagetensorrt_llm.executor); that import edit is the only change to the moved code. A new layering contract intests/unittest/serve/test_layering.pypins the module as a leaf: it imports neither server, norresponses_utils, nor the disaggregated server, service and client modules, and oftensorrt_llm.executorexactly those four names fromtensorrt_llm.executor.utils(the contract's allowance; it only shrinks).Left in
responses_utils.pyon purpose:rejected_responses_streamdrives aResponsesStreamingProcessor, the Responses implementation itself;ResponsesStreamingProcessorandServerArrivalTimeMiddlewareare not stream lifecycle, the Responses package move and #21447 ([TRTLLM-17262][refactor] Shared middleware package for the aggregated and disaggregated servers) take them.Hand edits beyond the generated move, all in the second commit: the module docstring; the import of the executor's error contract written as one
tensorrt_llm.executor.utilsimport (above); the section-header comment inresponses_utils.pythat introduced the keepalive code (all of which moved) is removed; the keepalive tests setRESPONSES_STREAM_KEEPALIVE_INTERVAL_Son the module that now defines it (sixmonkeypatch.setattrsites intests/unittest/llmapi/apps/test_responses_streaming_events.pyandtests/unittest/disaggregated/test_disagg_responses_api.py) and the disaggregated tests callkeep_responses_stream_aliveandrelay_after_openingthere;tests/unittest/serve/test_disagg_imports.pydrops its expected failure foropenai_disagg_service, which no longer importsresponses_utils; the layering contract for the disaggregated modules loses its one allowance (openai_disagg_servicetakingresponses_done_generatorfromresponses_utils), and the contract for the shared module is added. Thepatch("tensorrt_llm.serve.responses_utils.logger")sites intest_responses_stream_final_consistency.pyobserve the whole-text parse in code that stays, not the guard, and are unchanged.Behavior
No behavior change. Every moved definition is identical to its original (check below; each moved span, comment block included, was also byte-compared). The constants that were adjacent in the old file stand two blank lines apart in the new one, the generated layout.
Moves
Pure-move check (head 922c861)
No shim at the old path: no code outside the repository imports these names from
responses_utils(GitHub code search on 2026-11-11 fortensorrt_llm.serve.responses_utilsand for the moved public names: every hit is a copy of this repository), every importer in the repository is rewritten, and private names are never re-exported.responses_utils.pystill bindsSTREAM_TERMINATION_CONTEXT_LENGTH_EXCEEDEDbecause its streaming processor imports it back for its own use; that is not a kept interface. Moves without a shim, "old path → new path" for the release notes:History:
git blame -C -C tensorrt_llm/serve/_base/responses_stream.pyrecovers original authorship for most moved lines.git log --followdoes not follow this partial extraction; use the old path when inspecting its earlier file history.Import graph after this PR:
openai_disagg_serviceimports nothing fromresponses_utilsany more, andtests/unittest/serve/test_disagg_imports.pynow asserts, in a fresh interpreter, that importing it does not load the Responses implementation at all (the assertion was an expected failure until now).openai_disagg_serverstill importsServerArrivalTimeMiddleware,ResponsesStreamingProcessorandrejected_responses_streamfrom it, until #21447 and the Responses package move take those;openai_serverkeepsConversationHistoryStore,ResponsesStreamingProcessor,ServerArrivalTimeMiddleware,rejected_responses_stream,create_responseandrequest_preprocess.CI bookkeeping
tensorrt_llm/serve/_base/, which the table already lists)./tensorrt_llm/servecovers the new file).tensorrt_llm/serve/_base/responses_stream.pyadded (legacy formatting group, likeresponses_utils.pythe code comes from); derived configs regenerated withpython scripts/legacy_utils.py gen-configs(ruff-legacy.tomland the managed blocks ofpyproject.tomland.pre-commit-config.yaml, 5 lines);ruff-legacy-baseline.json: none. The stale entriesgen-configswarns about predate this PR and are left alone.l0_cpu.ymllistsunittest/llmapi/apps/test_responses_streaming_events.pyand theunittest/disaggregatedandunittest/servedirectories).Open PRs on the moved code
[None][feat] Responses: configurable keepalive interval (TRTLLM_RESPONSES_STREAM_KEEPALIVE_S, 0 turns it off) #21808 ([None][feat] Responses: configurable keepalive interval (TRTLLM_RESPONSES_STREAM_KEEPALIVE_S, 0 turns it off)) edits
RESPONSES_STREAM_KEEPALIVE_INTERVAL_S,_with_ticksandkeep_responses_stream_aliveand adds the environment-variable reader next to them, all of which move here. It is not part of this PR's base: the move is complete without it. Whichever lands second re-applies: after this PR, [None][feat] Responses: configurable keepalive interval (TRTLLM_RESPONSES_STREAM_KEEPALIVE_S, 0 turns it off) #21808's change goes totensorrt_llm/serve/_base/responses_stream.py(its tests patch the interval there too); after [None][feat] Responses: configurable keepalive interval (TRTLLM_RESPONSES_STREAM_KEEPALIVE_S, 0 turns it off) #21808, this move is regenerated and carries its lines.[None][fix] Disaggregated Responses: a relayed stream cut before its terminal event ends with response.failed #22034 ([None][fix] Disaggregated Responses: a relayed stream cut before its terminal event ends with response.failed), stacked on [None][fix] Disaggregated Responses: open the stream and keep it alive while the context phase runs #21452, changes
openai_disagg_server.pyplus a test and documentation. Its current head merges cleanly with this move in either order. It usesguard_responses_streamandstream_error_eventthrough the imports this PR already retargets; no additional import edit is needed.[None][fix] Responses streaming: bind the first result and record the first-token time #21622 ([None][fix] Responses streaming: bind the first result and record the first-token time) edits the streaming generator in
openai_server.pyandtest_responses_streaming_events.py; no moved code. Only that test file's import block meets this PR's rewrite of it, a trivial conflict for whichever lands second.[None][feat] Responses: server-wide default max_output_tokens (TRTLLM_RESPONSES_DEFAULT_MAX_OUTPUT_TOKENS) #21806 ([None][feat] Responses: server-wide default max_output_tokens (TRTLLM_RESPONSES_DEFAULT_MAX_OUTPUT_TOKENS)) edits request preprocessing in
responses_utils.pyandopenai_protocol.py; no overlap with the moved code.[None][feat] request trace: engine request, prompt digest and stream ending #22177 ([None][feat] request trace: engine request, prompt digest and stream ending), the follow-up to [None][feat] trtllm-serve: record each request and the exact reply the client received #22046, adds an
on_terminationcallback toguard_responses_stream, anote_stream_terminationhelper, and callback wiring in both servers. If that follow-up lands first, regenerate this move so it carries those additions. If this move lands first, apply the callback and pass-through helper intensorrt_llm/serve/_base/responses_stream.py, and retarget the callers and tests to that module.For contributors
tensorrt_llm/serve/responses_utils.pytotensorrt_llm/serve/_base/responses_stream.py. Import them from the new module. The old path keeps none of them as an interface (no shim, no warning); the thirteen public names are listed above for the release notes, andSTREAM_TERMINATION_CONTEXT_LENGTH_EXCEEDEDresolves onresponses_utilsonly because its own code imports it back._with_ticks,_top_level_member,_stamp_sequence_number,_format_sse_event,_sse_event,_stream_terminal_event,_response_failed_frame,_sse_delimiter_and_markers, the frame markers and_JSON_DECODER) are not re-exported: import them from, and patch them on,tensorrt_llm.serve._base.responses_stream, where the moved code looks them up. Amonkeypatch.setattr(responses_utils, "RESPONSES_STREAM_KEEPALIVE_INTERVAL_S", ...)no longer reaches the keepalive (the attribute is gone, so it fails loudly); patchtensorrt_llm.serve._base.responses_streaminstead. The guard's log line comes from the new module'sloggernow.rejected_responses_streamstays inresponses_utils(it drives the Responses streaming processor); keep importing it from there.mainright before merging, so a PR that lands first on the old code is carried along; a PR that lands after it re-applies its lines to the new module (the list above says which).Test Coverage
Environment: CPU-only host; the worktree's Python code over source-built compiled artifacts older than this tree, so a local test plugin declares the C++ ops they lack (used identically on both sides of every A/B).
import tensorrt_llmresolved to the worktree. GPU stages are left to CI.29 definitions moved verbatim; 0 differences; shim exports 0 names. Independent byte comparison of every moved span (comment block plus definition) betweenresponses_utils.pyat the base and the new module: 29/29 identical; the new module holds no definition outside the list;responses_utils.pykeeps no copy of a moved name.*.py(module- and function-levelfrom tensorrt_llm.serve.responses_utils import ...) prints nothing, andgit grepforresponses_utils.<moved name>prints nothing.pytest tests/unittest/llmapi/apps/test_responses_*.py -q(8 files) → 345 passed.pytest tests/unittest/disaggregated/test_disagg_responses_api.py tests/unittest/disaggregated/test_openai_disagg_service.py tests/unittest/disaggregated/test_openai_disagg_server.py -q→ 255 passed, 2 failed:test_responses_stream_with_a_fast_context_failure_is_an_http_error[over_length_400-context_first]and[...-generation_first]. The same 2 fail on the integration base: the [None][fix] Disaggregated Responses: open the stream and keep it alive while the context phase runs #21452 test expects HTTP 400 for an over-length rejection, which [None][fix] Responses: fail over-length streamed requests in-stream with context_length_exceeded #21060 now answers in-stream; whichever of those two lands second adjusts it. Not caused by this PR.pytest -m cpu_only tests/unittest/serve -q→ 123 passed, 1 failed:test_no_object_new.py::test_no_new_object_new_sites(theobject.__new__ratchet sees the server fixtures [None][fix] Responses: fail over-length streamed requests in-stream with context_length_exceeded #21060 and [None][fix] Disaggregated Responses: open the stream and keep it alive while the context phase runs #21452 add); the same failure on the integration base.test_layering.py(the new contract included) andtest_disagg_imports.pypass, the latter without its former expected failure.pytest -m cpu_only tests/unittest/disaggregated -q→ 1622 passed, 46 failed, 1 skipped, 2 xfailed: the 2 Responses cases above plus 44 intest_context_first_profile.py(26),test_lifecycle_activation.py(16) andtest_peer.py(2), none touching serving code (KVCacheManagerV2 has no attribute low_precision_dtype: the compiled artifacts predate those tests). Integration base, same environment: the 2 Responses cases fail there too, and the three other files run at the base give the identical set of 44 failures (44 failed, 139 passed). Nothing in the directory fails because of this PR.pytest tests/unittest/api_stability -q→ 99 passed.guard_responses_streamin the new module made to raise at entry, 31 tests of the three files that exercise it (test_responses_streaming_events.py,test_disagg_responses_api.py,test_context_length_error.py) fail that pass otherwise (17, 12 and 2); withkeep_responses_stream_alivemade to raise, 28 of them fail; with the new module's default keepalive interval set to 1e9 s and the retargeted patches in place, all 43 keepalive and relay tests selected from the two edited test files pass, so the patches land where the code reads the constant. All three edits were reverted.pre-commit run --files <the 14 changed files>twice: clean both times, no file modified on the second run.python scripts/legacy_utils.py check-configs→ all generated configs up to date.type-checkhook ran on this pinned integration base, which still uses mypy. Main now uses Pyrefly after [None][infra] Introduce Pyrefly to do type check #21325; the move will use that checker when it is regenerated on main.PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
Reviewers by CODEOWNERS: @NVIDIA/trt-llm-runtime-devs (
tensorrt_llm/serve,tests/unittest/llmapi), @NVIDIA/trt-llm-disagg-devs (tensorrt_llm/serve/openai_disagg_server.py,tests/unittest/disaggregated), @NVIDIA/trt-llm-devs (tests/unittest/serve,legacy-files.txt), @NVIDIA/trt-llm-infra-devs (.pre-commit-config.yaml,ruff-legacy.toml), @NVIDIA/trt-llm-oss-compliance (pyproject.toml).GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.🤖 Generated with Claude Code