Repository navigation
[TRTLLM-17263][refactor] Move the Anthropic serving modules into serve/api/anthropic - #21518
Draft
JunyiXu-nv wants to merge 14 commits into
Draft
JunyiXu-nv wants to merge 14 commits into
JunyiXu-nv wants to merge 14 commits into
Conversation
…apshots and a CPU test seam for trtllm-serve Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
… baselines at the merge base The route reference is generated from the live apps; the object.__new__ baseline and the gRPC import allowance record what exists on main today and only shrink. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…n the snapshot matrix An attached frontend of a multi-frontend server registers ten launcher forwarding routes before the local table; seven share a method and path with local handlers and three use a path parameter that matches every batch id, so registration order decides which handler serves them. The new configuration records all of it, a live invariant pins the forwarding-before-local order independently of the reference, and the route-class failure message now says why the msgpack route class matters. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
- AGENTS.md: the CPU test command sets LLM_MODELS_ROOT, which the runner requires, and the stage-mapping note says the mapper matches list-entry text, so a file listed through its directory maps by the directory. - check_contracts matches each from-import as <module>.<name>, so 'from pkg import sub' and 'from .. import sub' count as imports of the submodule; new test for both forms and for the allowance entry. - The VisualGen /v1/models test asserts the 200 status. - The AGENTS.md command check also recognizes commands that start with LLM_MODELS_ROOT=. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…on the current main The disaggregated frontend now registers its routes with KvHintsRoute (NVIDIA#20269), which strips kv_hints from request bodies unless the server trusts its ingress. Regenerate the route reference: the six disaggregated configurations record KvHintsRoute where they recorded APIRoute; their route sets and order are unchanged. The route-class invariant, which a regenerated reference cannot hide, now requires KvHintsRoute on every API route of the disaggregated frontend and APIRoute on the coordinator app. Dropping the KV-hint policy from the frontend fails it even after a regeneration, and both apps still reject the aggregated server's msgpack route class. The object.__new__ baseline records the sites on main at the merge base: test_kv_hints.py builds one server that way (NVIDIA#20269), and test_harmony_parsing.py no longer builds any (NVIDIA#21176). The harmony parsing tests are now registered as a whole file (NVIDIA#21411), so the node-id entry that the CPU-selection test used as its example is gone. The node-id check now runs select_entries on a one-entry list of its own, so it no longer depends on which node-id entries l0_cpu.yml holds. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…ail on silent regressions The route snapshot now requires every configuration other than the three invalid ones to build, so a configuration that starts failing at construction cannot pass through a regenerated reference. The object.__new__ ratchet resolves 'from ... import <server class> as <name>' aliases and records each baseline site as its enclosing function or class plus the server class, so swapping one site for another in the same file no longer passes. The baseline keeps the same 52 sites in the same 26 files. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…lm/serve/_base/msgspec_route.py Generated move: _msgpack_decoder, _MsgspecRequest and _MsgspecRoute leave tensorrt_llm/serve/openai_server.py byte for byte; the module imports _MsgspecRoute back for its app router and keeps no re-export (the three names are private). The two disaggregated test importers now import the new module. OpenAIServer.health, .version and .get_model are removed here and replaced by the system router added in the following commits. tensorrt_llm/serve/_base/msgspec_route.py joins legacy-files.txt (its code comes from a legacy-formatted file), and the generated lint configs are regenerated with scripts/legacy_utils.py gen-configs. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…ve the system endpoints through it New structure: tensorrt_llm/serve/_base/ (shared serving infrastructure) and tensorrt_llm/serve/api/ (endpoint packages, one router plus one serving class each; the package __init__ imports nothing so routers load lazily, by role). serve/_base/routing.py provides make_router(), which always uses the msgpack-decoding route class, and attach_routers(), which imports and attaches only the routers whose RouterSpec matches the server role. The first endpoint package, serve/api/ops/system/, serves /health, /version and /v1/models for every role. OpenAIServer builds a SystemServing with the injected _health_probe (today's probe body, SIGINT side effect included) and attaches the routers before the role dispatch; the 12 add_api_route lines for the three paths leave the four role tables. Same routes, methods, route classes, bodies and conflict-relevant order: the recorded route reference is unchanged. The layering tests gain the new packages' contracts, the route-snapshot failure message points new endpoints at the router pattern, and the serve agent docs describe where a new endpoint goes. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…w serve packages tensorrt_llm/serve/_base/ and tensorrt_llm/serve/api/ops/ join the multi-GPU trigger list: the msgpack route class decodes the disaggregated frontend's worker requests, and /health is the disaggregated readiness probe, so changes under these directories keep running the multi-GPU stages that exercise that path. The three new modules join the strict mypy file set, and the type-check pre-commit hook now also triggers on changes under tensorrt_llm/serve/ and tensorrt_llm/commands/serve_cli/, so edits to typed serving modules get local type checking; the hook still checks only the declared file set. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
… tests that build routes by hand Two multi-frontend tests build an OpenAIServer without its constructor and call register_routes() themselves. Since /health, /version and /v1/models are attached through the system router by the constructor, those tests no longer saw the three routes: one asserted that /version stays local on an attached frontend (404), the other reported the three per-frontend-safe entries as removed. Both helpers now set the serving object and attach the routers before the hand registration, in the constructor's order, and the route enumeration walks included routers, which FastAPI 0.139 keeps as objects without a path in app.routes. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…n the OpenEngine metrics test The HTTP-enablement case builds a real OpenAIServer through its constructor and looked for /prometheus/metrics by reading .path off every entry of app.routes. Since the constructor attaches the system router, FastAPI 0.139 keeps an included-router object there that has no .path, and the child process failed with AttributeError. The test now matches a GET /prometheus/metrics request against the routes with the public route-matching API, which works on both ends of the pinned FastAPI range. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…d name The check that endpoint packages under tensorrt_llm/serve/api do not import each other read the package off the from-module alone, so `from tensorrt_llm.serve import api` inside an endpoint package passed and `from tensorrt_llm.serve.api.ops import system` inside ops/system failed. It now reads the package off <module>.<name> per imported name, as the layering contracts do, and a synthetic-edge test pins both forms. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…ve/api/anthropic Move tensorrt_llm/serve/anthropic_adapter.py, anthropic_protocol.py and anthropic_batches.py whole into tensorrt_llm/serve/api/anthropic/ as adapter.py, protocol.py and batches.py, and rewrite every import of them in the repository. The old paths are deleted without a compatibility shim: no code outside the repository imports them. Generated change: the three files are copied byte for byte apart from their import lines; the importers change only in their import lines. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
… mark the new package Add tensorrt_llm/serve/api/anthropic/__init__.py, retarget the two module imports of the adapter in test_anthropic_adapter.py to the new path, and add tests/unittest/serve/api/anthropic/test_messages_behavior.py: characterization tests of what the aggregated and the disaggregated server answer on the Anthropic routes today (success, streaming, a stream without chunks, rejected requests, context length, upstream errors, unexpected errors, engine failures, an unconvertible chat response, and the aggregated Message Batches), with the served model and the requested model named differently so that each server's own answer is pinned. The follow-up PR that gives both servers one Anthropic Messages serving class must keep this file unchanged and green. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
JunyiXu-nv
force-pushed
the
dev-junyix-feat-serve-anthropic-package-move
branch
from
November 4, 2026 12:50
dd234c2 to
147c114
Compare
1 task done
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #21448 ([TRTLLM-17261][refactor] Add the trtllm-serve router skeleton and move /health, /version and /v1/models into ops.system), which is itself stacked on #21406 ([TRTLLM-17261][test] Add route, middleware-order and shim-identity snapshots and a CPU test seam for trtllm-serve): the first commits below belong to those PRs and disappear from this diff when they merge; this PR's own change is the last 2 commits.
@coderabbitai summary
Description
This PR moves the three Anthropic Messages modules next to the servers,
tensorrt_llm/serve/anthropic_adapter.py,anthropic_protocol.pyandanthropic_batches.py, whole into the endpoint packagetensorrt_llm/serve/api/anthropic/(adapter.py,protocol.py,batches.py), deletes the old paths and rewrites every import of them in the repository. It also addstests/unittest/serve/api/anthropic/test_messages_behavior.py, which pins what the aggregated and the disaggregated server answer on the Anthropic routes today, so that the follow-up PR that converges the two servers' Anthropic handling into one serving class can show it keeps every answer.Part of the trtllm-serve refactor (epic TRTLLM-17056), task TRTLLM-17263 "Typed return contract and Anthropic convergence". It builds on #21448 (the
tensorrt_llm/serve/api/package) and on #21406 ([TRTLLM-17261][test] Add route, middleware-order and shim-identity snapshots and a CPU test seam for trtllm-serve), whose server factory and fake generators the new tests use.Behavior
No behavior change. The three modules move byte-identical apart from their import lines (the pure-move check below compares every definition), and the new
test_messages_behavior.pypins both servers' Anthropic answers with the served model (server-model) and the requested model (client-model) named differently, including a stream without chunks, the one case where the two servers report different model names today (the aggregated server its served name, the disaggregated one the client's). The follow-up PR must keep that file unchanged and green.Moves
Pure-move check
Pure-move check: base d3aacdd, head 147c114, tools 517fc468330a
AnthropicRequestErrorAnthropicResponseErrorAnthropicStreamReframerSTOP_REASON_MAPSTREAM_PING_INTERVAL_S_ANTHROPIC_BILLING_SYSTEM_BLOCK_KEEP_ALL_THINKING_KWARGS_OUTPUT_FORMAT_SCHEMA_NAME_ToolCallStream_convert_messages_convert_tool_choice_convert_tools_image_part_is_anthropic_billing_system_block_iter_openai_sse_lines_map_stop_result_system_text_parts_thinking_retention_kwargs_tool_result_text_with_ticksanthropic_error_responseconvert_anthropic_count_tokens_requestconvert_anthropic_requestconvert_chat_responseconvert_usagemap_stop_reasonreframe_openai_streamthinking_signatureAnthropicBatchStoreBATCH_CONCURRENCY_ENVBATCH_TTLBatchItemOutcomeBatchItemRunnerBatchStoreFullErrorDEFAULT_BATCH_CONCURRENCYDEFAULT_MAX_RETAINED_BATCHESMAX_RETAINED_BATCHES_ENV_BatchRecord_ItemResult_int_env_now_rfc3339results_to_jsonlANTHROPIC_BATCH_MAX_REQUESTSAnthropicBaseModelAnthropicBatchDeleteResponseAnthropicBatchListAnthropicBatchProcessingStatusAnthropicBatchRequestCountsAnthropicBatchRequestItemAnthropicBatchResultTypeAnthropicContentBlockDeltaEventAnthropicContentBlockStartEventAnthropicContentBlockStopEventAnthropicContentBlockAnthropicContentDeltaAnthropicCountTokensRequestAnthropicCountTokensResponseAnthropicCreateBatchRequestAnthropicErrorEventAnthropicErrorResponseAnthropicErrorTypeAnthropicErrorAnthropicImageBlockAnthropicImageSourceAnthropicInputJsonDeltaAnthropicMessageBatchAnthropicMessageDeltaEventAnthropicMessageDeltaAnthropicMessageStartEventAnthropicMessageStopEventAnthropicMessageAnthropicMessagesRequestAnthropicMessagesResponseAnthropicPingEventAnthropicRedactedThinkingBlockAnthropicSignatureDeltaAnthropicStopReasonAnthropicTextBlockAnthropicTextDeltaAnthropicThinkingAdaptiveAnthropicThinkingBlockAnthropicThinkingConfigAnthropicThinkingDeltaAnthropicThinkingDisabledAnthropicThinkingDisplayAnthropicThinkingEnabledAnthropicToolChoiceAnthropicToolResultBlockAnthropicToolUseBlockAnthropicToolAnthropicUnknownBlockAnthropicUsageSCHEMA_CLIENT_TOOL_TYPE_PREFIXESSERVER_TOOL_TYPE_PREFIXES_CONTENT_BLOCK_MODELS_CUSTOM_ID_PATTERN_KNOWN_CONTENT_BLOCK_TYPES_content_block_taganthropic_ssecurrent_timestamp101 definitions moved verbatim; 0 differences; 3 files moved; shim exports 0 names