[TRTLLM-17273][chore] Choose the trtllm-serve post-processing path once per server by JunyiXu-nv · Pull Request #21392 · NVIDIA/TensorRT-LLM

JunyiXu-nv · 2026-11-01T15:18:52Z

Dev Engineer Review

OpenAIServer selects and caches a post-processing strategy from num_postprocess_workers. Response handlers use it for worker-result retrieval or inline processing, including an async callback for inline Responses API handling. Submission-side checks still use postproc_worker_enabled.

CompletionOutput.postprocess_result exposes the stored worker result and returns None when no worker stored a result. The implementation documents that worker mode stores a result only on the first output when there are multiple choices, unlike inline mode’s per-candidate metrics behavior. No current review findings were supplied, so severity counts are unavailable.

QA Engineer Review

tests/unittest/serve/test_postproc_strategy.py adds eight tests for strategy selection, sync and async behavior, server-level caching, and the accessor. The file is listed in tests/integration/test_lists/test-db/l0_cpu.yml. Modified tests update worker-result access in disaggregated Responses API and Anthropic streaming coverage. The executor test adds an accessor assertion, and the gRPC metrics test sets num_postprocess_workers to zero. The Anthropic route and executor tests are listed in l0_cpu.yml; the disaggregated test, run_llm_with_postproc.py, and gRPC metrics test were not found in the searched CI or manual-QA lists.

The author reports passing CPU test runs, an LLM API apps failure that also occurred on the base commit, and mixed CI results. The latest referenced run has no reported outcome. Coverage verdict: needs follow-up for the outstanding CI result and reported base-commit failure.

Per-File QA Perspective

tensorrt_llm/executor/result.py — Adds the postprocess_result accessor. Verify access to a stored worker result and the None case.
tensorrt_llm/serve/openai_server.py — Routes response post-processing through a cached strategy. Verify worker and inline paths across streaming, non-streaming, and async Responses API requests.
tensorrt_llm/serve/postprocess_handlers.py — Adds worker and inline strategies with sync and async methods. Verify that worker mode does not invoke inline processing and inline mode invokes the correct processor.
tests/integration/test_lists/test-db/l0_cpu.yml — Adds unittest/serve/test_postproc_strategy.py. It also lists the changed Anthropic route and executor tests.
tests/unittest/api_stability/references/completion_output.yaml — Adds postprocess_result to the API-stability reference.
tests/unittest/disaggregated/test_disagg_responses_api.py — Updates access to the worker result in disaggregated response tests. The file was not found in the searched CI or manual-QA lists.
tests/unittest/llmapi/apps/test_anthropic_routes.py — Updates access to streamed frames. The file is listed in l0_cpu.yml.
tests/unittest/llmapi/run_llm_with_postproc.py — Updates access to generated output results. The file was not found in the searched CI or manual-QA lists.
tests/unittest/llmapi/test_executor.py — Adds an accessor assertion for the stored result. The file is listed in l0_cpu.yml.
tests/unittest/serve/test_postproc_strategy.py — Covers strategy selection, sync and async behavior, server-level caching, and the accessor. The file is listed in l0_cpu.yml.
tests/unittest/grpc/openengine/test_metrics.py — Sets num_postprocess_workers to zero in test arguments. The file was not found in the searched CI or manual-QA lists.

Description

OpenAIServer decided nine times, in four handlers, whether a response's post-processed value comes from a post-processing worker (read the value the worker stored on outputs[0]) or must be computed in the server process (call the post-processor inline). The flag that drives the decision, postproc_worker_enabled, never changes after construction. This PR chooses the path once per server: select_postproc_strategy (new, in tensorrt_llm/serve/postprocess_handlers.py) picks WorkerPostprocStrategy or InlinePostprocStrategy from num_postprocess_workers; __init__ pre-caches the choice and a functools.cached_property (the same pattern as _is_visual_gen) keeps the many unit-test servers built with object.__new__ working — a plain attribute set only in __init__ would break every one of them. CompletionOutput gains the read-only public accessor postprocess_result, so the serving code stops reading the executor's private _postprocess_result field; the field and its writer (the frontend's handling of a postproc worker's output) are unchanged.

Part of the trtllm-serve refactor (epic TRTLLM-17056, task TRTLLM-17273): cleanups that land before the handler moves, so that moved handlers carry one strategy call per site instead of nine copies of a private-field read.

The nine decision points

All in tensorrt_llm/serve/openai_server.py; each if/ternary on self.postproc_worker_enabled becomes one call.

Handler	Site	Old	New
`_create_chat_response`	non-streaming chat	if/else on `promise.outputs[0]._postprocess_result` vs `post_processor(promise, args)`	`self._postproc.apply(promise, postproc_params)`
`openai_chat` → `chat_stream_generator`	first response	ternary	`self._postproc.apply(first_response, postproc_params)`
same	loop	ternary	`self._postproc.apply(res, postproc_params)`
`openai_completion` → `completion_response`	non-streaming	if/else	`self._postproc.apply(response, postproc_params)`
`openai_completion` → `completion_generator`	loop	if/else	`self._postproc.apply(output, params)`
`chat_harmony` → `create_streaming_generator`	first response	ternary	`self._postproc.apply(first_response, postproc_params)`
same	loop	ternary	`self._postproc.apply(res, postproc_params)`
`openai_responses` → `create_response`	non-streaming	if/else; the inline branch awaits `responses_api_create_response(...)`	`await self._postproc.apply_async(promise, lambda: responses_api_create_response(...))` — the lambda is evaluated only in inline mode, so no coroutine is created and `self.enable_store` / `self.tokenizer` are read exactly as before
`openai_responses` → `create_streaming_generator`	loop	ternary	`self._postproc.apply(res, postproc_params)`

The two conditional unpacks that only fed the ternaries (if not self.postproc_worker_enabled: post_processor, args = ... in the chat and harmony stream generators) are deleted; in the Responses streaming generator the unpack keeps only args (post_processor became unused, args.streaming_processor is still read).

As a concrete example, the chat stream generator's first-response site (second row of the table) was:

                if not self.postproc_worker_enabled:
                    post_processor, args = postproc_params.post_processor, postproc_params.postproc_args
                first_response = await anext(promise)
                ...
                pp_results = first_response.outputs[
                    0]._postprocess_result if self.postproc_worker_enabled else post_processor(
                        first_response, args)

and is now:

                first_response = await anext(promise)
                ...
                pp_results = self._postproc.apply(first_response,
                                                  postproc_params)

The strategies themselves (in tensorrt_llm/serve/postprocess_handlers.py):

class WorkerPostprocStrategy(PostprocStrategy):

    def apply(self, result, postproc_params):
        return result.outputs[0].postprocess_result

    async def apply_async(self, result, inline):
        return result.outputs[0].postprocess_result


class InlinePostprocStrategy(PostprocStrategy):

    def apply(self, result, postproc_params):
        return postproc_params.post_processor(result,
                                              postproc_params.postproc_args)

    async def apply_async(self, result, inline):
        return await inline()


def select_postproc_strategy(postproc_worker_enabled: bool) -> PostprocStrategy:
    return (WorkerPostprocStrategy()
            if postproc_worker_enabled else InlinePostprocStrategy())

PostprocStrategy is the abstract base of the two; its docstring records how a server chooses (once, from num_postprocess_workers) and the known n > 1 recording difference below.

Deliberately unchanged — the eleven submission-side uses of postproc_worker_enabled: what generate_async receives (_postproc_params=... if ... else None, and the completion path's unconditional pass), filling postproc_args.tokenizer/num_prompt_tokens for in-server post-processing, the Responses store decision and its warning, plus the property itself. They decide what a request carries, not how a result is read; unifying them would change behavior.

Behavior

No behavior change. With more than one candidate (n > 1) the two modes already differ in what the frontend records (worker mode: only outputs[0] carries the result and metrics arrive merged; inline mode records metrics per candidate). That difference is documented in the PostprocStrategy docstring and deliberately not fixed here; it is a separate behavior-change PR.

The accessor (API review)

CompletionOutput.postprocess_result (read-only property, Any): what the post-processor returned for this output when post-processing ran in a postproc worker; None otherwise.
tests/unittest/api_stability/references/completion_output.yaml records the new property. references_committed/ is unchanged. Requesting the api-compatible label.

Moves

None.

CI bookkeeping

Multi-GPU table: none (no new directories; openai_server.py and executor/ are already listed, so the multi-GPU stages run on this PR)
CODEOWNERS: none
legacy-files.txt: none (the three edited source files are already listed; the new test file is a new-code file under the default lint)
Test lists: unittest/serve/test_postproc_strategy.py registered in tests/integration/test_lists/test-db/l0_cpu.yml (after the unittest/scripts entry)

In-flight work on the same code

Checked open PRs touching serve/openai_server.py, serve/postprocess_handlers.py and executor/result.py at branch time. "Typed chat completion core behind a thin HTTP wrapper" has not landed, and the chat SSE generator's error-handling change (#20867, merged) already moved the two chat stream decision points to their current shape, which this PR edits in place. The Responses streaming-generator rewrite (keepalive, sequence renumbering) has not landed; the Responses decision points are edited where they are today. Open PRs that touch nearby code but not the nine decision points: #21072, #21060, #14092 (Responses handlers), #21111 (chat path), #19461 (completion merge helper), #20946 (# type: ignore comments). Whichever lands second rebases.

Test Coverage

New (CPU, registered in l0_cpu.yml; scripts/test_to_stage_mapping.py names its four stages: CPU-Generic-arm-1, CPU-Generic-arm-2, CPU-Generic-x86-1, CPU-Generic-x86-2):

tests/unittest/serve/test_postproc_strategy.py: the flag→strategy mapping; the worker strategy reads outputs[0].postprocess_result of a real GenerationResultBase and accepts postproc_params=None; the inline strategy calls the post-processor exactly once and returns its value; worker apply_async returns the stored value without calling inline; inline apply_async awaits it exactly once; a server chooses from num_postprocess_workers (0 → inline, 2 → worker) through the real postproc_worker_enabled property and caches the choice; CompletionOutput(0).postprocess_result is None and returns the stored value.

Changed tests:

tests/unittest/grpc/openengine/test_metrics.py: its minimal generator args now explicitly set num_postprocess_workers=0, so the real server constructor can select the inline strategy during the HTTP metrics contract test.
tests/unittest/llmapi/test_executor.py: also asserts the public accessor returns the worker's value.
tests/unittest/llmapi/run_llm_with_postproc.py (GPU, postproc-worker e2e): reads the public accessor, exercising it end to end.
tests/unittest/llmapi/apps/test_anthropic_routes.py, tests/unittest/disaggregated/test_disagg_responses_api.py: the fake engine outputs now carry the public attribute name the worker strategy reads (one-word rename each; assertions unchanged).

Commands run locally (CPU-only dev box, source-built artifacts matching the base commit; each file in its own pytest process, as the CPU stage runs them):

pytest -m cpu_only tests/unittest/serve/test_postproc_strategy.py -q — 8 passed
pytest -m cpu_only tests/unittest/llmapi/test_executor.py -q — 44 passed (needs LLM_MODELS_ROOT: 3 of its 44 tests load tokenizers)
pytest tests/unittest/api_stability -q — 64 passed (run without -m, so the serve API and CLI references are checked too)
pytest -m cpu_only tests/unittest/disaggregated/test_disagg_responses_api.py tests/unittest/disaggregated/test_openai_disagg_service.py tests/unittest/disaggregated/test_openai_server_stream_errors.py tests/unittest/disaggregated/test_router.py -q — 322 passed
pytest -m cpu_only tests/unittest/disaggregated -q — 1411 passed, 1 skipped, 2 xfailed
pytest -m cpu_only tests/unittest/llmapi/apps -q — 2260 passed, 1 failed: test_render_standalone.py::TestCpuOnlyStartup::test_the_launcher_loads_a_cuda_stub_only_when_the_driver_is_missing fails identically on the unmodified base commit in this environment (driver-detection test; unrelated to this change)
git grep -n "_postprocess_result" -- tensorrt_llm/serve — no output; grep -c "postproc_worker_enabled" tensorrt_llm/serve/openai_server.py — 13
python scripts/test_to_stage_mapping.py --tests "unittest/serve/test_postproc_strategy.py" — names the four CPU stages listed above; grep -c "pytest.mark.cpu_only" — 1
pre-commit run --files <all changed> — twice, all hooks pass, no file re-modified
A differential property check (Hypothesis, 300 generated cases) that the strategy dispatch returns/raises exactly what the old per-site expressions did — including post-processor call counts and arguments, and that the Responses inline coroutine is created only in inline mode — against the shipped code (AST-extracted), gated by 7 planted mutations (flag inversion, wrong post-processor arguments, missing await, accessor returning None, wrong output index, worker-mode inline call): 7/7 detected

Follow-up validation on 5806927953: reproduced the CPU CI http_enablement failure before the fixture correction; 73 CPU tests passed across OpenEngine metrics, post-processing strategy, runtime control, RL control authentication, and sleep-mode construction after the correction. Focused lint and commit hooks passed. The all-files hook run passed its formatting/lint/consistency checks, but type checking could not complete because mypy is unavailable and that hook interpreter has a native-library ABI mismatch.

GPU coverage is left to CI: the e2e wrappers that run every handler with num_postprocess_workers 0 and 2 (chat and completions on the single-GPU stage; harmony and Responses on the H100 stage), and the postproc-worker example test that now reads the public accessor.

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

🤖 Generated with Claude Code

…ce per server OpenAIServer decided nine times, in four handlers, whether a response's post-processed value comes from a postproc worker (read the value the worker stored on outputs[0]) or must be computed in the server process (call the post-processor inline). The flag never changes after construction, so the server now chooses a strategy object once: select_postproc_strategy picks WorkerPostprocStrategy or InlinePostprocStrategy from num_postprocess_workers; __init__ pre-caches it and a cached property keeps server instances that unit tests build without __init__ working. The nine decision points become one strategy call each, and the conditional post-processor unpacks they fed are gone. CompletionOutput gains the read-only public accessor postprocess_result so the server no longer reads the executor's private field; the api-stability reference records the new property. The submission-side uses of postproc_worker_enabled (what generate_async receives, filling postproc_args for in-server post-processing, the Responses store decision) are deliberately unchanged. No behavior change. With more than one candidate the two modes still differ in what the frontend records (worker mode: only outputs[0] carries the result and metrics arrive merged; inline mode records metrics per candidate); that known difference is documented on the strategy class and left for a separate change. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

JunyiXu-nv · 2026-11-01T15:18:59Z

/bot run

coderabbitai · 2026-11-01T15:25:43Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Last reviewed commit: cbd31ea
Latest commit, not reviewed: 534ce27

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

Resume reviews
Trigger review

No actionable comments were generated in the recent review.

Walkthrough

Completion outputs now expose the stored postprocessed value. OpenAI server handlers select and use worker or inline strategies for chat, completion, Harmony, and Responses API processing.

Changes

Postprocessing strategy

Layer / File(s)	Summary
Result access and strategy behavior `tensorrt_llm/executor/result.py`, `tensorrt_llm/serve/postprocess_handlers.py`, `tests/unittest/serve/test_postproc_strategy.py`, `tests/unittest/api_stability/references/completion_output.yaml`, `tests/unittest/llmapi/test_executor.py`	`CompletionOutput.postprocess_result` returns the stored value. Worker and inline strategies implement synchronous and asynchronous processing. Tests cover strategy behavior, selection, and the public property.
Server strategy selection and response handling `tensorrt_llm/serve/openai_server.py`, `tests/unittest/serve/test_postproc_strategy.py`, `tests/unittest/grpc/openengine/test_metrics.py`, `tests/unittest/disaggregated/test_disagg_responses_api.py`, `tests/unittest/llmapi/apps/test_anthropic_routes.py`, `tests/unittest/llmapi/run_llm_with_postproc.py`, `tests/integration/test_lists/test-db/l0_cpu.yml`	`OpenAIServer` caches a selected strategy and uses it for chat, completion, Harmony, and Responses API processing. Supporting fixtures use the public property, the CPU test list includes the new strategy tests, and the metrics test arguments set `num_postprocess_workers` to `0`.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Refactor

Sequence Diagram(s)

sequenceDiagram
  participant OpenAIServer
  participant select_postproc_strategy
  participant PostprocStrategy
  participant CompletionOutput
  OpenAIServer->>select_postproc_strategy: select strategy from worker setting
  select_postproc_strategy-->>OpenAIServer: return strategy
  OpenAIServer->>PostprocStrategy: apply generated result
  PostprocStrategy->>CompletionOutput: read postprocess_result when using worker strategy
  CompletionOutput-->>PostprocStrategy: return stored value
  PostprocStrategy-->>OpenAIServer: return processed result

Possibly related issues

NVIDIA/TensorRT-LLM#21309 reports repeated streamed text for n > 1 in the trtllm-serve flow. This change refactors streaming postprocessing but does not fix the reported detokenization cause.

Suggested reviewers: juney-nvidia

Merge Risk: Minimal · previous review cbd31

No actionable merge-blocking risk was established in the reviewed changes.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 18.75% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Title check		The title clearly identifies the ticket, change type, and primary change: selecting the trtllm-serve post-processing path once per server.
Description check		The description includes the required Description, Test Coverage, and PR Checklist sections. It explains the motivation, implementation, API change, behavior, tests, CI status, and known limitations.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 18.75% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 16 functions across 9 files.

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches

Generate unit tests (beta)

Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Recent review info

Run configuration

Source mode: Auto

Repository: NVIDIA/TensorRT-LLM

Branch: main

File: .coderabbit.yaml

Commit: f6405a5

Layers: Repository YAML

Review profile: CHILL

Plan: Enterprise

Run ID: `db648bc3-6938-4f36-bf35-0b67730d0775`

Commits

Reviewing files that changed from the base of the PR and between `6fb090c` and `cbd31ea`.

Files selected for processing: 6

`tensorrt_llm/executor/result.py`

`tensorrt_llm/serve/openai_server.py`

`tensorrt_llm/serve/postprocess_handlers.py`

`tests/unittest/disaggregated/test_disagg_responses_api.py`

`tests/unittest/llmapi/apps/test_anthropic_routes.py`

`tests/unittest/llmapi/run_llm_with_postproc.py`

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

tensorrt-cicd · 2026-11-01T15:26:07Z

PR_Github #81846 [ run ] triggered by Bot. Commit: e6df420 Link to invocation

tensorrt-cicd · 2026-11-01T18:58:22Z

PR_Github #81846 [ run ] completed with state FAILURE. Commit: e6df420
/LLM/main/L0_MergeRequest_PR pipeline #67498 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

JunyiXu-nv · 2026-11-01T19:39:09Z

/bot run --disable-multi-gpu-test

tensorrt-cicd · 2026-11-01T19:46:47Z

PR_Github #81865 [ run ] triggered by Bot. Commit: 5806927 Link to invocation

tensorrt-cicd · 2026-11-01T23:58:55Z

PR_Github #81865 [ run ] completed with state SUCCESS. Commit: 5806927
/LLM/main/L0_MergeRequest_PR pipeline #67517 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

…tproc-strategy Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

JunyiXu-nv · 2026-11-04T12:53:32Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-11-04T12:59:42Z

PR_Github #82500 [ run ] triggered by Bot. Commit: 96571b7 Link to invocation

tensorrt-cicd · 2026-11-04T17:12:55Z

PR_Github #82500 [ run ] completed with state SUCCESS. Commit: 96571b7
/LLM/main/L0_MergeRequest_PR pipeline #68107 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…tproc-strategy Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

JunyiXu-nv · 2026-11-04T18:47:05Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-11-04T18:55:31Z

PR_Github #82556 [ run ] triggered by Bot. Commit: 6fb090c Link to invocation

…tproc-strategy Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

JunyiXu-nv · 2026-11-04T20:22:14Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-11-04T20:32:54Z

PR_Github #82577 [ run ] triggered by Bot. Commit: cbd31ea Link to invocation

tensorrt-cicd · 2026-11-04T20:36:02Z

PR_Github #82556 [ run ] completed with state ABORTED. Commit: 6fb090c

Link to invocation

tensorrt-cicd · 2026-11-05T01:01:04Z

PR_Github #82577 [ run ] completed with state FAILURE. Commit: cbd31ea
/LLM/main/L0_MergeRequest_PR pipeline #68174 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…tproc-strategy Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

JunyiXu-nv · 2026-11-05T07:10:29Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-11-05T07:19:43Z

PR_Github #82709 [ run ] triggered by Bot. Commit: d31dd69 Link to invocation

tensorrt-cicd · 2026-11-05T12:38:20Z

PR_Github #82709 [ run ] completed with state FAILURE. Commit: d31dd69
/LLM/main/L0_MergeRequest_PR pipeline #68294 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…tproc-strategy Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

JunyiXu-nv · 2026-11-05T12:39:33Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-11-05T12:46:45Z

PR_Github #82808 [ run ] triggered by Bot. Commit: 0230b12 Link to invocation

tensorrt-cicd · 2026-11-05T16:56:51Z

PR_Github #82808 [ run ] completed with state FAILURE. Commit: 0230b12
/LLM/main/L0_MergeRequest_PR pipeline #68378 completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Either:

Wait for the PR to be fully approved — the label is added automatically once approval is complete. Having unresolved open comments is fine, or
If needed, ask a member of NVIDIA/trt-llm-ci-approvers to add the label manually.
Then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

Pin main at 1ceb4e6. Keep both source changes and use the existing unittest/serve directory registration instead of a redundant per-file entry. Validation: 51 focused postprocessing and disaggregated tests passed; focused pre-commit passed. Protected API collection is blocked by the missing preallocate_nccl_window_buffer.bytes operator, identically on pinned main. No private-access baseline files are introduced because NVIDIA#21393 has not landed. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

Pin main at 1ceb4e6. Keep both source changes and the main-side package license headers. Drop redundant unittest/serve per-file registrations and the duplicate stand-in health tests, already covered by the real-constructor system endpoint tests. Validation: 122 focused tests passed, including 22 public runtime health and 14 private-access lint tests. The 3 VisualGen setup errors match pinned main exactly (12 shared health tests passed on both). Protected API collection also hits the same missing preallocate_nccl_window_buffer.bytes operator on main. Focused pre-commit passed. Keep the current 9 postprocess accesses and _save_bytes baseline location because NVIDIA#21392 and NVIDIA#21552 have not landed. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

JunyiXu-nv · 2026-11-12T08:20:55Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-11-12T08:27:19Z

PR_Github #84413 [ run ] triggered by Bot. Commit: 534ce27 Link to invocation

JunyiXu-nv requested review from a team as code owners November 1, 2026 15:18

JunyiXu-nv requested a review from QiJune November 1, 2026 15:19

github-actions Bot assigned JunyiXu-nv Nov 1, 2026

JunyiXu-nv requested review from BowenFu, allisonlim-nv, bo-nv, dpitman-nvda, niukuo and reasonsolo November 1, 2026 15:19

JunyiXu-nv added the api-compatible Accepted LLM API contract change that is backwards-compatible label Nov 1, 2026

[TRTLLM-17273][test] Initialize postprocessing mode in metrics fixture

5806927

Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

JunyiXu-nv requested a review from a team as a code owner November 1, 2026 19:38

BowenFu approved these changes Nov 2, 2026

View reviewed changes

JunyiXu-nv mentioned this pull request Nov 3, 2026

[TRTLLM-17263][refactor] Split the chat handler into a typed core and lift the SSE generator into serve/_base #21515

Draft

1 task done

[None][chore] Merge main (e62e13c) into dev-junyix-feat-serve-one-pos…

96571b7

…tproc-strategy Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

JunyiXu-nv mentioned this pull request Nov 4, 2026

[TRTLLM-17267][refactor] Move the completions handler into a serving class #21640

Draft

1 task done

[None][chore] Merge main (a0ce6c8) into dev-junyix-feat-serve-one-pos…

6fb090c

…tproc-strategy Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

[None][chore] Merge main (f6405a5) into dev-junyix-feat-serve-one-pos…

cbd31ea

…tproc-strategy Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

[None][chore] Merge main (38ab02f) into dev-junyix-feat-serve-one-pos…

d31dd69

…tproc-strategy Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

[None][chore] Merge main (51798b8) into dev-junyix-feat-serve-one-pos…

0230b12

…tproc-strategy Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

Conversation

JunyiXu-nv commented Nov 1, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dev Engineer Review

QA Engineer Review

Per-File QA Perspective

Description

The nine decision points

Behavior

The accessor (API review)

Moves

CI bookkeeping

In-flight work on the same code

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

JunyiXu-nv commented Nov 1, 2026

Uh oh!

coderabbitai Bot commented Nov 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

tensorrt-cicd commented Nov 1, 2026

Uh oh!

tensorrt-cicd commented Nov 1, 2026

Uh oh!

JunyiXu-nv commented Nov 1, 2026

Uh oh!

tensorrt-cicd commented Nov 1, 2026

Uh oh!

tensorrt-cicd commented Nov 1, 2026

Uh oh!

JunyiXu-nv commented Nov 4, 2026

Uh oh!

tensorrt-cicd commented Nov 4, 2026

Uh oh!

tensorrt-cicd commented Nov 4, 2026

Uh oh!

JunyiXu-nv commented Nov 4, 2026

Uh oh!

tensorrt-cicd commented Nov 4, 2026

Uh oh!

JunyiXu-nv commented Nov 4, 2026

Uh oh!

tensorrt-cicd commented Nov 4, 2026

Uh oh!

tensorrt-cicd commented Nov 4, 2026

Uh oh!

tensorrt-cicd commented Nov 5, 2026

Uh oh!

JunyiXu-nv commented Nov 5, 2026

Uh oh!

tensorrt-cicd commented Nov 5, 2026

Uh oh!

tensorrt-cicd commented Nov 5, 2026

Uh oh!

JunyiXu-nv commented Nov 5, 2026

Uh oh!

tensorrt-cicd commented Nov 5, 2026

Uh oh!

tensorrt-cicd commented Nov 5, 2026

Uh oh!

JunyiXu-nv commented Nov 12, 2026

Uh oh!

tensorrt-cicd commented Nov 12, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

JunyiXu-nv commented Nov 1, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Nov 1, 2026 •

edited

Loading