[TRTLLM-17083][feat] Cancel paired GEN on CTX failure or cancellation by chienchunhung · Pull Request #20491 · NVIDIA/TensorRT-LLM

chienchunhung · 2026-10-16T19:01:41Z

Summary

Generation-first serving can leave one worker waiting after its paired request fails or the caller cancels. Cancel and drain pending sibling work while preserving worker errors and already-completed GEN responses.

Scope

Paired orchestration watches both legs, cancels pending work on failure/cancellation, and preserves completed non-streaming GEN responses without swallowing cancellation.
SSE observation detects completion and Responses API errors without altering forwarded bytes; incremental scanning and a mutable buffer avoid repeated copying/scanning of fragmented large events.
HTTP diagnostics preserve worker status and a bounded 2,048-byte streaming error prefix; streaming status errors remain non-retryable.
Delegated placement cleanup drains cancelled selection before queuing /finish with the original request ID, including content-key selection, without dispatching inference.

HTTP cancellation does not prove physical KV quiescence or authorize memory reuse. Other retry behavior, including main's safeguards from #20938, is unchanged; coordinated paired retries remain separate work. Coordinator cleanup retains asynchronous expiry fallback on outages/queue overflow; local timeout does not fence remote selection.

Verification

Current head: ca7ca1e5bda9b5f37da532b63e73297288029290.

Service tests cover failure/cancellation races, completed responses and fragmented SSE. Five new SSE cases cover a 1 MiB event's scan bound, split delimiters and buffered error tails.
Client tests cover diagnostics and retry compatibility; seven real-HTTP cases check abort delivery/reservation release, and ten router cases check cancellation/selection races.
Historical native evidence at 708ff2707d: CI #66922 passed 407 tests on x86 and 407 on ARM across these four modules. These are module results, not a full-pipeline pass or verification of the new SSE fix.
Current-head full self-review, independent review and applicable pre-commit checks passed. Native targeted execution was blocked by unavailable PyTorch. Full CI requested with fail-fast disabled; Blossom is queued for authorization/scanning, so Jenkins launch and test results remain pending.

Notes

Complements #20459 on release/1.3; this PR targets main. #20702 handles idle-worker cancellation separately; physical retirement safeguards such as #19674 remain necessary.

Unchanged follow-up: streaming CTX placement acquired before GEN selection can remain until expiry if cancellation occurs before CTX dispatch.

Reviewed the contribution checklist; scope and verification limitations are documented.

chienchunhung · 2026-10-16T19:47:19Z

/bot run

tensorrt-cicd · 2026-10-16T19:54:20Z

PR_Github #78549 [ run ] triggered by Bot. Commit: 4df975b Link to invocation

coderabbitai · 2026-10-16T22:58:47Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Last reviewed commit: 9edf8b2
Latest commit, not reviewed: 708ff27

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

Walkthrough

The HTTP client now handles errors for streaming and non-streaming responses. The generation-first service coordinates context and generation requests, detects in-band generation SSE errors, and cleans up unfinished tasks.

Changes

Paired request handling

Layer / File(s)	Summary
Streaming coordination and SSE error detection `tensorrt_llm/serve/openai_disagg_service.py`, `tests/unittest/disaggregated/test_openai_disagg_service.py`	The service observes complete generation SSE error events across chunks while preserving the original stream. It runs context and generation work concurrently and cancels pending context work when it detects a generation error. Tests cover event parsing, stream forwarding, failures, and cancellation.
Non-streaming coordination and cleanup `tensorrt_llm/serve/openai_disagg_service.py`, `tests/unittest/disaggregated/test_openai_disagg_service.py`, `tests/unittest/disaggregated/test_paired_request_cancellation.py`	The service can return a completed generation response after a context failure. Other failures and cancellations remain observable, and unfinished tasks are cancelled and drained. Tests cover task outcomes and paired worker cancellation.

HTTP client error responses

Layer / File(s)	Summary
Shared HTTP status handling `tensorrt_llm/serve/openai_client.py`, `tests/unittest/disaggregated/test_disagg_openai_client.py`	The client handles HTTP status errors before the streaming and non-streaming response branches. Streaming diagnostic reads are limited to 2048 bytes. Read errors after a streaming rejection propagate without retry. Tests check error details, retry behavior, and request cleanup.

Priority: Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant OpenAIDisaggService
  participant ContextRequest
  participant GenerationRequest
  Client->>OpenAIDisaggService: Send generation-first streaming request
  OpenAIDisaggService->>ContextRequest: Start context request
  OpenAIDisaggService->>GenerationRequest: Start generation stream
  GenerationRequest-->>OpenAIDisaggService: Forward chunks and inspect SSE events
  OpenAIDisaggService->>ContextRequest: Cancel after detecting a generation error event
  OpenAIDisaggService-->>Client: Return generation stream

Merge Risk: Low · up to 9edf8

Rejected streaming responses now read a bounded error body, and paired work is cancelled on failure. A short worker error body that arrives in several fragments may be cut off in the reported error message. This does not affect the success path or cancellation behavior, so the change is mergeable with a small follow-up.

Pre-merge checks: 4

Check name	Status	Explanation
Title check		The title clearly identifies the main change: canceling paired GEN work when CTX fails or is canceled. It uses the required ticket and type format and is concise.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Description check		The description is complete and relevant. It explains the problem, solution scope, limitations, test coverage, CI status, and checklist review. It uses equivalent headings instead of the template's ex…

Full details: Description check

Explanation

The description is complete and relevant. It explains the problem, solution scope, limitations, test coverage, CI status, and checklist review. It uses equivalent headings instead of the template's exact "Description" and "Test Coverage" headings, but the required information is present.

Finishing Touches

🧪 Generate unit tests (beta)

Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

tensorrt-cicd · 2026-10-17T00:05:30Z

PR_Github #78549 [ run ] completed with state FAILURE. Commit: 4df975b
/LLM/main/L0_MergeRequest_PR pipeline #64727 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

mikeiovine

Stamp on behalf of runtime devs, delegating proper review to @NVIDIA/trt-llm-disagg-devs

chienchunhung · 2026-10-20T00:05:31Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-20T00:12:59Z

PR_Github #79059 [ run ] triggered by Bot. Commit: f870cd4 Link to invocation

tensorrt-cicd · 2026-10-20T07:24:57Z

PR_Github #79059 [ run ] completed with state FAILURE. Commit: f870cd4
/LLM/main/L0_MergeRequest_PR pipeline #65127 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-10-20T17:27:06Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-20T17:34:30Z

PR_Github #79328 [ run ] triggered by Bot. Commit: f870cd4 Link to invocation

tensorrt-cicd · 2026-10-20T19:31:59Z

PR_Github #79328 [ run ] completed with state SUCCESS. Commit: f870cd4
/LLM/main/L0_MergeRequest_PR pipeline #65335 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-10-20T21:27:30Z

/bot run --disable-fail-fast

nv-xtf

LGTM

tensorrt-cicd · 2026-10-27T08:30:58Z

PR_Github #80780 [ run ] completed with state SUCCESS. Commit: 41e8f0a
/LLM/main/L0_MergeRequest_PR pipeline #66520 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung · 2026-10-28T18:49:41Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-28T18:57:29Z

PR_Github #81232 [ run ] triggered by Bot. Commit: 708ff27 Link to invocation

tensorrt-cicd · 2026-10-29T00:42:00Z

PR_Github #81232 [ run ] completed with state FAILURE. Commit: 708ff27
/LLM/main/L0_MergeRequest_PR pipeline #66922 completed with status: 'FAILURE'

CI Report

⚠️ Resource Maintenance — Test Coverage Reduced:

Resource maintenance in LLM/main/L0_Test-x86_64-Single-GPU #10093: L40S-PyTorch-2 is temporarily paused due to Kubernetes agent provisioning and node issues.
Affected stages:
L40S-PyTorch-2

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-10-29T18:18:01Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-29T18:27:30Z

PR_Github #81475 [ run ] triggered by Bot. Commit: 708ff27 Link to invocation

tensorrt-cicd · 2026-10-29T22:56:12Z

PR_Github #81475 [ run ] completed with state SUCCESS. Commit: 708ff27
/LLM/main/L0_MergeRequest_PR pipeline #67146 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung · 2026-10-30T17:56:22Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-30T18:03:22Z

PR_Github #81727 [ run ] triggered by Bot. Commit: ca7ca1e Link to invocation

tensorrt-cicd · 2026-10-31T00:06:27Z

PR_Github #81727 [ run ] completed with state SUCCESS. Commit: ca7ca1e
/LLM/main/L0_MergeRequest_PR pipeline #67387 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-10-31T01:21:47Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-31T01:29:01Z

PR_Github #81778 [ run ] triggered by Bot. Commit: ca7ca1e Link to invocation

tensorrt-cicd · 2026-10-31T02:01:51Z

PR_Github #81778 [ run ] completed with state SUCCESS. Commit: ca7ca1e
/LLM/main/L0_MergeRequest_PR pipeline #67436 completed with status: 'SUCCESS'

CI Report

Link to invocation

…t chunk with an HTTP error A generation server whose KV-cache transfer fails raises in its stream generator after the 200 headers. The disaggregated server relayed that as a 200 whose body was cut off, so the client saw a truncated stream instead of an error. _wrap_entry_point now pulls a relayed stream's first chunk before it sends the headers. A chat or completion stream that fails before its first chunk reaches _handle_exception and becomes an HTTP error. The returned relay is already started, so closing it before it is iterated still closes the upstream stream and releases the router reservation. Responses streams keep their 200 plus an `error` event, because guard_responses_stream stays inside the first-chunk pull. /v1/messages keeps sending its headers at once: its reframed stream pings until the first chunk, and a failure before that chunk ends in an Anthropic `error` event, as before. total_responses now counts a held stream only once its first chunk arrived. The client's retry rules for generation legs are main's (NVIDIA#20938, NVIDIA#20491). The client tests gain the cases main does not cover: the chat and Responses request kinds failing after the headers over a real connection, the transient retry budget and TRTLLM_DISAGG_NO_RETRY for an undelivered leg, and three more possibly-delivered errors. disagg-serving.md describes the retry rules, including the frontend-affinity fallback that TRTLLM_DISAGG_NO_RETRY does not disable. Signed-off-by: Simeng Liu <simengl@nvidia.com>

github-actions Bot assigned chienchunhung Oct 16, 2026

chienchunhung changed the title ~~[None][fix] Cancel paired GEN requests when CTX terminates~~ [None][fix] Cancel paired GEN on CTX failure or cancellation Oct 16, 2026

chienchunhung requested review from Shixiaowei02, bo-nv, chuangz0, pcastonguay and schetlur-nv October 16, 2026 19:54

chienchunhung changed the title ~~[None][fix] Cancel paired GEN on CTX failure or cancellation~~ [TRTLLM-17083[fix] Cancel paired GEN on CTX failure or cancellation Oct 16, 2026

chienchunhung changed the title ~~[TRTLLM-17083[fix] Cancel paired GEN on CTX failure or cancellation~~ [TRTLLM-17083][feature] Cancel paired GEN on CTX failure or cancellation Oct 16, 2026

chienchunhung changed the title ~~[TRTLLM-17083][feature] Cancel paired GEN on CTX failure or cancellation~~ [TRTLLM-17083][feat] Cancel paired GEN on CTX failure or cancellation Oct 16, 2026

chienchunhung marked this pull request as ready for review October 16, 2026 22:54

chienchunhung requested review from a team as code owners October 16, 2026 22:54

Shixiaowei02 reviewed Oct 19, 2026

View reviewed changes

Comment thread tensorrt_llm/serve/openai_disagg_service.py Resolved

mikeiovine approved these changes Oct 19, 2026

View reviewed changes

chienchunhung force-pushed the dev/pact-early-paired-cancellation branch from 4df975b to f870cd4 Compare October 20, 2026 00:04

chienchunhung mentioned this pull request Oct 20, 2026

[https://nvbugs/6621362][fix] Unblock Python disagg transfers stalled by a context-side timeout #20459

Merged

1 task done

chienchunhung requested a review from Shixiaowei02 October 20, 2026 00:21

chienchunhung mentioned this pull request Oct 20, 2026

[None][fix] Progress idle disaggregated worker cancellation #20702

Merged

1 task done

nv-xtf approved these changes Oct 27, 2026

View reviewed changes

This was referenced Oct 27, 2026

[None][fix] Responses: fail over-length streamed requests in-stream with context_length_exceeded #21060

Open

[None][fix] Disaggregated client: relay a worker's rejection once instead of retrying it #21157

Open

test: cover early cancellation of paired disaggregated requests

c497014

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung force-pushed the dev/pact-early-paired-cancellation branch from 41e8f0a to 708ff27 Compare October 28, 2026 18:48

bo-nv approved these changes Oct 30, 2026

View reviewed changes

Shixiaowei02 reviewed Oct 30, 2026

View reviewed changes

Comment thread tensorrt_llm/serve/openai_disagg_service.py Resolved

fix: close paired generation requests when context terminates

ca7ca1e

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung force-pushed the dev/pact-early-paired-cancellation branch from 708ff27 to ca7ca1e Compare October 30, 2026 17:55

chienchunhung enabled auto-merge (squash) October 30, 2026 18:02

chienchunhung merged commit 5ec25c5 into NVIDIA:main Oct 31, 2026
14 checks passed

This was referenced Nov 1, 2026

[None][fix] Disaggregated Anthropic Messages: start the stream and ping while the context phase runs #21388

Open

[TRTLLM-17262][refactor] Split the disaggregated server's entry-point wrapper into prepare, run, finish and serialize steps #21442

Draft

SimengLiu-nv mentioned this pull request Nov 3, 2026

[None][feat] Close the Mooncake store feature gap: async loads, attention DP, speculative prefill and KV connector support in KVCacheManagerV2 #21195

Closed

chienchunhung mentioned this pull request Nov 5, 2026

[None][feat] Add opt-in request-wide deadlines for disaggregated serving #21734

Open

Conversation

chienchunhung commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Scope

Verification

Notes

Uh oh!

chienchunhung commented Oct 16, 2026

Uh oh!

tensorrt-cicd commented Oct 16, 2026

Uh oh!

coderabbitai Bot commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

tensorrt-cicd commented Oct 17, 2026

Uh oh!

Uh oh!

mikeiovine left a comment

Choose a reason for hiding this comment

Uh oh!

chienchunhung commented Oct 20, 2026

Uh oh!

tensorrt-cicd commented Oct 20, 2026

Uh oh!

tensorrt-cicd commented Oct 20, 2026

Uh oh!

chienchunhung commented Oct 20, 2026

Uh oh!

tensorrt-cicd commented Oct 20, 2026

Uh oh!

tensorrt-cicd commented Oct 20, 2026

Uh oh!

chienchunhung commented Oct 20, 2026

Uh oh!

nv-xtf left a comment

Choose a reason for hiding this comment

Uh oh!

tensorrt-cicd commented Oct 27, 2026

Uh oh!

chienchunhung commented Oct 28, 2026

Uh oh!

tensorrt-cicd commented Oct 28, 2026

Uh oh!

tensorrt-cicd commented Oct 29, 2026

Uh oh!

chienchunhung commented Oct 29, 2026

Uh oh!

tensorrt-cicd commented Oct 29, 2026

Uh oh!

tensorrt-cicd commented Oct 29, 2026

Uh oh!

Uh oh!

chienchunhung commented Oct 30, 2026

Uh oh!

tensorrt-cicd commented Oct 30, 2026

Uh oh!

tensorrt-cicd commented Oct 31, 2026

Uh oh!

chienchunhung commented Oct 31, 2026

Uh oh!

tensorrt-cicd commented Oct 31, 2026

Uh oh!

tensorrt-cicd commented Oct 31, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

chienchunhung commented Oct 16, 2026 •

edited

Loading

coderabbitai Bot commented Oct 16, 2026 •

edited

Loading