Repository navigation
[None][fix] Retire timed-out disaggregated KV receives while idle - #13
Closed
SimengLiu-nv wants to merge 1 commit into
Closed
SimengLiu-nv wants to merge 1 commit into
SimengLiu-nv wants to merge 1 commit into
Conversation
…generation server Without in-flight cancellation, the generation-side KV-transfer timeout only flags the request in check_transfer_timeouts. The cancel and the error response happen in the response pass, which runs only after a forward step. A generation server with nothing else scheduled therefore kept a request whose KV never arrived until the client gave up, even after logging "Observed timeout on generation request". DisaggTransferCoordinator.poll_gen_transfers, which runs at every loop head, now cancels and fails the receives that check_transfer_timeouts flagged whenever in-flight cancellation is inactive. This covers any transceiver without that capability, including KvCacheTransceiverV2 and TRTLLM_DISAGG_ENABLE_INFLIGHT_CANCEL=1 on an unsupporting transceiver. A cancel that finishes at once fails the request through fail_timed_out, inline or buffered for handle_timeouts_synced under multi-rank ADP. A cancel the transceiver refuses is asked only once: the Python transceiver has then marked the session cancelled and the next reap_gen_receives fails the request through the transfer-error path, while the C++ transceiver cannot cancel a receive already in flight and would only repeat its "Cannot cancel request" warning at the idle loop rate. Requests the user cancelled are left to the cancel path. No collective is added. The in-flight cancellation path is unchanged. fail_timed_out now moves a request still in DISAGG_GENERATION_TRANS_IN_PROGRESS to DISAGG_TRANS_ERROR before failing it. cancel_request does not change the state, so PyExecutor._handle_errors parked the failure for a transfer that no longer existed, and _sweep_transfer_held_failures then dropped it because the request had already left active_requests: no error response was sent and the KV was never freed. This also fixes the same loss on the existing response-pass timeout path. Main's idle request cancellation (NVIDIA#20702) retires requests the client cancelled while no batch runs, where the transceiver reports supports_idle_request_cancellation. It does not act on transfer timeouts, which this change covers; client cancels stay on that path. As on the response-pass timeout path, cancel_request decides whether a receive can stop, and refuses one that is still writing. Signed-off-by: Simeng Liu <simengl@nvidia.com>
This was referenced Nov 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Superseded by NVIDIA#21659 targeting
NVIDIA/TensorRT-LLM:main, per the requested upstream destination. The source branch is retained.A generation executor with no forward work still needs to progress timed-out KV receives and deliver a terminal error.
This PR extracts idle receive timeout handling, transfer retirement, and coordinator/executor regression coverage. It is independent of the connector and HTTP stream changes.
Extracted from NVIDIA/TensorRT-LLM#21195 at
4918c76ee1958bf0b6d01840588f742a7d0e743b, rooted at fork maincbef70dbc9879798b4106c694928d59976cf0ac1. Original feature commits are retained where possible, with split-specific activation and review fixes in separate commits.Part 5 of 5: C18. Targets
main.Draft coordination gate: NVIDIA/TensorRT-LLM#21079 remains open (observed head
64a0641700d80f02b11693a370734c656913dc0c). Reconcile its group-consensus retirement andcancels_timed_out_gen_receivescontract before upstream merge. This extraction does not claim that rank-local cancellation is safe to combine with that change.Validation: scoped pre-commit, Python parse, and diff checks passed. Native/GPU/MPI tests have not run. Before merge, validate idle timeout without a forward step, exactly one error response, accepted/refused cancellation, ownership before page release, synchronized rank retirement, and PP/in-flight cancellation.
Five-PR split: runtime → Mooncake → scheduling. Independent: HTTP streams and idle receive timeouts. The source PR remains the umbrella.