Repository navigation
Conversation
Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>
Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>
… API Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>
Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>
2 of 5 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Enable Mooncake context reuse with attention-DP owners and unified-pool EAGLE-style MTP. Route connector hooks to the local request owner, carry the MTP prefill layout and lookahead/alignment requirements, and reserve completion capacity while preserving cancellation and preemption lifetimes.
Support is limited to context-only speculative requests with a cache transceiver and an actual unified target/draft KV pool. Vanilla MTP and separate draft pools remain unsupported; attention DP requires PP=1 and CP=1. Configuration must resolve to the supported model mode after loading its checkpoint metadata.
The implementation uses main's pool/role API and retains checkpoint identity, reuse scope, LoRA/multimodal keys, worker readiness, and shutdown draining. It does not restore the old donor CLI or old connector configuration.
Dependency: #20326. This draft targets main and includes the prerequisite commits; merge/rebase after that dependency lands.
Test Coverage
Carries the source ADP, connector-layout, Mooncake, and native reuse-alignment cases. Python AST parsing, fatal Python static checks,
git diff --check, and independent source review passed. No builds, tests, GPU execution, or benchmarks were run. Validate multi-rank reuse, chunk boundaries, cancellation, transfer/recompute boundaries, and MTP correctness before leaving draft.PR Checklist