Skip to content

[None][feat] Support Mooncake context reuse with attention DP and EAGLE-style MTP - #20327

Draft
dongfengy wants to merge 4 commits into
NVIDIA:mainfrom
dongfengy:codex/glm-upstream-p05-mooncake-adp-20261014
Draft

dongfengy wants to merge 4 commits into
NVIDIA:mainfrom
dongfengy:codex/glm-upstream-p05-mooncake-adp-20261014

Conversation

@dongfengy

Copy link
Copy Markdown
Collaborator

Description

Enable Mooncake context reuse with attention-DP owners and unified-pool EAGLE-style MTP. Route connector hooks to the local request owner, carry the MTP prefill layout and lookahead/alignment requirements, and reserve completion capacity while preserving cancellation and preemption lifetimes.

Support is limited to context-only speculative requests with a cache transceiver and an actual unified target/draft KV pool. Vanilla MTP and separate draft pools remain unsupported; attention DP requires PP=1 and CP=1. Configuration must resolve to the supported model mode after loading its checkpoint metadata.

The implementation uses main's pool/role API and retains checkpoint identity, reuse scope, LoRA/multimodal keys, worker readiness, and shutdown draining. It does not restore the old donor CLI or old connector configuration.

Dependency: #20326. This draft targets main and includes the prerequisite commits; merge/rebase after that dependency lands.

Test Coverage

Carries the source ADP, connector-layout, Mooncake, and native reuse-alignment cases. Python AST parsing, fatal Python static checks, git diff --check, and independent source review passed. No builds, tests, GPU execution, or benchmarks were run. Validate multi-rank reuse, chunk boundaries, cancellation, transfer/recompute boundaries, and MTP correctness before leaving draft.

PR Checklist

  • Description explains the supported behavior and scope.
  • Commits include DCO sign-off.
  • Runtime correctness and failure-path validation completed.
  • Confirm CI platform and test-list coverage for the carried regression cases before leaving draft.
  • Please check this after reviewing the above items as appropriate for this PR.

Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>
Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>
… API

Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>
Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api-compatible Accepted LLM API contract change that is backwards-compatible

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant