Skip to content

[None][fix] Reserve and reclaim speculative KV slack during overlap - #20326

Draft
dongfengy wants to merge 2 commits into
NVIDIA:mainfrom
dongfengy:codex/glm-upstream-p03-mtp-capacity-20261014
Draft

dongfengy wants to merge 2 commits into
NVIDIA:mainfrom
dongfengy:codex/glm-upstream-p03-mtp-capacity-20261014

Conversation

@dongfengy

Copy link
Copy Markdown
Collaborator

Description

Speculative overlap can advance device-side KV positions beyond host bookkeeping before draft KV is appended. Reserve an additional verification window, track the full allocated draft width through padding and unpaired draft growth, and reclaim unused slots without shrinking below non-helix committed history.

Use the same upper bound for runtime quotas, static cache sizing, and block-offset metadata. Also fail early when disaggregated one-model MTP through the legacy cache transceiver puts its draft layers outside the transferred target pool; connector-managed layouts keep their own validation.

Test Coverage

The existing disaggregated transition capacity case now expects 137 slots for a 128-token capacity and four draft tokens. Python AST parsing and git diff --check passed; independent source review checked runtime allocation against the common sizing helper.

No builds, tests, model execution, or benchmarks were run. Before leaving draft, validate overlap across block boundaries, padding/ragged speculation, dynamic draft reserves, disaggregated MTP, and helix-specific capacity accounting.

PR Checklist

  • Description explains the behavior and scope.
  • Commits include DCO sign-off.
  • Runtime regression validation completed.
  • Please check this after reviewing the above items as appropriate for this PR.

Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>
Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant