Repository navigation
Conversation
jiapingW
changed the title
yushengsu-thu
added a commit
to yushengsu-thu/SpecForge
that referenced
this pull request
Oct 15, 2026
…yers 1-7 of the sgl-project#942 split), matching the dense-1/8 branch The combined PR sgl-project#942 merge was reverted in the previous commit; this takes the split stack instead (sgl-project#957 streaming-read fixes, sgl-project#958 DDP sync once per accumulation window, sgl-project#959 no implicit qwen3-instruct system prompt, sgl-project#960 is_causal, sgl-project#961 fused DFlash head, sgl-project#962 selective checkpoint contexts, sgl-project#963 shared flatten_vocab_projection / cache_vocab_projection), so the MoE arm carries exactly the same upstream code as the dense-1/8 arm. Conflicts in specforge/training/backend.py and controller.py resolved in the stack's favour. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
maocheng23
reviewed
Oct 21, 2026
maocheng23
left a comment
maocheng23
left a comment
Collaborator
There was a problem hiding this comment.
Same cmt with PR 961, here shall we refactor the loss function first, then decide if we need this selective checkpoint?
Also, I think we need to split the checkpoint with chunking. Not sure if that is still beneficial.
jiapingW
force-pushed
the
codex/pr942-05-dflash-fused-head
branch
from
October 22, 2026 00:46
a7e88e7 to
903d860
Compare
jiapingW
requested review from
FlamingoPg,
shuaills,
sleepcoo and
zyksir
as code owners
October 22, 2026 00:46
jiapingW
changed the title
jiapingW
force-pushed
the
codex/pr942-06-projection-checkpointing
branch
from
October 22, 2026 00:58
f85e6ae to
f5b38c6
Compare
jiapingW
changed the base branch from
codex/pr942-05-dflash-fused-head
to
main
October 22, 2026 00:58
Collaborator
Author
|
It has a small gain about 1-2% speedup compared to checkpoint. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add an optional
context_fntocheckpointed_chunk_reduceand a DFlash-family selective checkpoint policy that retains eligible frozen vocabulary-projection outputs during recomputation.This PR only supplies the shared API and policy; no training algorithm enables projection caching here. Existing chunking/checkpoint behavior remains unchanged. The PR targets main directly and does not depend on the earlier stacked changes.
Validation
38 targeted tests passed on H20 after rebuilding on main:
Coverage includes context integration, frozen/trainable-head gradient parity, chunk reduction, and existing loss behavior. Black/isort and
git diff --checkpassed.Performance scope
This change does not establish an end-to-end performance benefit. The usefulness of projection caching should be evaluated after the logits-loss refactor in #961, separately from chunk size and the checkpoint boundary. Chunking and checkpointing are still coupled in the existing API; this PR has not yet addressed that review suggestion.