Repository navigation
[3/10] fix(data): omit the implicit Qwen3 instruct system prompt - #959
Merged
Merged
Conversation
jiapingW
requested review from
FrankLeeeee,
shuaills,
sleepcoo and
zyksir
as code owners
October 14, 2026 03:27
yushengsu-thu
added a commit
to yushengsu-thu/SpecForge
that referenced
this pull request
Oct 15, 2026
… into the MoE 8-expert experiment branch Brings is_causal (configurable draft attention range/direction), dspark_flatten_projection / dspark_cache_projection, ddp_bucket_cap_mb, the DDP once-per-window synchronization (sgl-project#958, which replaces this branch's prepare_micro_step fix; the matching test is dropped), the removal of the implicit qwen3-instruct system prompt (sgl-project#959), balanced consumer optimizer windows (sgl-project#966) and the DSpine algorithm. Conflicts in specforge/training/backend.py and controller.py were resolved in favour of the PR. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
yushengsu-thu
added a commit
to yushengsu-thu/SpecForge
that referenced
this pull request
Oct 15, 2026
…yers 1-7 of the sgl-project#942 split), matching the dense-1/8 branch The combined PR sgl-project#942 merge was reverted in the previous commit; this takes the split stack instead (sgl-project#957 streaming-read fixes, sgl-project#958 DDP sync once per accumulation window, sgl-project#959 no implicit qwen3-instruct system prompt, sgl-project#960 is_causal, sgl-project#961 fused DFlash head, sgl-project#962 selective checkpoint contexts, sgl-project#963 shared flatten_vocab_projection / cache_vocab_projection), so the MoE arm carries exactly the same upstream code as the dense-1/8 arm. Conflicts in specforge/training/backend.py and controller.py resolved in the stack's favour. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
maocheng23
approved these changes
Oct 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The qwen3-instruct template injected a default system prompt into conversations that did not supply one.
Modifications
Use an empty default for qwen3-instruct and update standard-conversation and tool-use token/loss-mask ground truths. Keep the separate qwen template unchanged.
Stack and dependencies
Layer 3/10 split from #942. Previous layer: #958 (
codex/pr942-02-ddp-accumulation). Review the diff against this PR's base branch; it contains only this layer. The full stack index is maintained in #942.Actual prerequisites: No earlier feature is required; this layer uses existing main APIs.
The linear stack is the review/merge order. Model-specific optimizations are independently opt-in; the projection-checkpointing and attention layers expose shared capabilities. When merging a lower PR with squash, restack the remaining branches before merging them so already-reviewed commits do not reappear. The repository GPU test workflow only auto-runs for PRs targeting main; upper layers need that CI run when retargeted, while lint runs for all PRs.
Source: #942 at
211b87fc7d707e3a628cb95de5c73340af863c04, split onto mainfdcc2eddf1b56b5b24a5a15d3abcb366adfeb5aa. The original branch is preserved.Accuracy Test
d3cdb81d37a8: 6 tests run, zero failures/errors on H20 with Python 3.11.12 / PyTorch 2.13.0+cu130.eaa14f102d6a: 405 integration tests run, zero failures/errors; 2 skipped on H20 with Python 3.11.12 / PyTorch 2.13.0+cu130. Each module runs in a fresh process to isolate process-global distributed state. This is targeted regression coverage, not a full-suite or downstream task-accuracy claim.Layer command (run in this branch's checkout,
PYTHONPATH=.):Qwen3 tokenizer regressions reuse the cached official tokenizer with
HF_HUB_OFFLINE=1andHF_HUB_CACHEpointing to that cache. No training job was launched.Benchmark & Profiling
No new end-to-end throughput or downstream accuracy benchmark was run for this split. Model-side tests exercise numerical parity and gradients; historical measurements remain in #942 with their original scope and caveats.
Checklist