[yushengsu-thu]

Stacked on kan/moe-3-qwen38 (the base of this PR), so the diff is only this branch's three commits: the drafter config + GB300 recipe, the serving-speed gate with its GB300 numbers, and the SGLang serving package of #941 (needed to serve the exports).

Conclusion

• Serving speed vs the official dense v1: no change (-1 to -2%). At a pinned accept length of 5.0 on one GB300, the 16 x 1024 top-2 DSpark MoE drafter serves at 321 / 1780 / 3698 output tok/s against the bf16 target (dense 327 / 1815 / 3752) and 391 / 2095 / 4206 against the FP8 target (dense 399 / 2143 / 4249) at c = 1 / 8 / 32, i.e. -5 / -35 / -54 tok/s (bf16). Kan's 512 x 512 top-10 drafter on the same setup: -9 / -160 / -340 tok/s (-2.8 / -8.8 / -9.1%).
• Size: FFN 283M per layer (1.06x the dense 267M; Kan's shape 4.06B), 63M active per token (0.24x dense), 1.94B parameters in all (3.9 GB bf16, Kan's 20B / 39 GB).
• Accept length after epoch 1: 5-6% below dense v1; two more epochs to go. Epoch 1 (4,958 steps, train/acc 0.585) exported to RadixArk/qwen38-dspark-moe-16x1024/1ep and benchmarked with specforge benchmark (bf16 target, DSPARK block 7, triton, mamba 160, 32 max running requests, 100 / 200 prompts, 512 new tokens) on the GB300 box next to a running capture server, so only the accept lengths are valid: gsm8k c=1/8/32 4.87 / 4.89 / 4.90 (dense v1 5.16 / 5.15 / 5.17), mt-bench 3.38 / 3.65 / 3.73 (dense v1 3.63 / 3.89 / 4.02). Kan's 512 x 512 drafter gained 9-11% from epoch 1 to epoch 3; break-even here is +3.5% over dense (5.33 / 4.03 at c=8), worthwhile +6% (5.45 / 4.12). Epochs 2-3 continue from the epoch-1 checkpoint on two 8x H200 nodes (capture node + trainer node over Mooncake RDMA, recipe qwen3.8-27b-dspark-moe-16x1024-cont2ep-2node-h200.yaml); results per epoch will be added here.

Motivation

Kan's MoE drafter (#811 / #812, kan/moe-3-qwen38, 512 experts x 512 wide, top-10) gains ~1% accept length over dense v1 but loses 9-13% of throughput at c >= 8: a 7-token block x top-10 touches ~67 of the 512 experts, so every draft step streams ~1 GB of expert weights per layer against 0.53 GB for the dense FFN (#941 tables, qwen3.8-27b-dspark-moe-vs-dense-serving-results.md). With 16 experts a block touches at most 14, so the expert bytes stay at one dense FFN no matter how the router behaves, and the question becomes whether a few-wide-experts drafter keeps the MoE's accept-length gain. The speed gate below settles the throughput side before any GPU-hours are spent on training; the training run settles the accept-length side.

Modifications

• configs/qwen3.8-27b-dspark-moe-16x1024.json: Kan's qwen3.8-27b-dspark-moe-v2.json with only three keys changed: n_routed_experts 512 -> 16, moe_intermediate_size 512 -> 1024, num_experts_per_tok 10 -> 2. Shared expert (2048, sigmoid-gated), qwen3_5_moe preset (softmax, renormalised top-k), aux loss 0.001, per-sample router centering (momentum 0.99) and grouped_mm dispatch are unchanged.
• examples/configs/online/disaggregated/managed-local/qwen3.8-27b-dspark-moe-16x1024-3ep-gb300.yaml: 3 epochs from scratch (only the target embedding is loaded, as in Kan's runs) on one 4x GB300 node: 2 TP1 NVFP4 capture servers (GPUs 0-1) + 2 FSDP ranks (GPUs 2-3), batch 4 x accumulation 32 x 2 = 256 (4,958 steps per epoch, 14,874 in all), AdamW on the GPUs, lr 5e-4 constant after 4% warm-up, grad clip 1.0, FULL_SHARD, DSpark CE 0.1 / L1 0.9 / confidence 1.0, 512 anchors, decay gamma 4, flex attention, checkpoints every 2,479 steps (epoch ends included). Mooncake / feature-store watermarks are sized for the devbox's 924 GB container limit (comments in the file).
• scripts/moe_speed/: make_moe_draft_from_dense.py builds a random-expert MoE export from the official dense drafter (attention / projector / heads copied, experts random, so routing is uniform = worst case for expert bytes); speed_test_moe_draft.sh serves each drafter with SGLANG_SIMULATE_ACC_LEN and runs sglang.bench_serving at c = 1 / 8 / 32, so tok/s differences are step-time differences. README has the tables below.
• specforge/serving/sglang_models/, patches/sglang/moe_configs/, scripts/gates/check_dspark_moe_sglang_equivalence.py, tests/test_serving/: the serving package of feat(serving): SGLang draft classes for the Qwen3.8-27B DSpark MoE drafter (SGLang FusedMoE path) + tuned fused-MoE configs #941 (SGLang TopK + FusedMoE draft classes loaded via SGLANG_EXTERNAL_MODEL_PACKAGE), unchanged, so the exports of this config serve on stock SGLang >= 0.5.19.

End-to-end serving speed (GB300, accept length pinned to 5.0)

One GB300 (284 GB) per server, servers one after another on the same GPU with identical flags: DSPARK block 7, triton attention and MoE runner (default kernel config), mem-fraction 0.85, mamba cache 160, 32 max running requests, drafts bf16, 8 prompts per concurrency level x 512 in / 512 out, SGLang 0.5.20. Output tok/s, delta vs dense at the same concurrency. The 16-expert rows are random-expert exports; Kan's row is the trained RadixArk/qwen38-dspark-moe-3ep-cont-step9916.

bf16 target Qwen/Qwen3.8-27B:

drafter	FFN params / layer	active / token	c=1	c=8	c=32
official dense v1 (17408 wide)	267M	267M	326.7	1815.0	3752.4
MoE 16 x 1024 top-2, shared 2048	283M	63M	321.4 (-1.6%)	1780.3 (-1.9%)	3698.3 (-1.4%)
MoE 16 x 2048 top-2, shared 2048	535M	94M	319.9 (-2.1%)	1763.5 (-2.8%)	3660.7 (-2.4%)
Kan MoE 512 x 512 top-10, shared 2048	4.06B	110M	317.5 (-2.8%)	1655.1 (-8.8%)	3412.7 (-9.1%)

FP8 target Qwen/Qwen3.8-27B-FP8:

drafter	FFN params / layer	active / token	c=1	c=8	c=32
official dense v1 (17408 wide)	267M	267M	399.1	2143.0	4248.9
MoE 16 x 1024 top-2, shared 2048	283M	63M	391.4 (-1.9%)	2094.8 (-2.2%)	4206.0 (-1.0%)
MoE 16 x 2048 top-2, shared 2048	535M	94M	389.1 (-2.5%)	2070.1 (-3.4%)	4158.9 (-2.1%)
MoE 16 x 2048 top-4, shared 2048	535M	157M	386.2 (-3.2%)	2041.0 (-4.8%)	4122.5 (-3.0%)
Kan MoE 512 x 512 top-10, shared 2048	4.06B	110M	385.5 (-3.4%)	1922.8 (-10.3%)	3846.6 (-9.5%)

Per step (c=1, bf16 target, TPOT): dense 2.92 ms, 16 x 1024 2.97 ms (+0.05 ms), 16 x 2048 2.98 ms, Kan 3.00 ms; at c=32: 7.39 / 7.51 / 7.61 / 8.16 ms. Doubling the expert width or the top-k buys nothing at a pinned accept length and costs another 1-2%, hence 16 x 1024 top-2 for the training run.

Training run

• Epoch 1 (done 2026-10-10 15:14 UTC): one 4x GB300 node, 2 NVFP4 capture servers (RadixArk/Qwen3.8-27B-NVFP4-BF16-LMHead) + 2 FSDP ranks, recipe qwen3.8-27b-dspark-moe-16x1024-3ep-gb300.yaml, 36.3 s/step (capture-bound, 7.0 samples/s), 50.7 h; train/acc 0.577 at step 4958.
• Epochs 2-3 (from 2026-10-11 03:48 UTC): weights-only warm start from the epoch-1 checkpoint on two 8x H200 nodes (one 16-GPU devbox moe-16exp-1024dim, rank 0 = capture, rank 1 = trainer), recipe qwen3.8-27b-dspark-moe-16x1024-cont2ep-2node-h200.yaml: 8 capture servers on one node, 8 FSDP ranks on the other, features over Mooncake RDMA (26 GB/s per client measured, 277 GB/s raw); 17 s/step, 15 samples/s (still capture-bound), trainer peak 58 GB per rank, ~24 h per epoch. train/acc resumes at 0.575 at step 10 (0.585 at the end of epoch 1). The teacher changes from the NVFP4 target to the bf16 Qwen/Qwen3.8-27B because the NVFP4 checkpoint cannot run on Hopper; the draft stays bf16 and the data, order (seed 42), optimizer and loss settings are unchanged. The two sibling H200 runs (256-expert 5-layer, 512-expert 2-layer) use the bf16 teacher throughout. The GB300 run was stopped at step 5,780 of its epoch 2 once the two-node run had passed step 100 with sane metrics.
• Epochs 4-8 (running since 2026-10-13 00:49 UTC): epochs 2-3 ended 2026-10-12 23:20 UTC (train/acc 0.604 and 0.616 at their ends, exports in 2ep/ and 3ep/); qwen3.8-27b-dspark-moe-16x1024-cont5ep-from-3ep-2node-h200.yaml warm-starts from its step-9916 checkpoint for five more epochs (24,790 steps) on the same two nodes; every epoch end is exported and uploaded to RadixArk/qwen38-dspark-moe-16x1024/{4..8}ep/ (private). Target: 8 epochs in all.
• W&B (same project, one run per phase): epoch 1 https://wandb.ai/miles_training/specforge-qwen38-dspark-moe-16x1024/runs/qwen38-dspark-moe-16x1024-3ep-gb300 ; epochs 2-3 https://wandb.ai/miles_training/specforge-qwen38-dspark-moe-16x1024/runs/qwen38-dspark-moe-16x1024-cont2ep-2node-h200-v2 ; epochs 4-8 https://wandb.ai/miles_training/specforge-qwen38-dspark-moe-16x1024/runs/qwen38-dspark-moe-16x1024-cont5ep-from-3ep-2node-h200 (a first attempt on two separate devboxes, run ...-cont2ep-2node-h200, was stopped at step 940 to merge the nodes into one devbox and is tagged superseded; same checkpoint, same settings)
• Data: RadixArk/Qwen3.8-27B-Regen-Mixture-v1 (1,269,290 conversations, max_length 8192, qwen3.5 chat template); bf16 training, bf16 serving, no fp8.
• Exports (private, one repo, one folder per epoch, README with the per-epoch table): https://huggingface.co/RadixArk/qwen38-dspark-moe-16x1024 (1ep/ uploaded 2026-10-10; 2ep/ … 8ep/ follow as each epoch ends)

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.