Late Training Replay + EMA + GPTQ-lite (val_bpb=1.1236, 2-seed, no TTT on eval)

[newjordan]

Summary

• val_bpb: 1.1236 (sliding window stride=64, 2-seed mean) | 15.59 MB (mean) | 8xH100 SXM
• Builds on PR Record: 11L EMA + GPTQ-lite + warmdown3500 + QAT@0.15 (val_bpb=1.1233) #414 baseline (1.1233) with one addition:
  • Late Training Replay: 2-epoch replay of last 100 training batches at 10% LR before EMA finalization

Important: No TTT on validation data

This submission does NOT perform test-time training on evaluation/validation tokens. The "late replay" step replays training data only (buffered from the training loop) as a late-stage fine-tuning step before weight averaging. No eval tokens are seen before scoring. Fully compliant with issue #402.

Results (3 seeds, 8xH100 SXM)

Seed	Steps	Sliding BPB (s64)	Artifact
1337	6991	1.1232	15.68 MB
42	6994	1.1240	16.37 MB*
2024	6987	1.1239	15.50 MB

Mean (1337+2024): 1.1236 | Std: 0.0004

*Seed 42 artifact over size limit due to compression variance; BPB validates the approach.

How Late Training Replay Works

During the warmdown phase (last ~15% of training), the 100 most recent training batches are buffered. After training ends but before EMA weight averaging, these buffered training batches are replayed for 2 epochs at 10% of the base learning rate. This is strictly a training-data-only operation — equivalent to extending training by ~200 steps with a tiny learning rate. The model never sees validation tokens before scoring.

Test plan

• 3 seeds run on 8xH100 SXM, 600s each
• Seeds 1337, 2024 under 16MB (15.68 MB, 15.50 MB)
• Post-quant int6 roundtrip verified
• Sliding window eval (stride=64) consistent across seeds (std=0.0004)
• train_gpt.py under 1500 lines (1443)
• No TTT on validation data — late replay uses training data only (per issue Invalid submissions due to information leakage during TTT #402)

🤖 Generated with Claude Code

[newjordan]

Updated per issue #402: renamed 'TTT Burst' to 'Late Training Replay' to clarify this does NOT perform test-time training on validation/eval tokens. The replay step buffers and replays training data only (last 100 training batches) as a late fine-tuning step before EMA finalization. No eval tokens are seen before scoring. Also closed our PR #390 (Sponge Bath) which did use TTT on eval tokens.