[jiapingW]

Motivation

DDP accumulated microbatches were still synchronizing because no_sync covered only backward.

Modifications

Expose a backend accumulation context and enter it around both forward and backward. Keep FSDP backward deferral, and add optional positive training.ddp_bucket_cap_mb configuration. A two-rank test checks collective counts and optimizer-weight parity.

Stack and dependencies

Layer 2/10 split from #942. Previous layer: #957 (codex/pr942-01-streaming-reads). Review the diff against this PR's base branch; it contains only this layer. The full stack index is maintained in #942.

Actual prerequisites: No earlier feature is required; this layer uses existing main APIs.

The linear stack is the review/merge order. Model-specific optimizations are independently opt-in; the projection-checkpointing and attention layers expose shared capabilities. When merging a lower PR with squash, restack the remaining branches before merging them so already-reviewed commits do not reappear. The repository GPU test workflow only auto-runs for PRs targeting main; upper layers need that CI run when retargeted, while lint runs for all PRs.

Source: #942 at 211b87fc7d707e3a628cb95de5c73340af863c04, split onto main fdcc2eddf1b56b5b24a5a15d3abcb366adfeb5aa. The original branch is preserved.

Accuracy Test

• This layer at 00f7af287641: 40 tests run, zero failures/errors on H20 with Python 3.11.12 / PyTorch 2.13.0+cu130.
• Final stack at eaa14f102d6a: 405 integration tests run, zero failures/errors; 2 skipped on H20 with Python 3.11.12 / PyTorch 2.13.0+cu130. Each module runs in a fresh process to isolate process-global distributed state. This is targeted regression coverage, not a full-suite or downstream task-accuracy claim.
• Repository pre-commit checks passed for the final stack; each layer was formatted with the repository's pinned Black/isort versions.

Layer command (run in this branch's checkout, PYTHONPATH=.):

CUDA_VISIBLE_DEVICES=0 OMP_NUM_THREADS=4 python -m unittest \
  tests.test_training.test_ddp_accumulation \
  tests.test_config.test_schema -v

Qwen3 tokenizer regressions reuse the cached official tokenizer with HF_HUB_OFFLINE=1 and HF_HUB_CACHE pointing to that cache. No training job was launched.

Benchmark & Profiling

No new end-to-end throughput or downstream accuracy benchmark was run for this split. Model-side tests exercise numerical parity and gradients; historical measurements remain in #942 with their original scope and caveats.

Checklist

• Format changed code.
• Run the layer's regression tests and final-stack integration tests.
• Keep feature tests/documentation with their owning layer.
• Complete GitHub CI and any required downstream accuracy/performance validation before merge.

[maocheng23]

LGTM, can you please simplify the test file, now it's too long