Repository navigation
Bump transformers from 4.57.1 to 5.0.0rc3 in /.ci/docker - #1
Closed
dependabot[bot] wants to merge 1 commit into
Closed
dependabot[bot] wants to merge 1 commit into
dependabot[bot] wants to merge 1 commit into
Conversation
Bumps [transformers](https://github.com/huggingface/transformers) from 4.57.1 to 5.0.0rc3. - [Release notes](https://github.com/huggingface/transformers/releases) - [Commits](huggingface/transformers@v4.57.1...v5.0.0rc3) --- updated-dependencies: - dependency-name: transformers dependency-version: 5.0.0rc3 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
dependabot
Bot
added
dependencies
ryan-li-ryanllli-aws
pushed a commit
that referenced
this pull request
Jun 11, 2026
…nto canonicalize_graph_pass (pytorch#3488) ## Summary Two related graph-pass changes in `graph_trainer`: 1. **Add `remove_b2b_transpose_pass`** — collapses back-to-back `aten.t(aten.t(x))` transpose pairs. These appear in traced fwd+bwd graphs from `F.linear` when `simple_fsdp` redistributes weight tensors. Two consecutive `aten.t` form an involution (identical shape *and* strides), so removing them is **bitwise numerics-preserving**. The pass also handles chains (`t(t(t(x))) -> t(x)`) and keeps the inner transpose when it still feeds other consumers. 2. **Bundle the no-op cleanup passes into a single `canonicalize_graph_pass` entry** in `compile_time_passes`: - `remove_detach_pass` - `remove_identity_view_pass` - `remove_b2b_transpose_pass` - `remove_identity_slice_pass` - `normalize_view_ops_as_reshape` `normalize_view_ops_as_reshape` moves from `passes.py` into `remove_noop_passes.py` alongside the other graph-cleanup passes. The sub-passes stay public so each is unit-tested in isolation. ## Why The cleanup passes are all numerics-preserving local rewrites that ran as four separate pass-list entries. Folding them into one `canonicalize_graph_pass` keeps `compile_time_passes` readable and groups them as one logical step, while the new b2b-transpose rewrite removes redundant transpose pairs that `F.linear` + FSDP introduce. ## Behavior note `--compile.disable_passes` now toggles `canonicalize_graph_pass` as a whole rather than the individual sub-passes. No callsite (configs/scripts/README) disabled them individually, so nothing breaks. ## Verification Ran the llama3 debug model (FSDP=4, TP=2, `aot_fx_trace`, `--compile.debug_graph_passes`). `remove_b2b_transpose_pass` logs its removal count directly: ``` [rank0]:[titan] - root - INFO - Removed 129 back-to-back transpose node(s) from the graph ``` `canonicalize_graph_pass` is pass #1; its op-count diff confirms the same 129 nodes (`t.default: 215 -> 86`): ``` nodes: 2460 -> 1851 (-609) t.default: 215 -> 86 (-129) <- remove_b2b_transpose_pass detach.default: 40 -> 0 (-40) view.default: 634 -> 0 (-634) _unsafe_view: 43 -> 0 (-43) slice.Tensor: 40 -> 0 (-40) reshape.default: 0 -> 277 (+277) ``` 3 training steps, loss `8.11353 -> 7.80330 -> 7.09355`. ## Test plan ```bash pytest torchtitan/experiments/graph_trainer/tests/test_passes.py \ -k "TestRemoveB2BTransposePass or TestCanonicalizeGraphPass or \ TestNormalizeViewOpsAsReshape or TestRemoveDetachPass or \ TestRemoveIdentityViewPass or TestRemoveIdentitySlicePass" ``` Added `TestRemoveB2BTransposePass` (pair removal + numerics, lone transpose kept, inner transpose with another user kept, chain collapse, no-op on transpose-free graphs) and `TestCanonicalizeGraphPass` (end-to-end bundle). All pass; `pre-commit run --all-files` clean.
Author
|
Superseded by #3. |
dependabot
Bot
deleted the
dependabot/pip/dot-ci/docker/transformers-5.0.0rc3
branch
July 2, 2026 12:02
aws-arashj
pushed a commit
that referenced
this pull request
Oct 5, 2026
`TorchFTOptimizersContainer` [wraps itself](https://github.com/pytorch/torchtitan/blob/d398a8fb9bc8a33ca404217a846cff65feab92f0/torchtitan/experiments/torchft/optimizer.py#L45) with `torchft.Optimizer`. On accepted steps, the wrapper [calls back into the same container](https://github.com/meta-pytorch/torchft/blob/90d7f68961e7c0bcd4278f7ebba83b5cd876c099/torchft/optim.py#L53-L56), repeatedly triggering PyTorch's [pre/post hooks](https://github.com/pytorch/pytorch/blob/e3f5bf0b18585511e6cd7d7a574ebf82f465e5ae/torch/optim/optimizer.py#L508-L540). When the base method is already wrapped, [super().step()](https://github.com/pytorch/torchtitan/blob/d398a8fb9bc8a33ca404217a846cff65feab92f0/torchtitan/experiments/torchft/optimizer.py#L76) adds a third invocation, causing redundant blocking [MoE load-balancing collectives](https://github.com/pytorch/torchtitan/blob/d398a8fb9bc8a33ca404217a846cff65feab92f0/torchtitan/components/optimizer/optimizer.py#L488-L493). Both diagrams show an accepted step. All hooks shown belong to the same FT container. Before, with the base `step()` already wrapped: ```text Trainer.train_step() `-- TorchFTOptimizersContainer.step() |-- pre_hook #1 |-- OptimizerWrapper.step() | |-- manager.should_commit() -> True | `-- TorchFTOptimizersContainer.step() (same container) | |-- pre_hook #2 | |-- super().step() | | |-- pre_hook #3 | | |-- optimizer update loop (once) | | `-- post_hook #1 | `-- post_hook #2 `-- post_hook #3 ``` <img width="1136" height="896" alt="screenshot_before" src="https://github.com/user-attachments/assets/b843dac7-2e4d-420c-935b-a9d15a0ba930" /> Keep a single public `step()` hook boundary, move optimizer updates into `_step_optimizers()`, and call the FT Manager directly. This also removes the temporary dispatch flag, which could remain disabled after an exception. After: ```text Trainer.train_step() `-- OptimizersContainer.step() (inherited public entry) |-- pre_hook #1 |-- TorchFTOptimizersContainer._step_optimizers() | |-- manager.should_commit() -> True | `-- OptimizersContainer._step_optimizers() | `-- optimizer update loop (once) `-- post_hook #1 ``` Validation: - Two-rank Gloo regression with real MoE hooks and collectives: three load synchronizations per rank before the change, one afterward; Manager decisions are mocked. - CPU validation: **48 passed, 1 failed** [torchtitan-test.log](https://github.com/user-attachments/files/32060730/torchtitan-test.log) > The single failure is a pre-existing LR restoration issue: restored optimizers use `[0.05, 0.1]` instead of `[0.0875, 0.175]`, producing a different next update. The same failure reproduces before and after this change. --------- Co-authored-by: jojoinfra <jojobugfree@outlook.com>
ryan-li-ryanllli-aws
pushed a commit
that referenced
this pull request
Oct 13, 2026
…instead of replaying them (pytorch#5123) **TLDR:** Under FullAC, and under RegionAC without `ep_communication` saved, DeepEP computes wrong gradients for the routed experts, the router and the MoE input: at DeepSeek-V3 671B shapes the routed-expert weight grads are 1.41 relative L2 off. This PR saves DeepEP's dispatch and combine instead of replaying them in backward, as SelectiveAC already does. Gradients now match a run without AC, and FullAC gets 4.8% faster at 16k tokens per rank, for +1.1 GiB of saved activations per MoE layer at 16k. A new 2-GPU test fails on main and passes here. It runs in the H100 workflow, whose DeepEP install this PR also fixes. **Problem** 1. DeepEP's dispatch op returns the received rows plus a handle that records where each row came from. The backward of dispatch (a combine) and of combine (a dispatch) both route gradients with that handle. 2. FullAC replays the block's forward in backward, so the dispatch runs a second time (and the combine too). RegionAC does the same when `ep_communication` is not in `save_regions`. 3. DeepEP assigns receive slots with atomics (`deep_ep/include/deep_ep/impls/dispatch.cuh:345`), so the second dispatch can receive the rows in another order. 4. The grouped GEMMs' activations now come from the second dispatch, while their gradients are routed with the first dispatch's handle. Each token's gradient meets another token's activations and score. 5. At 4k tokens per rank, 1 MoE layer: routed-expert weight grads 1.41 relative L2 off vs no AC, router grads 1.20, input grads 0.93. The expert and router grads are uncorrelated with the right ones (cosine similarity 0.01-0.03). The loss barely shows it: DSv3 debugmodel's step-2 loss is 6.1263 with FullAC vs 6.1208 without. ```text forward: dispatch #1 receives A, B, C (handle #1 records this order) backward replay: dispatch #2 receives B, A, C -> grouped GEMM activations in order B, A, C backward grads: routed with handle #1 -> gradients in order A, B, C A's gradient meets B's activations ``` **Solution** 1. Never replay DeepEP's comm: save the forward's outputs. 2. `deepep::dispatch` and `deepep::combine` are registered as ordered effects. PyTorch's selective checkpointing saves an ordered-effect op unless the policy explicitly asks `MUST_RECOMPUTE` (`torch/utils/checkpoint.py:1618-1626`). That covers FullAC, SelectiveAC and its subclasses; FullAC's docstring already relies on it. 3. `dispatch_tokens` and `combine_tokens` always retain their remat regions, which covers RegionAC (torch_remat). Their `recompute` argument is gone, and `ep_communication` in `save_regions` no longer affects DeepEP. 4. Combine is saved too, because a replayed combine would look up a handle the forward combine already removed from `_handle_cache`. 5. Unchanged: no AC, SelectiveAC (it already saved these ops), and HybridEP, whose receive order is deterministic (FullAC grads bitwise equal to no AC). **Test** 1. `tests/unit_tests/gpu/test_deepep_activation_checkpoint.py` (2 GPUs) applies FullAC, SelectiveAC and RegionAC with no saved regions to a DeepEP MoE block. 2. It counts `ElasticBuffer.dispatch` and `combine` calls, and compares every gradient with the same block without AC. 3. On main, FullAC and RegionAC call each op 3 times instead of 2 ("AC replayed DeepEP: {'dispatch': 3, 'combine': 3}"), and the gradients are 0.95-1.31 relative L2 off. With this PR the counts match and the gradients are bitwise equal at these sizes. 4. It skips without `deep_ep`, so the unit-test workflows skip it. The H100 workflow runs it right after installing DeepEP, as the B200 workflow does for Dist-MoE. 5. That install has failed every night since 2026-09-30, so no DeepEP test ran. It cloned DeepEP's default branch with `--recursive`, and DeepEP's 2026-09-29 release added an SSH-only submodule there. This PR clones without `--recursive`; the pinned commit's submodules are still fetched after checkout. **Results** ```text DeepSeek-V3 671B shapes (D=7168, 256 experts, top-8), EP=8 over NVLink, 8x H100, bf16, eager + compiled SwiGLU region, main e5c5552 1 MoE layer, 4k tokens/rank: grad difference vs the same layer without AC, relative L2, worst of 8 ranks routed experts router gate MoE input FullAC, main 1.41 1.20 0.93 FullAC, this PR 2.7e-5 bitwise 6.3e-4 RegionAC without ep_communication, main 1.41 1.21 0.93 RegionAC without ep_communication, this PR 2.7e-5 bitwise 6.3e-4 noise: no AC vs no AC 2.7e-5 bitwise 6.3e-4 ``` The noise comes from DeepEP's receive order (expert weight grads) and the bf16 atomic adds in torchtitan's eager gather/scatter around the grouped GEMM (input grads). ```text 2 MoE layers fwd+bwd under FullAC, ms per iteration, main -> this PR, median of 3 processes [range] 4k tokens/rank 78.21 [77.75-79.04] -> 74.29 [72.99-74.89] -5.0% 16k tokens/rank 274.71 [274.53-275.41] -> 261.41 [260.99-261.41] -4.8% saved activations, 2 layers: 114 -> 677 MiB at 4k, 456 -> 2705 MiB at 16k SelectiveAC on main: 74.10 ms, 744 MiB at 4k; 256.97 ms, 2972 MiB at 16k ``` Training, DSv3 debugmodel, FSDP8+EP8, seed 0, step-2 loss / grad norm: ```text no AC 6.12075 / 4.4591 SelectiveAC 6.12076 / 4.4592 FullAC, main 6.12634 / 4.4703 FullAC, this PR 6.12075 / 4.4592 ``` NOTE: no recipe pairs DeepEP with FullAC or RegionAC today: the DeepEP recipes and the CI test use SelectiveAC. Nothing rejects the combination, and on main it trains with these wrong gradients without an error. NOTE: in DeepEP MoE layers, FullAC now saves almost as much as SelectiveAC (677 vs 744 MiB for 2 layers at 4k), at the same speed at 4k (74.3 vs 74.1 ms) and 1.7% slower at 16k (261.4 vs 257.0 ms). Attention and dense layers are unchanged: FullAC still recomputes them. NOTE: RegionAC configs without `ep_communication` now also keep DeepEP's comm outputs (+281 MiB per MoE layer at 4k). NOTE: needs torch nightly 20260926 or later: older torch registers the effect, but its checkpointing still replays the ops. torchtitan's FullAC already relies on the same mechanism. NOTE: not run on GPU: pipeline parallelism. On a CPU probe with a fake DeepEP buffer, FullAC + DeepEP under split input/weight backward (ZBV, DualPipeV, InterleavedZeroBubble) now raises "Trying to backward an extra time" instead of silently giving wrong gradients. SelectiveAC already fails there, so DeepEP with these schedules has no working activation checkpointing until the split backward supports saved effectful ops. You may ask: - Why `_lib._register_effectful_op`, a private method? `custom_op(...).register_effect` is the public form, but these ops are defined with `torch.library.Library("deepep", "DEF")`, and `register_effect` only calls `_lib._register_effectful_op` itself (`torch/_library/custom_ops.py:605-606`). - Why not add a `deepep` check to `_full_ac_policy`? It puts a backend name in a generic policy, and a SelectiveAC subclass that overrides `get_save_ops` without the deepep ops would replay them again (1.2 relative L2 on a CPU probe). The effect covers every policy. - Why not keep replaying with a deterministic DeepEP buffer? DeepEP then sorts every dispatch's rows: +7% at 4k and +4.5% at 16k per step, measured without AC, plus the replay itself. It would keep FullAC's memory. - Why not replay as a cached dispatch with the forward's handle, which gives the same order? The replayed call has no link to its forward call, and a per-layer slot picks the wrong handle once pipeline parallelism keeps several microbatches of a layer in flight. - Why not reject FullAC with DeepEP in config validation? It works with this fix. - Why is the test not in the multi-GPU unit-test workflow? That workflow doesn't install `deep_ep`, so the test would always skip there. **Test plan** - `pytest tests/unit_tests/gpu/test_deepep_activation_checkpoint.py` on 2x H100: passed all 8 runs on this PR; fails on main with the replay counts above. Skips without `deep_ep`. - Gradient check (script not in this PR): torchtitan's DSv3 671B config cut to 1 MoE layer, EP=8, same inputs and weights, AC applied to the MoE directly (`FullAC`, `SelectiveAC`, `RegionAC` with and without `ep_communication`); all gradients compared with a run without AC. Timing: the same harness with 2 MoE layers, 3 fresh processes per row, no other process on the GPUs. - Training: DSv3 debugmodel, FSDP8+EP8, 20 steps, no AC, SelectiveAC and FullAC. - CPU MoE and AC tests (`test_moe.py`, `test_expert_parallel.py`, `test_inference_moe.py`, `test_config_validation.py`, `test_transforms.py`, the DSv3, Qwen3.5 and GPT-OSS MoE tests) and `tests/unit_tests/gpu/test_activation_checkpoint.py`: 98 passed. Pyrefly: same 41 errors as main. - DeepEP install: cloning without `--recursive`, checking out the pin and updating submodules fetches only `third-party/fmt` (run locally; the full CI job needs `ciflow/h100.8`). Hardware: 8x H100 SXM, no RDMA NIC (`EP_DISABLE_GIN=1`), DeepEP v2.1.0 (`01dc3aa`). Torch 2.15.0.dev20260926 for the layer tests, 2.15.0.dev20261004 for training and the new test. --------- Co-authored-by: Felipe Mello <felipemello@fb.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bumps transformers from 4.57.1 to 5.0.0rc3.
Release notes
Sourced from transformers's releases.
... (truncated)
Commits
cb5079fv5.0.0rc3d1808f2[ci] Fixing some failing tests for important models (#43231)3d27645Add LightOnOCR model implementation (#41621)77146ccfix crash in when running FSDP2+TP (#43226)61317f5[CB] Ensure parallel decoding test passes using FA (#43277)1efe1a6Fix failingPegasusX,Mvp&LEDmodel integration tests (#43245)e8ae373[consistency] Ensure models are added to the_toctree.yml(#43264)c85be98[docs] tensorrt-llm (#43176)38022fd[style] Fix init isort and align makefile and CI (#43260)e977446Fix failingHiera,SwiftFormer&LEDModel integration tests (#43225)Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting
@dependabot rebase.Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)You can disable automated security fix PRs for this repo from the Security Alerts page.