Skip to content

Repository navigation

Bump pytest from 7.3.2 to 9.0.3 in /.ci/docker - #2

Open
dependabot[bot] wants to merge 1 commit into
mainfrom
dependabot/pip/dot-ci/docker/pytest-9.0.3
Open

Bump pytest from 7.3.2 to 9.0.3 in /.ci/docker#2
dependabot[bot] wants to merge 1 commit into
mainfrom
dependabot/pip/dot-ci/docker/pytest-9.0.3

Conversation

@dependabot

@dependabot dependabot Bot commented on behalf of github Apr 13, 2026

Copy link
Copy Markdown

Bumps pytest from 7.3.2 to 9.0.3.

Release notes

Sourced from pytest's releases.

9.0.3

pytest 9.0.3 (2026-04-07)

Bug fixes

  • #12444: Fixed pytest.approx which now correctly takes into account ~collections.abc.Mapping keys order to compare them.

  • #13634: Blocking a conftest.py file using the -p no: option is now explicitly disallowed.

    Previously this resulted in an internal assertion failure during plugin loading.

    Pytest now raises a clear UsageError explaining that conftest files are not plugins and cannot be disabled via -p.

  • #13734: Fixed crash when a test raises an exceptiongroup with __tracebackhide__ = True.

  • #14195: Fixed an issue where non-string messages passed to unittest.TestCase.subTest() were not printed.

  • #14343: Fixed use of insecure temporary directory (CVE-2025-71176).

Improved documentation

  • #13388: Clarified documentation for -p vs PYTEST_PLUGINS plugin loading and fixed an incorrect -p example.
  • #13731: Clarified that capture fixtures (e.g. capsys and capfd) take precedence over the -s / --capture=no command-line options in Accessing captured output from a test function <accessing-captured-output>.
  • #14088: Clarified that the default pytest_collection hook sets session.items before it calls pytest_collection_finish, not after.
  • #14255: TOML integer log levels must be quoted: Updating reference documentation.

Contributor-facing changes

  • #12689: The test reports are now published to Codecov from GitHub Actions. The test statistics is visible on the web interface.

    -- by aleguy02

9.0.2

pytest 9.0.2 (2025-12-06)

Bug fixes

  • #13896: The terminal progress feature added in pytest 9.0.0 has been disabled by default, except on Windows, due to compatibility issues with some terminal emulators.

    You may enable it again by passing -p terminalprogress. We may enable it by default again once compatibility improves in the future.

    Additionally, when the environment variable TERM is dumb, the escape codes are no longer emitted, even if the plugin is enabled.

  • #13904: Fixed the TOML type of the tmp_path_retention_count settings in the API reference from number to string.

  • #13946: The private config.inicfg attribute was changed in a breaking manner in pytest 9.0.0. Due to its usage in the ecosystem, it is now restored to working order using a compatibility shim. It will be deprecated in pytest 9.1 and removed in pytest 10.

... (truncated)

Commits

Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

  • @dependabot rebase will rebase this PR
  • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
  • @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
  • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    You can disable automated security fix PRs for this repo from the Security Alerts page.

@dependabot
Bump pytest from 7.3.2 to 9.0.3 in /.ci/docker
3f3f0ee 
Bumps [pytest](https://github.com/pytest-dev/pytest) from 7.3.2 to 9.0.3.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](pytest-dev/pytest@7.3.2...9.0.3)

---
updated-dependencies:
- dependency-name: pytest
  dependency-version: 9.0.3
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
@dependabot dependabot Bot added dependencies Pull requests that update a dependency file python Pull requests that update python code labels Apr 13, 2026
aws-arashj pushed a commit that referenced this pull request Oct 5, 2026
@jojoinfra jojoinfra
Fix recursive TorchFT optimizer dispatch (pytorch#4578)
59bbedf 
`TorchFTOptimizersContainer` [wraps
itself](https://github.com/pytorch/torchtitan/blob/d398a8fb9bc8a33ca404217a846cff65feab92f0/torchtitan/experiments/torchft/optimizer.py#L45)
with `torchft.Optimizer`. On accepted steps, the wrapper [calls back
into the same
container](https://github.com/meta-pytorch/torchft/blob/90d7f68961e7c0bcd4278f7ebba83b5cd876c099/torchft/optim.py#L53-L56),
repeatedly triggering PyTorch's [pre/post
hooks](https://github.com/pytorch/pytorch/blob/e3f5bf0b18585511e6cd7d7a574ebf82f465e5ae/torch/optim/optimizer.py#L508-L540).
When the base method is already wrapped,
[super().step()](https://github.com/pytorch/torchtitan/blob/d398a8fb9bc8a33ca404217a846cff65feab92f0/torchtitan/experiments/torchft/optimizer.py#L76)
adds a third invocation, causing redundant blocking [MoE load-balancing
collectives](https://github.com/pytorch/torchtitan/blob/d398a8fb9bc8a33ca404217a846cff65feab92f0/torchtitan/components/optimizer/optimizer.py#L488-L493).

Both diagrams show an accepted step. All hooks shown belong to the same
FT container.

Before, with the base `step()` already wrapped:

```text
Trainer.train_step()
`-- TorchFTOptimizersContainer.step()
    |-- pre_hook                              #1
    |-- OptimizerWrapper.step()
    |   |-- manager.should_commit() -> True
    |   `-- TorchFTOptimizersContainer.step()  (same container)
    |       |-- pre_hook                      #2
    |       |-- super().step()
    |       |   |-- pre_hook                  #3
    |       |   |-- optimizer update loop     (once)
    |       |   `-- post_hook                 #1
    |       `-- post_hook                     #2
    `-- post_hook                             #3
```
<img width="1136" height="896" alt="screenshot_before"
src="https://github.com/user-attachments/assets/b843dac7-2e4d-420c-935b-a9d15a0ba930"
/>


Keep a single public `step()` hook boundary, move optimizer updates into
`_step_optimizers()`, and call the FT Manager directly. This also
removes the temporary dispatch flag, which could remain disabled after
an exception.

After:

```text
Trainer.train_step()
`-- OptimizersContainer.step()                (inherited public entry)
    |-- pre_hook                              #1
    |-- TorchFTOptimizersContainer._step_optimizers()
    |   |-- manager.should_commit() -> True
    |   `-- OptimizersContainer._step_optimizers()
    |       `-- optimizer update loop         (once)
    `-- post_hook                             #1
```

Validation:

- Two-rank Gloo regression with real MoE hooks and collectives: three
load synchronizations per rank before the change, one afterward; Manager
decisions are mocked.
- CPU validation: **48 passed, 1 failed**

[torchtitan-test.log](https://github.com/user-attachments/files/32060730/torchtitan-test.log)
> The single failure is a pre-existing LR restoration issue: restored
optimizers use `[0.05, 0.1]` instead of `[0.0875, 0.175]`, producing a
different next update. The same failure reproduces before and after this
change.

---------

Co-authored-by: jojoinfra <jojobugfree@outlook.com>
ryan-li-ryanllli-aws pushed a commit that referenced this pull request Oct 13, 2026
@felipemello1 Felipe Mello
[deepep] Fix wrong gradients under FullAC: save dispatch and combine …
0167526 
…instead of replaying them (pytorch#5123)

**TLDR:** Under FullAC, and under RegionAC without `ep_communication`
saved, DeepEP computes wrong gradients for the routed experts, the
router and the MoE input: at DeepSeek-V3 671B shapes the routed-expert
weight grads are 1.41 relative L2 off. This PR saves DeepEP's dispatch
and combine instead of replaying them in backward, as SelectiveAC
already does. Gradients now match a run without AC, and FullAC gets 4.8%
faster at 16k tokens per rank, for +1.1 GiB of saved activations per MoE
layer at 16k. A new 2-GPU test fails on main and passes here. It runs in
the H100 workflow, whose DeepEP install this PR also fixes.

**Problem**
1. DeepEP's dispatch op returns the received rows plus a handle that
records where each row came from. The backward of dispatch (a combine)
and of combine (a dispatch) both route gradients with that handle.
2. FullAC replays the block's forward in backward, so the dispatch runs
a second time (and the combine too). RegionAC does the same when
`ep_communication` is not in `save_regions`.
3. DeepEP assigns receive slots with atomics
(`deep_ep/include/deep_ep/impls/dispatch.cuh:345`), so the second
dispatch can receive the rows in another order.
4. The grouped GEMMs' activations now come from the second dispatch,
while their gradients are routed with the first dispatch's handle. Each
token's gradient meets another token's activations and score.
5. At 4k tokens per rank, 1 MoE layer: routed-expert weight grads 1.41
relative L2 off vs no AC, router grads 1.20, input grads 0.93. The
expert and router grads are uncorrelated with the right ones (cosine
similarity 0.01-0.03). The loss barely shows it: DSv3 debugmodel's
step-2 loss is 6.1263 with FullAC vs 6.1208 without.

```text
forward:          dispatch #1 receives A, B, C   (handle #1 records this order)
backward replay:  dispatch #2 receives B, A, C   -> grouped GEMM activations in order B, A, C
backward grads:   routed with handle #1          -> gradients in order A, B, C
                  A's gradient meets B's activations
```

**Solution**
1. Never replay DeepEP's comm: save the forward's outputs.
2. `deepep::dispatch` and `deepep::combine` are registered as ordered
effects. PyTorch's selective checkpointing saves an ordered-effect op
unless the policy explicitly asks `MUST_RECOMPUTE`
(`torch/utils/checkpoint.py:1618-1626`). That covers FullAC, SelectiveAC
and its subclasses; FullAC's docstring already relies on it.
3. `dispatch_tokens` and `combine_tokens` always retain their remat
regions, which covers RegionAC (torch_remat). Their `recompute` argument
is gone, and `ep_communication` in `save_regions` no longer affects
DeepEP.
4. Combine is saved too, because a replayed combine would look up a
handle the forward combine already removed from `_handle_cache`.
5. Unchanged: no AC, SelectiveAC (it already saved these ops), and
HybridEP, whose receive order is deterministic (FullAC grads bitwise
equal to no AC).

**Test**
1. `tests/unit_tests/gpu/test_deepep_activation_checkpoint.py` (2 GPUs)
applies FullAC, SelectiveAC and RegionAC with no saved regions to a
DeepEP MoE block.
2. It counts `ElasticBuffer.dispatch` and `combine` calls, and compares
every gradient with the same block without AC.
3. On main, FullAC and RegionAC call each op 3 times instead of 2 ("AC
replayed DeepEP: {'dispatch': 3, 'combine': 3}"), and the gradients are
0.95-1.31 relative L2 off. With this PR the counts match and the
gradients are bitwise equal at these sizes.
4. It skips without `deep_ep`, so the unit-test workflows skip it. The
H100 workflow runs it right after installing DeepEP, as the B200
workflow does for Dist-MoE.
5. That install has failed every night since 2026-09-30, so no DeepEP
test ran. It cloned DeepEP's default branch with `--recursive`, and
DeepEP's 2026-09-29 release added an SSH-only submodule there. This PR
clones without `--recursive`; the pinned commit's submodules are still
fetched after checkout.

**Results**
```text
DeepSeek-V3 671B shapes (D=7168, 256 experts, top-8), EP=8 over NVLink, 8x H100, bf16, eager + compiled SwiGLU region, main e5c5552
1 MoE layer, 4k tokens/rank: grad difference vs the same layer without AC, relative L2, worst of 8 ranks
                                               routed experts   router gate   MoE input
  FullAC, main                                 1.41             1.20          0.93
  FullAC, this PR                              2.7e-5           bitwise       6.3e-4
  RegionAC without ep_communication, main      1.41             1.21          0.93
  RegionAC without ep_communication, this PR   2.7e-5           bitwise       6.3e-4
  noise: no AC vs no AC                        2.7e-5           bitwise       6.3e-4
```
The noise comes from DeepEP's receive order (expert weight grads) and
the bf16 atomic adds in torchtitan's eager gather/scatter around the
grouped GEMM (input grads).

```text
2 MoE layers fwd+bwd under FullAC, ms per iteration, main -> this PR, median of 3 processes [range]
  4k tokens/rank     78.21 [77.75-79.04]   ->  74.29 [72.99-74.89]    -5.0%
  16k tokens/rank   274.71 [274.53-275.41] -> 261.41 [260.99-261.41]  -4.8%
saved activations, 2 layers:  114 -> 677 MiB at 4k,  456 -> 2705 MiB at 16k
SelectiveAC on main:          74.10 ms, 744 MiB at 4k;  256.97 ms, 2972 MiB at 16k
```

Training, DSv3 debugmodel, FSDP8+EP8, seed 0, step-2 loss / grad norm:
```text
no AC                6.12075 / 4.4591
SelectiveAC          6.12076 / 4.4592
FullAC, main         6.12634 / 4.4703
FullAC, this PR      6.12075 / 4.4592
```

NOTE: no recipe pairs DeepEP with FullAC or RegionAC today: the DeepEP
recipes and the CI test use SelectiveAC. Nothing rejects the
combination, and on main it trains with these wrong gradients without an
error.

NOTE: in DeepEP MoE layers, FullAC now saves almost as much as
SelectiveAC (677 vs 744 MiB for 2 layers at 4k), at the same speed at 4k
(74.3 vs 74.1 ms) and 1.7% slower at 16k (261.4 vs 257.0 ms). Attention
and dense layers are unchanged: FullAC still recomputes them.

NOTE: RegionAC configs without `ep_communication` now also keep DeepEP's
comm outputs (+281 MiB per MoE layer at 4k).

NOTE: needs torch nightly 20260926 or later: older torch registers the
effect, but its checkpointing still replays the ops. torchtitan's FullAC
already relies on the same mechanism.

NOTE: not run on GPU: pipeline parallelism. On a CPU probe with a fake
DeepEP buffer, FullAC + DeepEP under split input/weight backward (ZBV,
DualPipeV, InterleavedZeroBubble) now raises "Trying to backward an
extra time" instead of silently giving wrong gradients. SelectiveAC
already fails there, so DeepEP with these schedules has no working
activation checkpointing until the split backward supports saved
effectful ops.

You may ask:
- Why `_lib._register_effectful_op`, a private method?
`custom_op(...).register_effect` is the public form, but these ops are
defined with `torch.library.Library("deepep", "DEF")`, and
`register_effect` only calls `_lib._register_effectful_op` itself
(`torch/_library/custom_ops.py:605-606`).
- Why not add a `deepep` check to `_full_ac_policy`? It puts a backend
name in a generic policy, and a SelectiveAC subclass that overrides
`get_save_ops` without the deepep ops would replay them again (1.2
relative L2 on a CPU probe). The effect covers every policy.
- Why not keep replaying with a deterministic DeepEP buffer? DeepEP then
sorts every dispatch's rows: +7% at 4k and +4.5% at 16k per step,
measured without AC, plus the replay itself. It would keep FullAC's
memory.
- Why not replay as a cached dispatch with the forward's handle, which
gives the same order? The replayed call has no link to its forward call,
and a per-layer slot picks the wrong handle once pipeline parallelism
keeps several microbatches of a layer in flight.
- Why not reject FullAC with DeepEP in config validation? It works with
this fix.
- Why is the test not in the multi-GPU unit-test workflow? That workflow
doesn't install `deep_ep`, so the test would always skip there.

**Test plan**
- `pytest tests/unit_tests/gpu/test_deepep_activation_checkpoint.py` on
2x H100: passed all 8 runs on this PR; fails on main with the replay
counts above. Skips without `deep_ep`.
- Gradient check (script not in this PR): torchtitan's DSv3 671B config
cut to 1 MoE layer, EP=8, same inputs and weights, AC applied to the MoE
directly (`FullAC`, `SelectiveAC`, `RegionAC` with and without
`ep_communication`); all gradients compared with a run without AC.
Timing: the same harness with 2 MoE layers, 3 fresh processes per row,
no other process on the GPUs.
- Training: DSv3 debugmodel, FSDP8+EP8, 20 steps, no AC, SelectiveAC and
FullAC.
- CPU MoE and AC tests (`test_moe.py`, `test_expert_parallel.py`,
`test_inference_moe.py`, `test_config_validation.py`,
`test_transforms.py`, the DSv3, Qwen3.5 and GPT-OSS MoE tests) and
`tests/unit_tests/gpu/test_activation_checkpoint.py`: 98 passed.
Pyrefly: same 41 errors as main.
- DeepEP install: cloning without `--recursive`, checking out the pin
and updating submodules fetches only `third-party/fmt` (run locally; the
full CI job needs `ciflow/h100.8`).

Hardware: 8x H100 SXM, no RDMA NIC (`EP_DISABLE_GIN=1`), DeepEP v2.1.0
(`01dc3aa`). Torch 2.15.0.dev20260926 for the layer tests,
2.15.0.dev20261004 for training and the new test.

---------

Co-authored-by: Felipe Mello <felipemello@fb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

dependencies Pull requests that update a dependency file python Pull requests that update python code

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

0 participants