Repository navigation
Avoid mutating sampler-owned batches during distributed padding - #4506
Open
davrens101 wants to merge 1 commit into
Open
davrens101 wants to merge 1 commit into
davrens101 wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes #4505.
Keep padding local to
BatchSamplerShardrather than extending lists retained by the underlying sampler. Currently, preparing and iterating a loader with precomputed batches can change the original loader's future samples:[0, 1]becomes[0, 1, 0, 1]. The unsplit path similarly changes a seven-sample original stream into eight samples.Replace two in-place concatenations in the padding paths with non-mutating concatenations. Padded shard contents and ordering stay the same; full batches do not acquire unconditional copies. The docstring now records the sampler-ownership invariant.
This is separate from #4395's reported shard-length correction: no length calculation changes here.
Verification
f47439c1fb24721ea71ab98f9b512726b5a97828, and pass with this fix. They check actual prepared/original loader sample streams, both ranks and repeated iterations, for split and whole-batch padding.tests/test_data_loader.py: 42 passed, including existing stateful-loader cases.Accelerator.prepare, backward and an SGD step: both padding modes match an independent analytic global-batch gradient on both ranks. Weights are0.099999994vs expected0.1(split) and0.704999983vs expected0.705(whole-batch). The original loaders still yield exactly 2 and 7 samples. On untouched main, all four corresponding rank/mode runs fail the original-sample invariant.The Windows
torchrunelastic rendezvous initially hit this build's missing libuv support. The successful DDP smoke launches two real processes directly and initializes Gloo with an explicit non-libuv TCP rendezvous. No GPU execution is claimed.Before submitting
AI-assisted investigation, implementation and independent agent review. No claim of human review of this specific patch.
Upstream CI status
The posted head is
fabb3f3d13f3df391894f399f90237828bf1a8a4. Run Tests 42300201084, Quality Check 42300201286, Run Import Tests 42300201235, and Build PR Documentation 42300201998 reportaction_required, with zero jobs started. Upstream tests are not claimed passing; the local affected suite and both actual two-process training modes passed again after publication.