Repository navigation
Conversation
tile_allgather and tile_allreduce create one team per thread, warp, warpgroup or block and set NVSHMEM_MAX_TEAMS to that count plus the internal teams. With NVLS, nvshmemi_duplicate_team() also creates duplicate teams for the collective CTAs whenever a team is split, so each team takes max(NVSHMEMI_REDUCESCATTER_CTA_COUNT_DEFAULT, NVSHMEMI_FCOLLECT_CTA_COUNT_DEFAULT / 2, NVSHMEMI_REDUCE_CTA_COUNT_DEFAULT) slots, or NVSHMEM_MAX_CTAS when it is set. Both tests then run out of teams during setup on NVLS systems. Multiply the requested count by the slots per team. Fixes NVIDIA#139. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #139.
tile_allgatherandtile_allreducesetNVSHMEM_MAX_TEAMSto the number of teams they create plus the internal teams, one slot each. With NVLS,nvshmemi_duplicate_team()also creates duplicate teams for the collective CTAs whenever a team is split:max(NVSHMEMI_REDUCESCATTER_CTA_COUNT_DEFAULT, NVSHMEMI_FCOLLECT_CTA_COUNT_DEFAULT / 2, NVSHMEMI_REDUCE_CTA_COUNT_DEFAULT)slots per team, orNVSHMEM_MAX_CTASwhen it is set. Both tests therefore run out of teams during setup on NVLS systems.This multiplies the requested count by the slots per team, using the same constants as the library (the tests already include
nvshmem_common.cuh). A user-providedNVSHMEM_MAX_TEAMSis still respected.test/host/team/shmem_team_split_2d.cppavoids the same problem by settingNVSHMEM_MAX_CTAS=2; I did not do that here, so that the tile collectives keep running with the default CTA count.Tested
B300,
develeaf955f, built forsm_103, PEs on one node over NVSwitch, default environment unless noted:develtile_allgathertile_allreducetile_allgather,NVSHMEM_MAX_CTAS=1tile_allreduce,NVSHMEM_MAX_CTAS=1tile_allgathertile_allreduceNot changed here: with
NVSHMEM_DISABLE_NVLS=1, the NVLS tile algorithms these tests use returnNVSHMEMX_ERROR_INVALID_VALUE, andtile_allreducecan then leave one PE waiting after the other returns (see #139).The commit carries a DCO sign-off.