Repository navigation
tile_allgather and tile_allreduce tests run out of teams with NVLS: the NVSHMEM_MAX_TEAMS they set ignores duplicate teams #139
Description
On a system with NVLS, test/device/tile/tile_allgather and tile_allreduce fail while creating their teams unless NVSHMEM_MAX_TEAMS is set by hand:
No more teams available (max = 64), try setting NVSHMEM_MAX_TEAMS environment variable
...
Unable to allocate enough duplicate teams. This will cause significant performance degradation. Please increase NVSHMEM_MAX_TEAMS. Exiting
When NVSHMEM_MAX_TEAMS is unset, both tests set it to num_teams + NVSHMEM_INTERNAL_TEAMS (tile_allreduce.cu:278-301, tile_allgather.cu:370-393), which is one slot per team. With NVLS available, every team split also runs nvshmemi_duplicate_team() (src/host/team/team_internal.cpp:950-985), which creates max(NVSHMEMI_REDUCESCATTER_CTA_COUNT_DEFAULT, NVSHMEMI_FCOLLECT_CTA_COUNT_DEFAULT / 2, NVSHMEMI_REDUCE_CTA_COUNT_DEFAULT) - 1 = 15 duplicate teams (nvshmem_common.cuh:117-119). The internal teams are duplicated at initialization as well (team_internal.cpp:1680-1689). Each team therefore needs about 16 slots.
B300, 2 PEs on one node over NVSwitch, devel 2dca770, built for sm_103:
| Setting | tile_allgather |
tile_allreduce |
|---|---|---|
| as is | exit 255, out of teams | exit 255, out of teams |
NVSHMEM_MAX_TEAMS=8192 |
pass | pass |
NVSHMEM_MAX_CTAS=1 (no duplicate teams) |
pass | pass |
NVSHMEM_DISABLE_NVLS=1 |
NVSHMEM Error :1 , Invalid value / argument, exit 1 |
same error on one PE; the other PE hangs until a 120 s timeout |
The other 115 test/device binaries passed in the same configuration.
Would you prefer the tests to size NVSHMEM_MAX_TEAMS for the duplicate teams (for example, multiply by the duplicate count when NVLS can be used), or should duplicate teams not count against NVSHMEM_MAX_TEAMS? I can send a PR for the test side if that is the preferred direction.
Activity