Repository navigation
Conversation
transport_common.h selects its x86 barriers with NVSHMEM_x86_64, but CMake defines NVSHMEM_X86_64 (src/CMakeLists.txt, src/modules/transport/CMakeLists.txt.in, src/device/CMakeLists.txt), and nothing defines the lowercase name. On x86_64, MEM_BARRIER(), STORE_BARRIER() and LOAD_BARRIER() therefore fall through to the generic compiler-only barrier, so the STORE_BARRIER() calls before the ibdevx, ibrc, staged, gpunetio and libfabric doorbell and flag writes never emit sfence. Use the macro that CMake defines, as the other x86 checks in the transports already do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Incremental review: one line in
src/modules/transport/common/transport_common.h, againstdevel(2dca770).Fixes #124.
Problem
transport_common.hselects its x86MEM_BARRIER()/STORE_BARRIER()/LOAD_BARRIER()with#if defined(NVSHMEM_x86_64). CMake defines onlyNVSHMEM_X86_64(src/CMakeLists.txt:533,src/modules/transport/CMakeLists.txt.in:88,src/device/CMakeLists.txt:131). On x86_64 the macros therefore fall back to the generic compiler-only barrier, and none of theSTORE_BARRIER()calls in ibdevx, ibrc, gpunetio, libfabric and staged emitssfence. That includesstaged_gdrcopy.cpp:194, whose comment says it is there to drain the write-combining buffers.Fix
Test
NVSHMEM_X86_64, the name every other x86 check in the transports already uses.Validation
git grep NVSHMEM_x86_64returns nothing after the change.nvshmemt_ibdevx_post_send(clang 21,-O2, with the define CMake passes):sfencebefore the doorbell-record store and before the BlueFlame store.-DNVSHMEM_AARCH64):dmb stin both cases.sfenceat the 8STORE_BARRIER()call sites, which are the only ones in the tree.MEM_BARRIER()andLOAD_BARRIER()have no callers, so nomfenceorlfenceappears. The two sites in CQ progress loops (ibdevx.cpp:1181,gpunetio.cpp:697) run once per consumed CQE.develplus this change on aarch64 (GH200) with-DCMAKE_CUDA_ARCHITECTURES=103and tests enabled. That build does not compile the x86 branch, so it only shows that nothing else broke.The commit carries a DCO sign-off.
🤖 Generated with Claude Code