Repository navigation
Conversation
On an InfiniBand port, ibgda and ibdevx write NVSHMEM_IB_TRAFFIC_CLASS into the QP primary address path even when the path is LID-routed and carries no GRH. Firmware rejects a nonzero tclass without a GRH, so INIT->RTR fails (errno 121) and the job aborts during connection setup as soon as the variable is set, which is what NVIDIA#45 reports. IBRC is not affected because verbs ignores grh.traffic_class when is_global is 0. Program tclass only where the GRH is enabled, next to the other GRH fields, which matches the IBRC behaviour. RoCE paths are unchanged. Fixes NVIDIA#45. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #45.
Problem
On an InfiniBand port, IBGDA (
ibgda_rc_init2rtr) and IBDevX (ep_connect) writeNVSHMEM_IB_TRAFFIC_CLASSinto the QP primary address path unconditionally, including LID-routed paths that carry no GRH (grh = 0). The traffic class is a GRH field. With any nonzero value, INIT->RTR is rejected and initialization aborts:IBGDA QP modify INIT->RTR failed: type RC ... link_layer InfiniBand ... traffic_class 106, thenibgda_rc_init2rtr failed on RC #0IBDEVX QP modify INIT->RTR failed ... link_layer InfiniBand ... traffic_class 106 ret 121 errno 121IBRC is not affected: it sets
ah_attr.grh.traffic_classthrough verbs, which ignores the GRH fields whenis_globalis 0.Fix
Set
primary_address_path.tclassonly inside the branch that enables the GRH, next to the other GRH fields, in both transports. This matches the IBRC behaviour. The Ethernet/RoCE branches are not touched.The SIGSEGV that follows the failed connection on
develis a separate error-path problem: under IBGDA it is in the proxy thread'sprogress_transports, and under IBDevX it is innvshmemt_ibdevx_finalize, both reached fromnvshmemi_setup_connectionscleanup. This PR does not change that path.Tested
Two GH200 nodes, one ConnectX-7 NDR port each (InfiniBand link layer, LID-routed, no GRH required), aarch64, CUDA 13.2, 2 PEs (1 per node),
shmem_put_signal_ping_pong_latency(8 B–4 KB):IB_TRAFFIC_CLASSIB_FORCE_GRHdevel2dca770The value in parentheses is the 8 B
shmem_put_signal_ping_pong_latencyround trip (200 iterations). It only shows that the connection carries traffic; there is no performance claim.shmem_put_latencywithIB_TRAFFIC_CLASS=106also passes on all three transports with this PR. WithIB_FORCE_GRH=1the traffic class is still programmed, now inside the GRH branch.Not tested
grh_required), other than forcing it withNVSHMEM_IB_FORCE_GRH=1.Related: #25 changes where the IBGDA traffic class comes from on RoCE. It edits the same functions but not the InfiniBand branch changed here; if it lands first, the moved line should use its traffic-class value as well.
The commit carries a DCO sign-off.