Repository navigation
[Issue]: IB env will fail if export NVSHMEM_IB_TRAFFIC_CLASS #45
Description
How is this issue impacting you?
Application crash
Share Your Debug Logs
15:35:48.149 /opt/nvshmem/nvshmem_src/src/modules/transport/ibgda/ibgda.cpp:1393: non-zero status: 121 Error in mlx5dv_devx_obj_modify for INIT2RTR_QP with syndrome 764216
15:35:48.149
15:35:48.149 /opt/nvshmem/nvshmem_src/src/modules/transport/ibgda/ibgda.cpp:3005: non-zero status: 7 ibgda_rc_init2rtr failed on RC #0.
15:35:48.149 /opt/nvshmem/nvshmem_src/src/host/transport/transport.cpp:420: non-zero status: 7 connect EPS failed
15:35:48.149
15:35:48.149 /opt/nvshmem/nvshmem_src/src/host/init/init.cu:1044: non-zero status: 7 nvshmem setup connections failed
Steps to Reproduce the Issue
No response
NVSHMEM Version
3.3.9 + cuda12.4
Your platform details
ibstatus
Infiniband device 'mlx5_0' port 1 status:
default gid: fe80:0000:0000:0000:a088:c203:0002:7710
base lid: 0x7db
sm lid: 0x66
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 400 Gb/sec (4X NDR)
link_layer: InfiniBand
Infiniband device 'mlx5_1' port 1 status:
default gid: fe80:0000:0000:0000:a088:c203:0002:703c
base lid: 0x7de
sm lid: 0x66
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 400 Gb/sec (4X NDR)
link_layer: InfiniBand
Error Message & Behavior
When I configure NVSHMEM_IB_TRAFFIC_CLASS in the InfiniBand (link_layer) environment, nvshmem initialization will fail and exit, seemingly due to a call to a related RoCE API?
unset it will be good
Activity
[-][Issue]:[/-][+][Issue]: IB env will fail if export NVSHMEM_IB_TRAFFIC_CLASS[/+]benhg commentedon Dec 13, 2025
what did you set
NVSHMEM_IB_TRAFFIC_CLASSto? If you put other traffic in that class, does your system work?moon657 commentedon Dec 13, 2025
thank you for your reply. I set NVSHMEM_IB_TRAFFIC_CLASS=100 it failed, when I keep it default 0, my system work
NVSHMEM_IB_TRAFFIC_CLASS is left over from my code in the roce environment. Normally, it has no impact in the infiniband environment?
benhg commentedon Dec 24, 2025
What I mean to ask is if you put non-NVSHMEM related traffic in class 100, do you see any other errors?
Qizhi697 commentedon Jan 6, 2026
Why is it sufficient for IBRC to only set the traffic_class, while IBGDA additionally requires setting eth_prio and dscp? Could you explain the rationale behind this design and the roles of these fields?