SanitizerError
AddressSanitizer: stack-buffer-overflow (/usr/lib/libcuda/libnvidia-ml.so.1+0x3b306)
==1292==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7b0fac6a3454 at pc 0x558b5a4009a8 bp 0x7b0fad884930 sp 0x7b0fad8840e0
WRITE of size 17 at 0x7b0fac6a3454 thread T33 (tf_CreateNcclCo)
#0 0x558b5a4009a7 in vsnprintf llvm/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_common_interceptors.inc:1799:1
#1 0x558b5a402096 in snprintf llvm/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_common_interceptors.inc:1872:1
#2 0x7b0fde03b306 (/usr/lib/libcuda/libnvidia-ml.so.1+0x3b306)
#3 0x7b0fde045547 in nvmlDeviceGetPciInfo_v3 (/usr/lib/libcuda/libnvidia-ml.so.1+0x45547)
#4 0x558b6aaa58b0 in ncclNvmlDeviceGetPciInfo(nvmlDevice_st*, nvmlPciInfo_st*) nccl/src/misc/nvmlwrap.cc:332
#5 0x558b6aa6068f in ncclTopoSetAttrFromNvml(ncclXmlNode*, nvmlDevice_st*, char const*) nccl/src/graph/xml.cc:451
#6 0x558b6aa61c34 in ncclTopoGetXmlFromSys(ncclXmlNode*, ncclXml*) nccl/src/graph/xml.cc:639
#7 0x558b6aa65854 in ncclTopoFillGpu(ncclXml*, char const*, ncclXmlNode**) nccl/src/graph/xml.cc:1065
#8 0x558b6aa528dc in ncclTopoGetSystem(ncclComm*, ncclTopoSystem**, char const*) nccl/src/graph/topo.cc:1813
#9 0x558b6aa83fea in initTransportsRank(ncclComm*, ncclComm*, unsigned long*) nccl/src/init.cc:1141
#10 0x558b6aa7a1e9 in ncclCommInitRankFunc(ncclAsyncJob*) nccl/src/init.cc:1921
#11 0x558b6aa67e1c in ncclAsyncJobMain(void*) nccl/src/group.cc:77
#12 0x558b6aa6e531 in asyncJobLaunch(ncclIntruQueue<ncclAsyncJob, &ncclAsyncJob::next>*, bool volatile*) nccl/src/group.cc:479
#13 0x558b6aa6b19f in groupLaunch(ncclAsyncJob*, ncclSimInfo_v22200*) nccl/src/group.cc:630
#14 0x558b6aa690a7 in ncclGroupEndInternal(ncclSimInfo_v22200*) nccl/src/group.cc:866
#15 0x558b6aa727ca in ncclCommInitRankConfig nccl/src/init.cc:2679
How is this issue impacting you?
Data corruption
Share Your Debug Logs
No response
Steps to Reproduce the Issue
ncclNvmlDeviceGetPciInfo.NCCL Version
2.30.7-1
Your platform details
No response
Error Message & Behavior