[BUG] conda-cpp-tests on gb300: routing PDP test hits illegal memory access, set_shmem_of_kernel fails after it #2212

Activity

1. ramakrishnap-nv

   added

   bugSomething isn't workingSomething isn't working

   on Oct 26, 2026
2. 0z5a commented on Nov 3, 2026

   

   0z5a

   on Nov 3, 2026

   More actions

   On an x86_64 B300 (sm_103, same sm_100f device code as GB300) with 26.12.00a58, cost_boundary_initialization.* and set_shmem_of_kernel.* passed 200/200, and compute-sanitizer memcheck reports no memory errors. So the GPU alone does not reproduce it.

   I think the cause is in the test itself. In test_cost_boundaries_reset, only the PDP branch does

   if constexpr (is_pdp) {
     const int pickup   = 0;
     const int delivery = 1;
     raft::copy(d_pickups.data(), &pickup, 1, stream);
     raft::copy(d_deliveries.data(), &delivery, 1, stream);
     ...
   }  // pickup and delivery end here, nothing has synced the stream

   raft::copy is a cudaMemcpyAsync from pageable memory. Where the driver reads that source when the stream reaches the copy instead of at the call (the same class as #1756, #2200, #2202, and NVIDIA/cudf#23517 on GB300/GH200), d_pickups/d_deliveries receive whatever is on the stack by then, set_route_data reads out of bounds, and the sticky error 700 then fails the two set_shmem_of_kernel tests that follow. That matches the gb300 runs on 10-25 and 11-01: the PDP variant fails and the VRP variant next to it passes. On the B300 a small probe shows pageable cudaMemcpyAsync reads the source at call time, which is why it does not reproduce there.

   #2260 keeps the host sources alive (device_copy from function-scope vectors, the way the test already handles cost_matrix and order_locations) and fixes two src sites with the same pattern.
3. ramakrishnap-nv

   assigned

   0z5a

   on Nov 3, 2026