Repository navigation
[BUG] conda-cpp-tests fail on gb300: pdlp_class batch/strong-branching, Population, ReducedMatrix #2176
Description
Describe the bug
In nightly run 38859152051 (2026-10-16), conda-cpp-tests / 13.3.0, 3.14, arm64, ubuntu26.04, gb300, latest-driver, latest-deps fails (job 116837303549): 20 failed, 6 flaky, 1057 passed. All 12 other conda-cpp-tests jobs in the run passed (incl. gh200, a100, l4, h100, rtxpro6000), so this looks gb300-specific. I have not yet checked earlier nights.
Failing tests
pdlp_classbatch / strong branching (~20):strong_branching_test,strong_branching_user_api,strong_branching_multi_bounds_per_climber,simple_batch_different_{bounds,objectives,offsets,objectives_and_offsets,constraint_bounds},simple_batch_everything_different,big_batch_fixed_path,run_batch_pdlp_many_different_bounds,batch_with_optimal_size_query,batch_settings_overrides_preserve_user_limits_and_tolerances,shared_sb_view_concurrent_mark,*_per_constraint_residual_batch_*stable3*. Example (pdlp_test.cu:3576):batch_sol2.get_termination_status(i)is0x01(Optimal) butbatch_sol.get_termination_status(i)is0x04.ReducedMatrixPrecision.SinglePrecisionWithAutomaticSizeSetting: status0x04, expectedOptimal.ReducedMatrixFallback.UnsupportedRestartScalingMatchesFullSolver: objective differs by 5.8e-4 (1.61874 vs 1.61816; tol 1e-9).Population.*(4):ExplicitLnsOriginSurvivesPublicationAndQueueDrain,ExternalQueueDrainAllowsReentrantProducerAndLeavesNewHeapPending,ExternalQueueValidatesWithSolverTolerancesBeforeReplacingIncumbent,LnsFeedTracksOnlyBestSlotAndDetachesOnDestruction(e.g.population_test.cu:508:received.size()is 0, expected 1).
The log also shows Error: conic variables must form a trailing block [linear | cone] 10 times; not yet checked whether that is expected negative-test output.
Expected behavior
Tests pass on gb300 as on the other GPUs.
Environment details
- Environment location: CI, gb300 arm64, CUDA 13.3.0, Python 3.14, ubuntu26.04, latest driver
- Method of cuOpt install: conda build from source (rattler-build)
Additional context
Related: #2136 (pdlp_class hang on GH200).
Activity
0z5a commentedon Nov 3, 2026
I ran the failing tests from this issue on an x86_64 B300 (sm_103, driver 610.57.04) with
libcuopt-tests26.12.00a58 (77e19719), in conda envs withcuda-version13.3 and 13.0. libcuopt shipssm_100fcubins only, so this is the same device code GB300 runs, just without the Grace host.All of them pass: the
pdlp_classstrong-branching and batch tests,ReducedMatrixPrecision.SinglePrecisionWithAutomaticSizeSetting,ReducedMatrixFallback.UnsupportedRestartScalingMatchesFullSolverand the fourPopulation.*tests, on both CUDA versions, and 20/20 repeats on 13.3.Looking at the gb300
conda-cpp-testsnightlies, this failure set last appears on 10-21 (a22) and 10-23 (a25,6cce8f76) and not since 10-24 (b5ad4d07). Later gb300 failures are #2212 (10-25, 11-01) and a matrix configuration error on 10-31. The 10-23 → 10-24 window contains #2202 and #2200 (host values going out of scope beforeset_value_asynccompletes), which fits a failure that shows up on the Grace host but not on x86 with the same GPU. Could this be closed as fixed by #2202, unless you have seen it since?ramakrishnap-nv commentedon Nov 3, 2026
Thank you @0z5a, Yeah, I haven't seen this issue since.