Repository navigation
[BUG] [Java] CagraRandomizedIT.testResultsTopKWithRandomValues fails randomly #1468
Description
CagraRandomizedIT.testResultsTopKWithRandomValues fails occasionally, causing CI pipelines to fail randomly.
This was seen on the second-last CI run on #1366. I was able to repro this locally:
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 3.481 s <<< FAILURE! -- in com.nvidia.cuvs.CagraRandomizedIT
[ERROR] com.nvidia.cuvs.CagraRandomizedIT.testResultsTopKWithRandomValues -- Time elapsed: 3.479 s <<< FAILURE!
java.lang.AssertionError: Not found in expected list: 448
at __randomizedtesting.SeedInfo.seed([DD841DFC69E8C00:24F49D4798CD7240]:0)
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.assertTrue(Assert.java:42)
at com.nvidia.cuvs@25.12.0/com.nvidia.cuvs.CuVSTestCase.compareResults(CuVSTestCase.java:104)
at com.nvidia.cuvs@25.12.0/com.nvidia.cuvs.CagraRandomizedIT.tmpResultsTopKWithRandomValues(CagraRandomizedIT.java:170)
at com.nvidia.cuvs@25.12.0/com.nvidia.cuvs.CagraRandomizedIT.testResultsTopKWithRandomValues(CagraRandomizedIT.java:44)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1763)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:426)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:716)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:637)
It would be good to get to the bottom of this. To avoid further churn in CI, it might be good to temporarily back this test out.
Activity
Disable flaky Java tests to avoid CI churn
ldematte commentedon Oct 28, 2025
@mythrocks is this a duplicate of #1387, or a new/different issue?
[Review] [Java] Disable flaky Java tests to reduce CI churn (#1469)
mythrocks commentedon Oct 28, 2025
Ah, there it is. Sorry, yes. This is a dupe of #1387.
On the bright side, we have 2 seeds for failure repro.
mythrocks commentedon Oct 28, 2025
It is extremely annoying that neither of the failure seeds are reproducing the failure on my workstation. These were failing reliably late last night.
ldematte commentedon Oct 29, 2025
I can reproduce it with the seed; this week is packed but I'll get around to investigate this as soon as I can.
ldematte commentedon Nov 6, 2025
@mythrocks I started investigating this.
First of all: it fails very rarely; the test generates random datasets and executes random queries with 3 different memory types, 100 times (for a total of 300 each test run).
The failure is that the results found by brute-forcing are different from the one from Cagra.
It is not a problem of the memory type: it fails with any (Heap, Native, Device), and for the failures I found I dumped the matrix content -- and it stays the same, so no copy error or anything like that.
I've not written this test so I'm not sure where the problem is. The brute-force looks reasonable to me. It might be a cuvs bug, or simply differences in the way CAGRA computes the graph leading to different results.
I think this test is valuable and we should run it to prove things operate as expected, but I'm not sure how we can proceed, especially in the second case (no bug, but differences that can happen. How do we weed them out while keeping the test relevant?)
ldematte commentedon Nov 6, 2025
Also, I think we should close either this or #1387. Keeping both open will just add noise.
mythrocks commentedon Nov 6, 2025
I thought I'd already closed #1387 as a dupe of this. Closed now.
We'll carry on here. Thanks for picking this up, @ldematte. I'll assign this to you.
I'll defer to the domain experts for this question: @benfred, @cjnolet.
cjnolet commentedon Nov 14, 2025
We chatted about this offline. We should never be expecting or relying upon any type of exactness in approximate algorithms. Instead we should be using a "summary statistic" like recall and verifying that we consistently stay above a particular recall value. @chatman @narangvivek10 can you take a look at this?
[Review] [Java] Disable flaky Java tests to reduce CI churn (NVIDIA#1469
bdice commentedon Oct 11, 2026
Root cause analysis from the flaky-test sweep (#2783):
CagraRandomizedIT.testResultsTopKWithRandomValues#1048, [BUG] [Java]CagraRandomizedIT.testResultsTopKWithRandomValuesfails randomly #1468). The Java bindings sized the prefilter host buffer withBitSet.toLongArray(), which drops trailing all-zero words, so the copy to the GPU read past the end of the buffer. Stray bits then let filtered-out rows in the first dropped 64-bit word through. The ids in the reports (192 and 448) are both multiples of 64. Fix prefilter bitmap host buffer under-allocation in Java bindings #2375 fixed this on 2026-07-29. Restoring the old buffer sizing in CI reproduces the same assertion,Not found in expected list: 192at n=195, and every leaked row is a filtered-out row at the start of a dropped word: https://github.com/NVIDIA/cuvs/actions/runs/381147592552k + 10that the test required. The returned ids and distances are correct (true rank 20 against a window of 20, for example). This still happens rarely on current main and in releases back to 25.02, so the test's expectation was wrong.#2830 re-enables
CagraRandomizedIT. For every query it checks distinct, valid ids, no rows excluded by the prefilter, sorted exact distances, and that a copy of a row is returned first. It checks recall only as a mean per memory kind, with bounds set from about 300 test runs measured in CI. While re-enabling the test it also found and fixed a third bindings bug: a prefilter that excludes every row was treated as "no filter". 400 CI runs of the new test (about 120,000 random cases) had no failures: https://github.com/NVIDIA/cuvs/actions/runs/38138513521