[aamijar]

Resolves #1452

The test_ivf_flat.py::test_filtered_ivf_flat[0.5] has been consistently failing in the cuvs CI recently.
This test uses a dataset of 10,000 points but uses the default parameters of n_lists=1024 and n_probes=20 which means there are around only 10 points per cluster. However filtering 50% [0.5] can invalidate a lot of nearest neighbors candidates. To solve this issue we need to properly tune n_lists to be smaller and have more points per cluster. We can also increase the n_probes to look at more clusters.

This PR changes the pytest to the following:

[0.5] 50% of points are filtered
n_lists = 50
n_probes = 10
search 2000 of 10,000 points

==================================================
Recall Statistics Comparison 0.5
==================================================
                OLD         NEW
--------------------------------------------------
Min Recall:     0.6900      0.8700
Max Recall:     0.9500      1.0000
Mean Recall:    0.8327      0.9615
Std Dev:        0.0431      0.0230
Total runs:     1000        1000
==================================================

[0.7] 30% of the points are filtered
n_lists = 70
n_probes = 10
search 1428 of 10,000 points

==================================================
Recall Statistics Comparison 0.7
==================================================
                OLD         NEW
--------------------------------------------------
Min Recall:     0.7300      0.8500
Max Recall:     0.9600      1.0000
Mean Recall:    0.8623      0.9509
Std Dev:        0.0392      0.0256
Total runs:     1000        1000
==================================================

[1.0] 0% of the points are filtered
n_lists = 100
n_probes = 10
search 1000 of 10,000 points

==================================================
Recall Statistics Comparison 1.0
==================================================
                OLD         NEW
--------------------------------------------------
Min Recall:     0.7200      0.8300
Max Recall:     0.9900      1.0000
Mean Recall:    0.8886      0.9430
Std Dev:        0.0369      0.0271
Total runs:     1000        1000
==================================================

Upon investigating this test, the variable sparsity is misleading, since higher value means more sparse, but in the code it means no filtering. We could rename it to density if desired.

Justification of initial value of n_lists choice under 0% filtering.

Setting n_lists = sqrt(n_samples) is a good starting point (where n_samples is the number of vectors in the dataset)

Reference from https://developer.nvidia.com/blog/accelerated-vector-search-approximating-with-nvidia-cuvs-ivf-flat/

[divyegala]

/merge