Repository navigation

[FEA] Multi-node Out of Core Streaming KMeans API - #2066

Merged

rapids-bot[bot] merged 234 commits into

NVIDIA:mainfrom

tarang-jain:mnmg-streaming

Jul 13, 2026

Merged

[FEA] Multi-node Out of Core Streaming KMeans API#2066
rapids-bot[bot] merged 234 commits into
NVIDIA:mainfrom
tarang-jain:mnmg-streaming

Commits on Apr 10, 2026
- combine impls
  tarang-jain
  committed
  66d7fd3
Commits on Apr 13, 2026
- Multi-GPU Batched KMeans
  viclafargue
  committed
  07707af
- Merge branch 'main' into mg-batched-kmeans
  viclafargue
  committed
  efc270f
- rm inertia_check
  tarang-jain
  committed
  0a09e6f
- change to warning
  tarang-jain
  committed
  99a5730
- style
  tarang-jain
  committed
  a077406
- add init_size param
  tarang-jain
  committed
  d659875
- Merge branch 'main' into combine-batch
  tarang-jain
  authored
  ec2e8b7
- docs
  tarang-jain
  committed
  03a6473
- Merge branch 'combine-batch' of https://github.com/tarang-jain/cuvs into combine-batch
  tarang-jain
  committed
  42a8d9d
- rm direct cuda api calls
  tarang-jain
  committed
  86af2fa
Commits on Apr 14, 2026
- std::swap instead of raft::copy
  tarang-jain
  committed
  d4e4e2c
- cache batch norms
  tarang-jain
  committed
  0819af5
- centroid norms can also be cached per iteration
  tarang-jain
  committed
  e0f079c
- mg n_iter
  tarang-jain
  committed
  c2f7390
- pre-commit
  tarang-jain
  committed
  b9c3102
- do not break c abi
  tarang-jain
  committed
  e3956c1
- Merge branch 'main' into combine-batch
  tarang-jain
  authored
  986d78a
- cluster_cost on device
  viclafargue
  committed
  7197b71
- Updated testing
  viclafargue
  committed
  84ab315
Commits on Apr 15, 2026
- templating
  viclafargue
  committed
  47d4b94
Commits on Apr 16, 2026
- Merge branch 'main' into combine-batch
  tarang-jain
  authored
  a8e1d26
Commits on Apr 21, 2026
- fix checkWeight
  tarang-jain
  committed
  384d054
- merge upstream:
  tarang-jain
  committed
  455b286
- Merge branch 'combine-batch' of https://github.com/tarang-jain/cuvs into combine-batch
  tarang-jain
  committed
  5462809
- fix compilation
  tarang-jain
  committed
  6ba759c
Commits on Apr 22, 2026
- rel_tol
  
  tarang-jain
  and
  viclafargue
  authored
  e76eaac
- pass workspace
  tarang-jain
  committed
  afbefdf
- Merge branch 'combine-batch' of https://github.com/tarang-jain/cuvs into combine-batch
  tarang-jain
  committed
  e62a63c
- style
  tarang-jain
  committed
  e4f08bf
- Merge branch 'main' of https://github.com/rapidsai/cuvs into combine-batch
  tarang-jain
  committed
  6e4a8f0
- do not use batch scratch space; rm update_centroids
  tarang-jain
  committed
  4a8a85c
- move the debug log
  tarang-jain
  committed
  bbf2a9f
- add new suffixed param struct
  tarang-jain
  committed
  410092c
- address pr reviews
  tarang-jain
  committed
  c515c1e
- fix docstring
  tarang-jain
  committed
  e8e63ab
- fix wt_sum warning
  tarang-jain
  committed
  30c457c
- rm deprecationwarning and instead add FutureWarning:=
  tarang-jain
  committed
  ab96623
- unweighted to never materialize batch weights
  tarang-jain
  committed
  269f23c
Commits on Apr 23, 2026
- add cpp tests
  tarang-jain
  committed
  80a22ca
- update cpp tests
  tarang-jain
  committed
  ac06b05
- Merge branch 'main' into mg-batched-kmeans
  viclafargue
  committed
  855624a
- refactor
  viclafargue
  committed
  0a6748d
- rename to mnmg_fit
  viclafargue
  committed
  7055272
- revert batch norms cache
  tarang-jain
  committed
  0569340
Commits on Apr 24, 2026
- increase zero cost threshold
  tarang-jain
  committed
  8cac63a
- apply cuda event plus re-add h_norm_cache
  tarang-jain
  committed
  f6df4ae
- rm cosine expanded stuff
  tarang-jain
  committed
  9fc74b1
Commits on Apr 28, 2026
- resolve merge conflicts
  tarang-jain
  committed
  dec3dc4
- change suffix of the params struct
  tarang-jain
  committed
  0d030a2
- replace 06 by 08, add todo and note
  tarang-jain
  committed
  b1c034e
- update to v2
  tarang-jain
  committed
  a482495
Commits on Apr 29, 2026
- avoid stream sync inside weight sum
  tarang-jain
  committed
  8ecfdc1
- Merge branch 'combine-batch' of https://github.com/tarang-jain/cuvs into combine-batch
  tarang-jain
  committed
  1e1525e
- empty
  tarang-jain
  committed
  ec22e07
- empty
  tarang-jain
  committed
  d2e410d
- Merge branch 'main' into combine-batch
  tarang-jain
  authored
  b791c38
- new signatures with new struct
  tarang-jain
  committed
  a05a006
- Merge branch 'combine-batch' of https://github.com/tarang-jain/cuvs into combine-batch
  tarang-jain
  committed
  73293cf
Commits on Apr 30, 2026
- Merge branch 'main' of https://github.com/rapidsai/cuvs into combine-batch
  tarang-jain
  committed
  880c7b9
- revert change to calls in py and rust; add c tests
  tarang-jain
  committed
  e2035ec
Commits on May 1, 2026
- Merge branch 'main' into combine-batch
  tarang-jain
  authored
  e28c200
Commits on May 5, 2026
- use to_dlpack
  tarang-jain
  committed
  55bbdad
- cache device weights
  tarang-jain
  committed
  9a9b8ee
- rm event
  tarang-jain
  committed
  a800b27
- update names
  tarang-jain
  committed
  3db8582
- rename
  tarang-jain
  committed
  c048352
- rm docs
  tarang-jain
  committed
  2f968f8
- empty
  tarang-jain
  committed
  affe85a
- fix norm cache
  tarang-jain
  committed
  c6dea64
Commits on May 6, 2026
- revert changes to minClusterDistanceCompute
  tarang-jain
  committed
  7dfab3e
- update tests to use mdspan instead of rmm
  tarang-jain
  committed
  7a383da
- Merge branch 'main' into combine-batch
  tarang-jain
  authored
  ce6c4b5
- Merge branch 'main' into combine-batch
  tarang-jain
  authored
  5a06a44
Commits on May 7, 2026
- consolidate all unsigned commits
  tarang-jain
  committed
  419619a
- rm diff
  tarang-jain
  committed
  2d716ae
- allow batch sample weights
  tarang-jain
  committed
  066092b
- Merge branch 'main' into mnmg-streaming
  tarang-jain
  authored
  bbdd66d
- single partition becomes special case
  tarang-jain
  committed
  12d682c
- Merge branch 'mnmg-streaming' of github.com:tarang-jain/cuvs into mnmg-streaming
  tarang-jain
  committed
  9e5e55c
- Merge branch 'combine-batch' into mg-batched-kmeans
  viclafargue
  committed
  28cda6a
- Addressing review
  viclafargue
  committed
  bfb5290
- optimize convergence check
  viclafargue
  committed
  add9db1
- Merge branch 'main' into mnmg-streaming
  tarang-jain
  authored
  6c08a7b
- Merge branch 'main' into mnmg-streaming
  tarang-jain
  authored
  acbcd5a
Commits on May 8, 2026
- Adressing review
  viclafargue
  committed
  af606bc
- Merge branch 'main' into mg-batched-kmeans
  viclafargue
  committed
  41c66b8
- results on all ranks for RAFT + small optimization
  viclafargue
  committed
  f664c2c
- Merge branch 'main' into mnmg-streaming
  tarang-jain
  authored
  5430f42
- merge origin
  tarang-jain
  committed
  b2ab5bd
- Merge branch 'mnmg-streaming' of github.com:tarang-jain/cuvs into mnmg-streaming
  tarang-jain
  committed
  bbdf521
- Merge branch 'main' into mnmg-streaming
  tarang-jain
  authored
  10e6def
- reduce diff
  tarang-jain
  committed
  2040145
- Merge branch 'mnmg-streaming' of github.com:tarang-jain/cuvs into mnmg-streaming
  tarang-jain
  committed
  1828462
- Merge branch 'main' into mnmg-streaming
  tarang-jain
  authored
  5c5b8c8
- rm prefetch
  tarang-jain
  committed
  05da5f3
- Merge branch 'mnmg-streaming' of github.com:tarang-jain/cuvs into mnmg-streaming
  tarang-jain
  committed
  90435c1
- Merge branch 'main' into mnmg-streaming
  tarang-jain
  authored
  db41338
Commits on May 11, 2026
- reviews
  viclafargue
  committed
  6c2c03d
- Global sampling for init
  viclafargue
  committed
  7f6d664
- SNMG -> MNMG
  viclafargue
  committed
  f8270e2
- Merge branch 'main' into mg-batched-kmeans
  viclafargue
  committed
  bbf0302
- adding asserts
  viclafargue
  committed
  a14a6bc
- consume new init
  tarang-jain
  committed
  7b54a42
- Merge branch 'mnmg-streaming' of github.com:tarang-jain/cuvs into mnmg-streaming
  tarang-jain
  committed
  d86b8b4
- reduce diff
  tarang-jain
  committed
  6e11f67
Commits on May 13, 2026
- Merge branch 'main' into mnmg-streaming
  tarang-jain
  authored
  9f5b6e5
- rm unnecessary functions
  tarang-jain
  committed
  aaef638
- Merge branch 'main' into mnmg-streaming
  tarang-jain
  authored
  920a460
Commits on May 14, 2026
Commits on May 20, 2026
- merge upstream
  tarang-jain
  committed
  51fbf6c
- cleanup; re-add device side overload
  tarang-jain
  committed
  d327569
- re-instate removed docs
  tarang-jain
  committed
  b5e66a3
Commits on May 21, 2026
- rm extra fit funcs
  tarang-jain
  committed
  a636188
Commits on May 26, 2026
- Merge branch 'release/26.06' into mnmg-streaming
  tarang-jain
  authored
  d3cafed
- Merge branch 'mnmg-streaming' of github.com:tarang-jain/cuvs into mnmg-streaming
  tarang-jain
  committed
  1b547f4
- cleanup
  tarang-jain
  committed
  72cfd43
- rm scaled_weights_cache
  tarang-jain
  committed
  4d25e95
- rm unnecessary new types
  tarang-jain
  committed
  81155e6
- rm unused helper
  tarang-jain
  committed
  85522aa
Commits on May 27, 2026
- rm unnecessary stream sync
  tarang-jain
  committed
  00336b5
- rm unnecessary lambda
  tarang-jain
  committed
  6585866
- cleanup impl
  tarang-jain
  committed
  aa6f28e
- rm unnecessary has_data guards
  tarang-jain
  committed
  178a7e7
- rm global_n host scalar
  tarang-jain
  committed
  713bc7c
- fixes
  tarang-jain
  committed
  c576d8f
- Merge branch 'release/26.06' into mnmg-streaming
  tarang-jain
  authored
  a401a0e
- fuse with in-memory impl
  tarang-jain
  committed
  8102596
- Merge branch 'mnmg-streaming' of github.com:tarang-jain/cuvs into mnmg-streaming
  tarang-jain
  committed
  00d0adb
- style
  tarang-jain
  committed
  caefd53
- fix compilation
  tarang-jain
  committed
  8f6f83d
Commits on May 28, 2026
- Merge branch 'release/26.06' into mnmg-streaming
  tarang-jain
  authored
  d88a991
- mg tests first commit
  tarang-jain
  committed
  1b57b74
- Merge branch 'mnmg-streaming' of github.com:tarang-jain/cuvs into mnmg-streaming
  tarang-jain
  committed
  7bac418
- merge upstream
  tarang-jain
  committed
  9851017
Commits on May 29, 2026
- update cmakelists
  tarang-jain
  committed
  f572877
- merge upstream
  tarang-jain
  committed
  edaa7e7
- rm batched tests
  tarang-jain
  committed
  588bb6a
- Merge branch 'main' into mnmg-streaming
  tarang-jain
  authored
  ad180ed
- rm unnecessary test stream sycns
  tarang-jain
  committed
  72cc34b
- reset bs; assertion
  tarang-jain
  committed
  ed50703
- Merge branch 'mnmg-streaming' of github.com:tarang-jain/cuvs into mnmg-streaming
  tarang-jain
  committed
  28f6036
- rm has_data flag
  tarang-jain
  committed
  a811c56
Commits on May 30, 2026
- Merge branch 'main' into mnmg-streaming
  tarang-jain
  authored
  95f334c
- fix export
  tarang-jain
  committed
  d176314
- Merge branch 'mnmg-streaming' of github.com:tarang-jain/cuvs into mnmg-streaming
  tarang-jain
  committed
  1db9e02
Commits on May 31, 2026
- avoid pinned scalar;get_nccl_comms before omp
  tarang-jain
  committed
  089e970
Commits on Jun 1, 2026
- use root from macro
  tarang-jain
  committed
  6cc895c
- avoid copy and rank alloc with initarray
  tarang-jain
  committed
  4abe6f2
- Merge branch 'main' of https://github.com/rapidsai/cuvs into mnmg-streaming
  tarang-jain
  committed
  ebf188a
- fix compilation; guardrail MG CMake flag
  tarang-jain
  committed
  785e4a3
- get n_features from centroids
  tarang-jain
  committed
  9a526c8
- add sigs to header
  tarang-jain
  committed
  f08e581
Commits on Jun 5, 2026
- Merge branch 'main' into mnmg-streaming
  tarang-jain
  authored
  51efb42
Commits on Jun 8, 2026
- Merge branch 'main' into mnmg-streaming
  tarang-jain
  authored
  3e3cac7
Commits on Jun 9, 2026
- Merge branch 'main' into mnmg-streaming
  tarang-jain
  authored
  7ffae6d
- put single mdspan in mg namespace
  tarang-jain
  committed
  447e136
- Merge branch 'mnmg-streaming' of https://github.com/tarang-jain/cuvs into mnmg-streaming
  tarang-jain
  committed
  50d0359
- single reduce
  tarang-jain
  committed
  b1e7521
- re-add distributed kmeanspp
  tarang-jain
  committed
  f8a3503
- revert differences from distributed init
  tarang-jain
  committed
  272a9d5
- Merge branch 'main' into mnmg-streaming
  tarang-jain
  authored
  1f9fd9e
- optional allredice
  tarang-jain
  committed
  ed705c6
- Merge branch 'mnmg-streaming' of https://github.com/tarang-jain/cuvs into mnmg-streaming
  tarang-jain
  committed
  c8135e5
- Merge branch 'main' into mnmg-streaming
  tarang-jain
  authored
  cf5e831
Commits on Jun 10, 2026
- apply distributed init fixes
  tarang-jain
  committed
  8b6eab2
- Merge branch 'mnmg-streaming' of https://github.com/tarang-jain/cuvs into mnmg-streaming
  tarang-jain
  committed
  100f7d6
- re-add host assertion for sum of weights
  tarang-jain
  committed
  46d18d1
- remaining samples should be global
  tarang-jain
  committed
  db23a72
- doc updates
  tarang-jain
  committed
  1e7c119
- add dask-like cpp test
  tarang-jain
  committed
  116a6cf
Commits on Jun 11, 2026
- add warning to docs
  tarang-jain
  committed
  ae284be
- Merge branch 'main' into mnmg-streaming
  tarang-jain
  authored
  d15cb28
- address pr reviews
  tarang-jain
  committed
  653aac1
- Merge branch 'mnmg-streaming' of https://github.com/tarang-jain/cuvs into mnmg-streaming
  tarang-jain
  committed
  92588ba
- add weight check for sg device weights
  tarang-jain
  committed
  ef60e3c
- re-update the docs
  tarang-jain
  committed
  8831789
- update docs
  tarang-jain
  committed
  3f800a1
- undo fern changes
  tarang-jain
  committed
  32be863
- Merge branch 'main' into mnmg-streaming
  tarang-jain
  authored
  d0fa3a9
Commits on Jun 12, 2026
- address reviews
  tarang-jain
  committed
  0b969c4
- Merge branch 'mnmg-streaming' of https://github.com/tarang-jain/cuvs into mnmg-streaming
  tarang-jain
  committed
  040e82b
- fix bcast
  tarang-jain
  committed
  d65287d
- style
  tarang-jain
  committed
  3321926
- update docstring
  tarang-jain
  committed
  de57f46
Commits on Jun 15, 2026
- add oversampling test
  tarang-jain
  committed
  1a6bbb5
Commits on Jun 16, 2026
- address reviews
  tarang-jain
  committed
  e627af4
- Merge branch 'main' into mnmg-streaming
  tarang-jain
  authored
  fd19b43
Commits on Jun 22, 2026
- Merge branch 'main' into mnmg-streaming
  tarang-jain
  authored
  993b813
Commits on Jun 23, 2026
- Merge branch 'main' into mnmg-streaming
  tarang-jain
  authored
  bd3e1de
Commits on Jun 24, 2026
- Merge branch 'main' into mnmg-streaming
  tarang-jain
  authored
  40c77ca
- Merge branch 'main' into mnmg-streaming
  tarang-jain
  authored
  079c216
- Merge branch 'main' into mnmg-streaming
  tarang-jain
  authored
  7a38f62
Commits on Jun 25, 2026
- do not use nullptr for bcast
  tarang-jain
  committed
  6ada5f6
- fix mnmg tests
  tarang-jain
  committed
  8ce1ade
Commits on Jun 26, 2026
- dependencies for devc
  tarang-jain
  committed
  53ca72b
- versioning for libucxx
  tarang-jain
  committed
  7cb1397
Commits on Jul 6, 2026
- correct the name for the dependency
  tarang-jain
  committed
  edcc1fc
- Merge branch 'main' of https://github.com/rapidsai/cuvs into mnmg-streaming
  tarang-jain
  committed
  a743386
- style
  tarang-jain
  committed
  2f15953
Commits on Jul 7, 2026
- update cmake find_package for ucxx
  tarang-jain
  committed
  87e8819
- fix conda recipe
  tarang-jain
  committed
  c856300
- only depend on libucxx
  tarang-jain
  committed
  2270ab6
- Merge branch 'main' into mnmg-streaming
  tarang-jain
  authored
  61808e4
- find ucxx in site-packages
  tarang-jain
  committed
  a0949e1
- Merge branch 'mnmg-streaming' of https://github.com/tarang-jain/cuvs into mnmg-streaming
  tarang-jain
  committed
  20fee2e
Commits on Jul 8, 2026
- rm from top level recipe
  tarang-jain
  committed
  defb785
- Add to cached host env
  tarang-jain
  authored
  36fcf36
- Merge branch 'main' into mnmg-streaming
  tarang-jain
  authored
  8dd17f3
- get deps from raft
  tarang-jain
  committed
  1b42f14
- Merge branch 'mnmg-streaming' of https://github.com/tarang-jain/cuvs into mnmg-streaming
  tarang-jain
  committed
  59eb36e
- cleanup recipe
  tarang-jain
  committed
  33ee038
- Merge branch 'main' into mnmg-streaming
  tarang-jain
  authored
  ccd5e43
- copyright check
  tarang-jain
  committed
  f0236cc
- Merge branch 'mnmg-streaming' of https://github.com/tarang-jain/cuvs into mnmg-streaming
  tarang-jain
  committed
  21864d2
- add find_package
  tarang-jain
  committed
  4da8077
Commits on Jul 9, 2026
- add to dependencies.yaml
  tarang-jain
  committed
  a0edd32
- add to all.includes
  tarang-jain
  committed
  c8db7ff
- several fixes
  tarang-jain
  committed
  76b9913
- Merge branch 'main' into mnmg-streaming
  tarang-jain
  authored
  fe98618
- add RUN_SERIAL
  tarang-jain
  committed
  b4872c5
- Merge branch 'mnmg-streaming' of https://github.com/tarang-jain/cuvs into mnmg-streaming
  tarang-jain
  committed
  07b5a39
Commits on Jul 10, 2026
- Merge branch 'main' of https://github.com/rapidsai/cuvs into mnmg-streaming
  tarang-jain
  committed
  543bef5
- call mg api
  tarang-jain
  committed
  531c9a6
- Merge branch 'main' into mnmg-streaming
  tarang-jain
  authored
  78a4278
- fix mangled whitespace
  tarang-jain
  committed
  484d102
- Merge branch 'mnmg-streaming' of https://github.com/tarang-jain/cuvs into mnmg-streaming
  tarang-jain
  committed
  31ea620
- Merge branch 'main' into mnmg-streaming
  tarang-jain
  authored
  2eed544
Commits on Jul 13, 2026
- Merge branch 'main' of https://github.com/rapidsai/cuvs into mnmg-streaming
  tarang-jain
  committed
  26b10c2
- rm mg namespace
  tarang-jain
  committed
  b1565c5
- style
  tarang-jain
  committed
  c1f661b
- fix docs
  tarang-jain
  committed
  ee6c302

Commits

Commits on Apr 10, 2026

Commits on Apr 13, 2026

Commits on Apr 14, 2026

Commits on Apr 15, 2026

Commits on Apr 16, 2026

Commits on Apr 21, 2026

Commits on Apr 22, 2026

Commits on Apr 23, 2026

Commits on Apr 24, 2026

Commits on Apr 28, 2026

Commits on Apr 29, 2026

Commits on Apr 30, 2026

Commits on May 1, 2026

Commits on May 5, 2026

Commits on May 6, 2026

Commits on May 7, 2026

Commits on May 8, 2026

Commits on May 11, 2026

Commits on May 13, 2026

Commits on May 14, 2026

Commits on May 20, 2026

Commits on May 21, 2026

Commits on May 26, 2026