You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[cudax][cuco] Use multiply-high range reduction for probing - #12360
Open-addressing hash tables pick the first slot to probe as hash(key) % n, with n the number of buckets known only at runtime, so every lookup pays for an integer division. This PR uses a multiply-high instead, (hash * n) >> 64, which spreads a well-mixed hash just as evenly but changes which bucket each key lands in.
Codegen
Whole-kernel static SASS instruction counts for fixed_capacity_map on CUDA 13.4, sm_120, default cuda::hash (XXH64):
Kernel
Modulo
Multiply-high
Find, 32-bit keys
277
228
Find, 64-bit keys
281
231
Insert, 32-bit keys
481
436
Insert, 64-bit keys
517
469
Counts include setup and cold paths, excluding NOP padding; they are not instructions executed per lookup.
Performance
RTX PRO 6000 Blackwell Max-Q, CUDA 13.4, GCC 14, Release, sm_120. fixed_capacity_map find, 32/64-bit keys, occupancy 0.5/0.8/0.9, all hits and all misses. Geometric-mean GPU time change over 12 configurations per size, median of 3 runs; negative is faster.
Hasher
800K keys
8M keys
100M keys
Default (XXH64)
-11.7%
-8.0%
-1.3%
XXH32
-5.2%
-2.4%
-0.4%
Insert with 100M keys is unchanged.
Checklist
New or existing tests cover these changes.
The documentation is up to date with these changes.
Replace the modulo in __hash_to_index with a multiply-high range reduction,
and restrict hash and index types to unsigned integers. Containers always
index with size_t, and signed hashes were never a documented input.
Containers always index with size_t, so drop the index type parameter and
the 32-bit index test instantiations. Mark the cooperative group
make_iterator overloads device-only since tiles exist only on device.
The helper is a few lines with a single consumer, so it does not need its
own header. Drop the casts and full namespace qualifications that the
size_t signature made redundant.
Replace the modulo in initial bucket selection and double-hashing step
reduction with multiply-high range reduction (`__probe_index`), ported
from NVIDIA/cuCollections#869. Probing only needs a well-distributed
index in [0, n), not the exact remainder; this removes the 64-bit
division call from the probing path.
Reviewing files that changed from the base of the PR and between 08e79df and b9ad33e.
You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.
Use the checkbox below for a quick retry:
Trigger review
Summary
Summary by CodeRabbit
Bug Fixes
Improved hash-to-index mapping across different hash and index widths, including hashes near the maximum value.
Updated bucket selection and probing steps to use consistent hash-to-index mapping.
Expanded coverage for all-ones hashes and final-slot allocation in dynamic- and fixed-capacity sets.
Changes
Cooperative-group iterator creation for linear probing and double hashing is now device-only.
Hash-to-index mapping now uses multiply-high range reduction for integer and two-word array hashes. Probing iterators use cuda::std::size_t indices. Cooperative-group overloads are device-only. Tests cover mapping, probing behavior, and maximum-hash cases.
The new overloads map unsigned integer hashes with multiply-high arithmetic and use the high word for two-word array hashes. The previous unsigned-magnitude remainder implementation was removed.
Both probing schemes require cuda::std::size_t capacity indices and use the new mapping. Cooperative-group iterator overloads are device-only.
Mapping and probing tests cudax/test/cuco/utility/test_probing_scheme.cu, cudax/test/cuco/fixed_capacity_set/test_capacity.cu
Tests cover mapping endpoints and midpoints, wraparound, probing cycles, wide group offsets, and maximum-hash final-slot behavior.
Suggested reviewers:srinivasyadav18
Change: Refactor
Merge Risk: Low · previous review 57134
Document the supported array hash representation so users can choose it confidently. This is a bounded API-clarity issue and does not otherwise block merging.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 9 remain after this review.
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.
The reason will be displayed to describe this comment to others. Learn more.
For reference, https://github.com/PointKernel/cccl/tree/cuco-hash-range-reduction is a cleanup that addresses most of the comments below: a size_t-only __hash_to_index with a single left-align + mul_hi path, folded into detail/probing_scheme_base.cuh, and size_t-only tests with fixed expected values. Note it uses the high word for 128-bit hashes; keeping the low word as in this PR is fine too.
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 9 remain after this review.
important: Document the supported two-word array hash. Both notes state that hashers must return unsigned integers. The public make_iterator overloads pass hash results to detail::__hash_to_index, which also accepts cuda::std::array<cuda::std::uint64_t, 2> with the low word first. The probing tests exercise this representation, so the current wording excludes a supported 128-bit hash result. Add this array form to both notes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cucocuCollectionscudaxFeature intended for the cudax experimental library
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
sleeepyjack
added
cudax
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
PointKernel
left a comment
Description
Port of NVIDIA/cuCollections#869 to
cudax::cuco.Open-addressing hash tables pick the first slot to probe as
hash(key) % n, withnthe number of buckets known only at runtime, so every lookup pays for an integer division. This PR uses a multiply-high instead,(hash * n) >> 64, which spreads a well-mixed hash just as evenly but changes which bucket each key lands in.Codegen
Whole-kernel static SASS instruction counts for
fixed_capacity_mapon CUDA 13.4,sm_120, defaultcuda::hash(XXH64):Counts include setup and cold paths, excluding NOP padding; they are not instructions executed per lookup.
Performance
RTX PRO 6000 Blackwell Max-Q, CUDA 13.4, GCC 14, Release,
sm_120.fixed_capacity_mapfind, 32/64-bit keys, occupancy 0.5/0.8/0.9, all hits and all misses. Geometric-mean GPU time change over 12 configurations per size, median of 3 runs; negative is faster.Insert with 100M keys is unchanged.
Checklist