Repository navigation
Trim redundant benchmark sweep points - #871
Open
PointKernel wants to merge 1 commit into
Open
PointKernel wants to merge 1 commit into
PointKernel wants to merge 1 commit into
Conversation
PointKernel
added
the
type: improvement
PointKernel
marked this pull request as ready for review
November 8, 2026 21:22
mhaseeb123
approved these changes
Nov 11, 2026
sleeepyjack
reviewed
Nov 11, 2026
sleeepyjack
left a comment
sleeepyjack
left a comment
Collaborator
There was a problem hiding this comment.
Very good idea. I made some suggestions. Happy to discuss.
Note that this will reset cudf's automated cuco benchmarks but that shouldn't be a problem.
|
|
||
| auto const BF_SIZE_MB_RANGE_CACHE = | ||
| std::vector<nvbench::int64_t>{1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048}; | ||
| auto const BF_SIZE_MB_RANGE_CACHE = std::vector<nvbench::int64_t>{1, 4, 16, 64, 256, 1024}; |
Collaborator
There was a problem hiding this comment.
Suggested change
| auto const BF_SIZE_MB_RANGE_CACHE = std::vector<nvbench::int64_t>{1, 4, 16, 64, 256, 1024}; | |
| auto const BF_SIZE_MB_RANGE_CACHE = std::vector<nvbench::int64_t>{16, 32, 64, 128, 256, 1024}; |
Sizes below 16MB are basically noise or launch overhead. I would add back 32 and 128MB as this is the interesting region where throughput drops from L2 to DRAM.
| .set_type_axes_names({"T", "Distribution"}) | ||
| .add_int64_power_of_two_axis("NumInputs", {28, 29, 30}) | ||
| .add_int64_axis("SketchSizeKB", {8, 16, 32, 64, 128, 256}) // 256KB uses gmem fallback kernel | ||
| .add_int64_power_of_two_axis("NumInputs", {28}) |
Collaborator
There was a problem hiding this comment.
Suggested change
| .add_int64_power_of_two_axis("NumInputs", {28}) | |
| .add_int64_power_of_two_axis("NumInputs", {29}) |
More stable results.
| .set_type_axes_names({"T", "Distribution"}) | ||
| .add_int64_power_of_two_axis("NumInputs", {28, 29, 30}) | ||
| .add_int64_axis("SketchSizeKB", {8, 16, 32, 64, 128, 256}) | ||
| .add_int64_power_of_two_axis("NumInputs", {28}) |
Collaborator
There was a problem hiding this comment.
Suggested change
| .add_int64_power_of_two_axis("NumInputs", {28}) | |
| .add_int64_power_of_two_axis("NumInputs", {29}) |
More stable results
Collaborator
There was a problem hiding this comment.
dang this is 8GB in keys. I guess 4Gb is fine too
| // on demand with an nvbench axis override, e.g. `-a "Occupancy=[0.1:0.9:0.1]"`. | ||
| auto const N_RANGE = std::vector<nvbench::int64_t>{10'000'000, 50'000'000, 90'000'000}; | ||
| auto const N_RANGE_CACHE = std::vector<nvbench::int64_t>{8'000, 800'000, 8'000'000, 80'000'000}; | ||
| auto const OCCUPANCY_RANGE = std::vector<nvbench::float64_t>{0.1, 0.5, 0.8, 0.9}; |
Collaborator
There was a problem hiding this comment.
Suggested change
| auto const OCCUPANCY_RANGE = std::vector<nvbench::float64_t>{0.1, 0.5, 0.8, 0.9}; | |
| auto const OCCUPANCY_RANGE = nvbench::range(0.5, 0.9, 0.1); |
0.1 is irrelevant.
| // Sweeps keep the end points and the points where the curves bend. Denser sweeps can be run | ||
| // on demand with an nvbench axis override, e.g. `-a "Occupancy=[0.1:0.9:0.1]"`. | ||
| auto const N_RANGE = std::vector<nvbench::int64_t>{10'000'000, 50'000'000, 90'000'000}; | ||
| auto const N_RANGE_CACHE = std::vector<nvbench::int64_t>{8'000, 800'000, 8'000'000, 80'000'000}; |
Collaborator
There was a problem hiding this comment.
Suggested change
| auto const N_RANGE_CACHE = std::vector<nvbench::int64_t>{8'000, 800'000, 8'000'000, 80'000'000}; | |
| auto const N_RANGE_CACHE = std::vector<nvbench::int64_t>{2'000'000, 4'000'000, 8'000'000, 16'000'000, 80'000'000}; |
Small table sizes are purely launch overhead. Added two more data points where the drop should be visible.
| auto const SKEW_RANGE = nvbench::range(0.1, 1., 0.1); | ||
| // Sweeps keep the end points and the points where the curves bend. Denser sweeps can be run | ||
| // on demand with an nvbench axis override, e.g. `-a "Occupancy=[0.1:0.9:0.1]"`. | ||
| auto const N_RANGE = std::vector<nvbench::int64_t>{10'000'000, 50'000'000, 90'000'000}; |
Collaborator
There was a problem hiding this comment.
Suggested change
| auto const N_RANGE = std::vector<nvbench::int64_t>{10'000'000, 50'000'000, 90'000'000}; | |
| auto const N_RANGE = std::vector<nvbench::int64_t>{20'000'000, 40'000'000, 80'000'000, 100'000'000}; |
Optional. I think having at least 4 data points on the major axis gives better interpretability. 90M lands on a weird spot where dynamic_map allocates a third submap with only a few entries in it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR trims the benchmark sweeps to the points where performance changes, reducing the measured states from 1,921 to 889 and roughly halving the suite runtime.
Matching-rate sweeps are linear, occupancy is flat below 0.5, skew flattens after 0.3, the 8K and 80K capacity points are both launch-bound, and several HyperLogLog, Bloom filter, string hash, and roaring bitmap points measure identical behavior. Each sweep keeps its end points and bends, every kept value comes from the existing ranges, and denser sweeps remain available through nvbench axis overrides such as
-a "Occupancy=[0.1:0.9:0.1]".