Repository navigation
Extend comparison script to allow comparison of two distributions of timing samples #313
Description
The results of an nvbench benchmark are often used when comparing two implementations of a kernel and one wants to decide which is "faster".
In general, this is a complex question to answer as performance data is rarely normally distributed, and so therefore approaches with comparing the mean are not statistically robust.
The ideal would be to enable comparing two collections of samples of performance data collected from nvbench benchmarks and determining which is faster according to more statistically rigorous comparison criteria.
Reacted by Federico Busato and Oleksandr Pavlyk
Reacted by Giannis Gonidelis
Metadata
Metadata
Assignees
Labels
No labels
Type
Projects
- Status
Activity
bernhardmgruber commentedon Feb 3, 2026
This is great! We should also care about the visualization of this comparison. We do a lot of benchmark runs when improving CUB algorithms, for example diffing the performance of
cub::DeviceTransformonBabelStreaminvolves about 200 runs, represented as terminal output of about 200 lines. We should make sure that comparing 200 distributions with 200 others can also be viewed in a similarly amount of screen space, so regression analysis for humans stays manageable.nvbench_compare.py#320