Repository navigation
Spike on tracking quality metrics for nvbench comparison scripts #415
Description
We should investigate how to track quality metrics for perf comparison:
- number of ambiguous comparison results
- number of false positives
After making changes to warmup run number, stopping criterion, and comparison scripts, we should be able to evaluate the change by tracking these metrics.
This issue can be close with:
- a list of metrics to be tracked with their descriptions (i.e. number of ambiguities, false positives etc.)
- methodology for getting these metrics (i.e. do I run a script that stores number of ambg? how do I tell a result is false positive?)
We can defer the following to a different issue:
- how to compare two sets of metrics (i.e. number of amb is lower but false positives is larger, is it better?)
Activity
fbusato commentedon Sep 22, 2026
Simplified description of this issue (from my understanding): The general idea is to find a way to evaluate the benchmarking methodology itself to allow NVBench/argument values modifications and confidently understand if they improve or not the evaluation.
Proposed methodology:
Self-Comparison
The most straightforward idea is compare a benchmark (a GPU kernel) against itself in a stable/controlled environment. The goal is to reduce false positives and ambiguity.
Objective function: maximize
SAMEoutcomesFAST,SLOWcould be considered worse thanAMBGThe comparison must be as reliable as possible. This implies:
Benchmark selection: CUB benchmarks (e.g. Smoke tests) or ad-hoc kernels (axpy, compute-bound/memory-bound microbenchmarks, etc.)
Stability: A single comparison is not sufficient (two runs: original, modified). Ideally, we should have multiple comparisons and evaluate the average between them.
Implementation: A script runs the new methodology on the benchmarks list. Then, compare the default results with the new ones. Potentially, store the new methodology parameters in a specific place to record the best one.
Synthetic Alterations
A reliable comparison must identify cases of clear slowdown or speedup. For this purpose, we can artificially alter the timing results of a benchmark (over several runs). The goal is to identify missed improvements or regressions, or even, in the worst case, swapping them.
Objective Function: maximize
SLOW/FASToutcomes. Applies to NVBench comparison script only.We can reuse the results used in *Self-Comparison and create two or more new sets, e.g. alter the first half, then the second one
SLOW/FASTImplementation: Store "frozen" in a specific place. A script artificially modify the timing, then run the comparison script (over multiple results), and evaluate the outcome.
Real Alterations
Conditions that modify the behavior of a benchmarks (slowdown/speedup) in a real-world setup would stretch the reliability of the methodology. On the other hand, they are also hard to introduce in a general way and obtain a desired behavior.
Objective Function: maximize
SLOW/FASToutcomes.SAMEis neutral.Change the GPU compute/memory frequency. Higher/lower frequencies imply (in general) faster/slower executions. If the benchmark methodology is intended to stress unlocked clocks, they are useless.
Artificial code slowdown at kernel begin:
volatileshared memory operations to generateLDS/STSasm volatilePTX with an arithmetic instruction with a dependency between input/outputArtificial compiler slowdown/speedup: e.g. debug mode vs. release mode, or
-fast-mathfor floating-point computation,--force-load/store-cache, base SM version vs. specific one (e.g.sm_100vs.sm_107)Implementation: A script compiles specific code according to 2) and/or 3), then runs NVBench over them, and evaluate the results.
gonidelis commentedon Sep 23, 2026
@fbusato
I saw the meeting recording retrospectively sorry for missing it. Some of my takes:
First of, just to make it clear. In practice are we planning to basically write quality python?? test(s) that will ignite when we make changes to
nvbench_compare.py? Or are we planning them to hit CI on nvbench PRs?Method one of comparing a benchmark against itself see which metrics remain consistent I like very much.
I had problem distinguishing the 2nd and 3rd method. Us introducing real alterations means we know the impact of these alterations which means we measure it with nvbench first? but we are using that same reference point to evaluate the tool (nvbenhc) that we used to obtain it in the first place? If you want to measure the accuracy of a measurement tool (without a second reliable measurement tool for verification) you got a chicken egg problem no?
In any case for either the synthetic or the real alterations I got tons of DBs with tunings that expect to give X speedup or slowdown for almost all of our algos. We can plug them in and see nvbench response? Though to be fair I am doing the exact opposite usually, I plug them in to see if the tuning reported speedup is true (chicken egg).
Side notes from some of my experience with nvbench:
a) When I was running verification benches for my tunings on different GPUs. One GPU would tell me tuning is super good and the other one would report very underwhelming results (many fewer
FASTs). I didn't know which one to trust. Maybe that's not an nvbench problem but rather a hardware consistency one.b) Also when I was tuning sm107 my benchmarks were maaany times hitting the gpu throttling threshold limit (75% i think) which i had to lower manually for the benchmark to be bale to reach to an end. Maybe some dynamic control of the threshold limit would help? Don't have this compeltely panned out in my mind yet.
fbusato commentedon Sep 23, 2026
thanks for looking at the issue, !
Yes, the Python script looks like a good solution. Deciding when to trigger them is a very late decision. We can start with manual testing and collect the benchmark references we want to improve. Then, we can enable CI automation.
For 2nd, it is not a problem because we are only collecting one single data point and performing an artificially altering it.
For 3rd , it could be a problem but if we know that the alteration "improves" the execution time, this should be reflected in the benchmark outcomes. To make the measurements more reliable, we should perform several of them, not just one. One useful technique is to run the benchmarks in reverse to see if the results are similar (related to 1st).
tuning is a good point that I missed, thanks for pointing it out.
it is definitely a HW consistency problem :)
@gevtushenko and I have slightly different opinions on this point. Many users and customers are unaware of the effects of GPU throttling and run applications with unlocked clocks. Others carefully select their strategy to maximize performance efficiency (e.g., power cap), control thermal effects depending on their cooling environment, or rely on max boost to maximize performance when the workload requires it and reduce power consumption when it is not needed. On the other hand, (IMO) having a benchmarking methodology that provides stable results is very important. Removing sources of noise is essential in this direction. For this reason, I would suggest locking the clocks. A good next question could be: Which locked clocks? It could be the default ones or the highest ones below throttling.