[oleksandr-pavlyk]

This PR builds on top of #338, so it should be merged after #338 is merged.

Closes #188

---

This PR add stopping criterion sample-count with target-samples integral parameter, which allows users to collect deterministic number of samples.

### "sample-count" Stopping Criterion Parameters

* `--target-samples <count>`
  * Stop after at least `<count>` samples are collected.
  * Default is 100 samples.
  * The total number of collected samples is
    `max(--min-samples, --target-samples)`.
  * Applies to the most recent `--benchmark`, or all benchmarks if specified
    before any `--benchmark` arguments.

Together with --warmup-runs <count> this permits users to replicate their prior art of running fixed warm-up count and fixed timed runs, but still benefit from NVBench's benchmark reporting facilities, and more accurate GPU timing.

The sample-count stopping criterion is essentially the "fixed" custom stopping criterion from examples/custom_criterion.cu, but interaction with --min-samples has changed.

• Previously, stopping criterion was applied once m_total_samples > m_min_samples, so running ./build/bin/nvbench.example.cpp20.custom_criterion --min-samples 50 would collect 51 samples.

• This PR replaced that condition with m_total_samples >= m_min_samples instead.

  • Edge case is handled stdrel stopping criterion's do_is_finished() method to ensure m_noise_tracker.back() is safe to call.

• The custom_criterion.cu example needs to be changed to implement an alternative stopping criterion in the future.

[oleksandr-pavlyk]

@coderabbitai full review

[coderabbitai]

✅ Actions performed

Full review triggered.

[coderabbitai]

📝 Walkthrough

Summary by CodeRabbit

• New Features

  • Added "sample-count" stopping criterion with configurable --target-samples (default: 100).

• Bug Fixes

  • Measurement loops now consider minimum sample counts met at the threshold (>=), preventing off-by-one early/late termination.
  • Standard-relative criterion now guards against running convergence checks before any noise samples exist.

• Tests

  • Added unit/regression tests covering sample-count behavior, validation of target-samples, and related stopping-criterion interactions.

• Documentation

  • CLI help updated with sample-count option, --target-samples semantics and defaults.

suggestion:

Walkthrough

Adds a "sample-count" stopping criterion with a --target-samples parameter (default 100), integrates it into the criterion manager and build, changes min-samples comparisons to inclusive (>=) in measurement loops, hardens stdrel against empty noise, updates CLI docs, and adds tests.

Changes

sample-count stopping criterion

Layer / File(s)	Summary
sample_count_criterion class definition and implementation nvbench/detail/sample_count_criterion.cuh, nvbench/detail/sample_count_criterion.cxx	Declares and implements sample_count_criterion as a final subclass of stopping_criterion_base, tracking measurement count and finishing when accumulated samples reach the target-samples parameter (default 100).
Criterion manager registration and build system nvbench/criterion_manager.cuh, nvbench/criterion_manager.cxx, nvbench/CMakeLists.txt	Registers the new criterion by including its header and emplacing it in the criterion map, and adds the implementation file to the library's compiled sources.
min_samples boundary condition fixes across measurement loops nvbench/detail/measure_cold.cu, nvbench/detail/measure_cpu_only.cxx, nvbench/detail/measure_hot.cuh	Adjusts termination logic from strictly m_total_samples > m_min_samples to inclusive m_total_samples >= m_min_samples, allowing measurements to finish immediately when the minimum is exactly reached.
stdrel_criterion empty noise tracker guard nvbench/detail/stdrel_criterion.cxx	Returns false early in do_is_finished() if m_noise_tracker is empty, preventing downstream noise-threshold and convergence logic from executing before measurements exist.
CLI documentation for sample-count criterion docs/cli_help.md	Documents "sample-count" as a stopping criterion option with --target-samples parameter, including default (100), interaction with --min-samples, and applicability scope.
Tests for sample-count criterion and related changes testing/CMakeLists.txt, testing/sample_count_criterion.cu, testing/criterion_manager.cu, testing/option_parser.cu, testing/stdrel_criterion.cu	Unit tests validating default (100) and custom target-samples behavior, option-parser tests confirming parameter independence from --min-samples and scoping rules, criterion manager registration, and stdrel robustness with minimal samples.

Assessment against linked issues

Objective	Addressed	Explanation
Add CLI parameter to fix the run count [#188]	✅

Suggested reviewers

• gevtushenko

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (3)

testing/option_parser.cu (1)

1385-1423: ⚡ Quick win

suggestion: add one regression that sets --target-samples globally before multiple --benchmark entries and then overrides it for one benchmark, then assert each state gets the expected scoped value. This guards the documented “most recent benchmark or all-before-benchmark” behavior.

As per coding guidelines testing/**/*: focus on whether tests cover observable behavior and remain deterministic.

testing/sample_count_criterion.cu (1)

24-55: ⚡ Quick win

suggestion: add boundary cases for target-samples=1 and a non-positive value to lock down criterion semantics at limits and prevent silent behavior drift.

As per coding guidelines testing/**/*: focus on whether tests cover observable behavior and remain deterministic.

nvbench/detail/sample_count_criterion.cxx (1)

28-35: ⚡ Quick win

important: validate target-samples > 0 in initialization (or clamp to 1) before using it in do_is_finished(). Without a guard, non-positive values can make the criterion complete immediately.

Based on learnings --target-samples <count> is documented as a sample count used to stop after at least that many samples.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 8f6f7214-dda4-456b-a3fb-7e0d7a3bb46b

📥 Commits

Reviewing files that changed from the base of the PR and between 9ea77bc and 5f15bd6.

📒 Files selected for processing (15)

• docs/cli_help.md
• nvbench/CMakeLists.txt
• nvbench/criterion_manager.cuh
• nvbench/criterion_manager.cxx
• nvbench/detail/measure_cold.cu
• nvbench/detail/measure_cpu_only.cxx
• nvbench/detail/measure_hot.cuh
• nvbench/detail/sample_count_criterion.cuh
• nvbench/detail/sample_count_criterion.cxx
• nvbench/detail/stdrel_criterion.cxx
• testing/CMakeLists.txt
• testing/criterion_manager.cu
• testing/option_parser.cu
• testing/sample_count_criterion.cu
• testing/stdrel_criterion.cu

[oleksandr-pavlyk]

@coderabbitai full review

[coderabbitai]

✅ Actions performed

Full review triggered.

[coderabbitai]

🧹 Nitpick comments (1)

testing/stdrel_criterion.cu (1)

81-94: ⚡ Quick win

suggestion: extend this test to also assert the boundary transition in the same setup (unfinished at 4, finished after the 5th sample) so it validates both sides of the sample-threshold behavior directly.

patch

 void test_stdrel_needs_enough_samples()
 {
   nvbench::criterion_params params;
   params.set_float64("min-time", 0.0);

   nvbench::detail::stdrel_criterion criterion;
   criterion.initialize(params);

   for (int i = 0; i < 4; ++i)
   {
     criterion.add_measurement(42.0);
   }
   ASSERT(!criterion.is_finished());
+
+  criterion.add_measurement(42.0);
+  ASSERT(criterion.is_finished());
 }

As per coding guidelines: `testing/**/*`: Focus on whether tests cover observable behavior and remain deterministic.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 66f9598f-ab4e-418c-86c4-10a4667d748b

📥 Commits

Reviewing files that changed from the base of the PR and between 338936b and 66c7561.

📒 Files selected for processing (15)

• docs/cli_help.md
• nvbench/CMakeLists.txt
• nvbench/criterion_manager.cuh
• nvbench/criterion_manager.cxx
• nvbench/detail/measure_cold.cu
• nvbench/detail/measure_cpu_only.cxx
• nvbench/detail/measure_hot.cuh
• nvbench/detail/sample_count_criterion.cuh
• nvbench/detail/sample_count_criterion.cxx
• nvbench/detail/stdrel_criterion.cxx
• testing/CMakeLists.txt
• testing/criterion_manager.cu
• testing/option_parser.cu
• testing/sample_count_criterion.cu
• testing/stdrel_criterion.cu