Repository navigation
Stabilize #[bench] and Bencher? #66287
Description
I’ll take the liberty of copying a section of @hsivonen’s Rust 2020 blog post:
Non-Nightly Benchmarking
The library support for the
cargo benchfeature has been in the state “basically, the design is problematic, but we haven’t had anyone work through those issues yet” since 2015. It’s a useful feature nonetheless. Like I said a year ago and the year before, it’s time to let go of the possibility of tweaking it for elegance and just let users use it on non-nighly Rust.
Indeed the existing benchmarking support has basically not changed in years, and I’m not aware of anyone planning to work on it. To keep reserving the right to make breaking changes is not useful at this point. Custom test frameworks offer another way forward for when someone does want to work on better benchmarking.
So I’d like to propose a plan:
- Move
test::Benchertostd::bench::Bencher.I have a PR coming soon thatMovetest::Bencherto a new (unstable)std::benchmodule #66290 demonstrates that this is possible. This move avoids the need to stabilize thetestcrate. - Optionally, make some surface-only API tweaks to
Bencher. For example, the publicbytesfield could become a parameter to some method.- Although not strictly necessary for changes to an unstable type, we can keep the existing
bytesfield anditermethod unchanged as unstable + deprecated for a while.
- Although not strictly necessary for changes to an unstable type, we can keep the existing
- Stabilize the
#[bench]attribute and just enough ofBencherto make it usable with#[bench]. (For example, no stable constructor.)
@rust-lang/libs, @rust-lang/lang, do you feel this needs an RFC?
Activity
Move `test::Bencher` to a new (unstable) `std::bench` module
Centril commentedon Nov 11, 2019
Given that "basically, the design is problematic" I'd say yes, we should have an RFC that justifies and explains the overall design (as if it was an unimplemented feature).
cc @rust-lang/dev-tools
alexcrichton commentedon Nov 11, 2019
I'd like to paste parts of this comment here:
I would ask again that the justification for stabilization has "it's not been touched for years" only taken as an input of whether to start the stabilization process, not an implicit ratification that the current design is "obviously good enough because no one has changed it".
There have been repeated attempts to stabilize the benchmarking infrastructure in the past and they've all been stymied because the libtest benchmarking support just isn't that great. It's great for quick and dirty things, but like
#[test], it does not have extensibility hooks, it's not always the easiest to use, etc. Stabilization often also gets quickly into the weeds of "what to do aboutstd::test?" or "what about custom test frameworks?" and such. I personally think it's very difficult to avoid these questions when proposing stabilization.Given the history of this feature I would personally expect an RFC to justify why the current design is suitable for stabilization (or the proposal here). Again, though, the RFC will need to justify its design irrespective of the amount of time the design has sat in
libtestfor now.Manishearth commentedon Nov 11, 2019
SimonSapin commentedon Nov 11, 2019
@Manishearth Yes, this proposal specifically does not involve opening a design discussion beyond small API-surface-only tweaks, so as to not divert efforts and limited team bandwidth from custom test frameworks or other work areas. (And I don’t feel up to personally leading such a discussion at the moment.)
Manishearth commentedon Nov 11, 2019
Oh, yeah, I agree, I actually think we should find a path for stabilizing bench itself at this point.
One thing I will point out is that a lot of people want some form of bench framework (and are likely willing to put in the work to design and stabilize just that), whereas far fewer people want custom test frameworks, and the work for that is proportionally larger.
I don't want to say that the justification for stabilization is "it hasn't been touched for years", but I do want to say that the fact that benching has been blocked on custom test frameworks has led to this annoying impasse where there just aren't enough interested people to do the huge amount of work to complete custom test frameworks. I'd still prefer if that happened, but that doesn't seem likely. I feel like as a community we have a tendency to fall into this trap, so I do see this issue as a push for a way out.
That said, I already opened an RFC for this and it got closed: rust-lang/rfcs#2287 . The reasoning there hasn't changed much, though I'd argue that the languishing of custom test frameworks is a valid argument to say that we should revisit this.
SimonSapin commentedon Nov 11, 2019
"It hasn't been touched for years" isn’t an argument for stabilization, it’s a sign that that more time as unstable is unlikely to lead to improvements to this benchmarking framework. Instead, I suspect that improvements will come as something entirely new that can be used through custom frameworks.
I think this leaves two realistic outcomes for
#[bench]:It is eventually removed, perhaps after alternatives become available through custom frameworks. The pFCP to close the previous RFC planned for this: Benchmarking / cargo bench rfcs#2287 (comment)
It is stabilized with its current design. People who are already happy enough with it can now use it on the Stable channel. It can then slowly fall out of use when something better comes up later. The Rust project is stuck with maintaining it "forever", but
#[bench]is very similar to#[test]that we’re already stuck with. I feel that the additional maintenance burden might be light enough that the benefit until an alternative is available may be worth it.I understand there’s not a lot of appetite for this and that we generally set the bar high, but perhaps we can live with something known not to be perfect.
gilescope commentedon Dec 10, 2020
As a random datapoint: I would love to see exercism.io students being able to do simple benchmarks on their solutions to see if one way is faster than another. I know criterion is the gold plated solution but it would be nice to have something that works out of the box on stable.
Don't let perfect be the enemy of the good: If measuring performance is easier, then more people will do it, lowering the CO2 impact that rust solutions have on the world.
Oliboy50 commentedon May 26, 2021
As a newcomer to Rust, I just wanted to say that it feels frustrating to learn that there is an awesome built-in benchmarking feature (which is something that I've never seen in the few languages I've worked with) but we "cannot" use it and doesn't seem like it will happen soon (since the following is written in the doc: The tracking issue for this feature is: None. )
Why is it even mentioned in The Book then? 😢
https://doc.rust-lang.org/book/ch11-01-writing-tests.html
Explain use of nightly clippy over whole monorepo
Explain use of nightly clippy over whole monorepo
Explain use of nightly clippy over whole monorepo (#31833)
nixpulvis commentedon Jul 18, 2025
So the eRFC for custom test frameworks seems to have stalled in #50297, and while I generally agree the same kinds of issues will present themselves with benchmarking and therefor should be considered together, I feel we are left in the interim with a stabilized
#[test]that has some deficiencies, and an unstabilized#[bench]with many of the same deficiencies. Both of which would be (and are) useful today.Wouldn't it be more consistent to provide both today and leave enhancements for a future edition if necessary?