Skip to content

Automated way to run benchmarks on a dedicated machine from PRs #18115

Closed
Feature
Closed
Automated way to run benchmarks on a dedicated machine from PRs#18115
Feature
Labels
enhancementNew feature or request

Description

@alamb
alamb
opened on Oct 17, 2025

Is your feature request related to a problem or challenge?

As @rluvaton says in #17979 (comment), the current benchmarking situation in DataFusion is not ideal

There are too much noise, making the benchmark results unreliable, can your script also print the machine you are working on (like a1.metal or c5a.large so it will be easier to understand the properties of the machine that it is running on)

also, is your machine a dedicated bare metal?

We have many benchmarks (bench.sh, cargo bench etc) which can be run by contributors. However they suffer from two problems

  1. Not widely accessable: It requires each contributor to run the scripts on their own machines
  2. Results are noisy: due to machines that are often used are either laptops or cloud VMs which are noisy environments due to thermal throttling and/or other workloads on the same machines

For example, I have a google cloud VM that I use for benchmarks (scripts in my datafusion-benchmarking repo, which I run on PRs at author request or part of my own review. This suffers from the same problems of inaccessibility and noisy results

Describe the solution you'd like

I would like some way for community members to be able to run benchmarks on machines that is

  1. Automated (non manual) to request
  2. Relatively stable (e.g. on bare metal or other similar predictable hardware)

Describe alternatives you've considered

@rluvaton offers the following suggestion on #17979 (comment):

it would be great if we could have like in node.js and just allow to run the specific benchmarks on dedicated machines

it will make sure that all benchmarks across PR are running on the same machine with the same config making the benchmark result more reliable

Example of how node.js does it:

image

Additional context

Related to

Node.js Benchmark CI

Activity

  1. alamb
    added
    enhancementNew feature or request
    on Oct 17, 2025
  2. alamb
    added the
    Feature
    issue type on Oct 17, 2025

  3. alamb commented on Oct 17, 2025

    @alamb
    alamb
    on Oct 17, 2025
    ContributorAuthor

    @rluvaton is there some link that describes the benchmarking setup for node.js?

  4. alamb commented on Oct 17, 2025

    @alamb
    alamb
    on Oct 17, 2025
    ContributorAuthor

    We also had a previous attempt to provide a "benchmark" command directly from PRs

    These scripts ran the benchmarks on github action workers which were even more noisy than normal cloud VMs

    Challenges I know of:

    1. Apache does not provide access to bare metal servers (so someone would have to contribute them)
    2. I am not sure how authentication / authorization would work (who could request / trigger benchmark runs?)
  5. alamb
    changed the title [-]Dedicated machine / setup for running benchmarks[/-] [+]Automated way to run benchmarks on a dedicated machine from PRs[/+] on Oct 17, 2025

  6. rluvaton commented on Oct 17, 2025

    @rluvaton
    rluvaton
    on Oct 17, 2025
    Member

    @rluvaton is there some link that describes the benchmarking setup for node.js?

    The Node.js build group which contain the various ansible playbook,

    The ansible inventory with the all the machines including benchmark ones:

    manual steps for setting up benchmark machines

    the script that run the benchmark is in:

    which is less relevant for us as we will use criterion or something else

    The shell script to execute in jenkins given the parameters is: 
    #!bash -xe

# Validate TARGET is set and looks like a SHA.
if ! echo "${TARGET}" | grep -qE '^[0-9a-fA-F]+$'; then
  echo "Error: TARGET does not look like a SHA"
  exit 1
fi

# this is needed by select-compiler
export nodes=benchmark-ubuntu2204-intel-64

# start from a known state
rm -rf node || true
rm -rf performance || true
rm -rf bench || true
rm -rf *.csv || true
export startDir=`pwd`

# get the additional files neeed for some benchmarks
git clone https://github.com/nodejs/performance

# Currently all benchmark machines are identical. Move to Ansible if this changes.
export CPUSET=0-11
. performance/benchmarks/community-benchmark/run.sh

    You can ask in nodejs build slack channel the link is in here

  7. rluvaton commented on Oct 17, 2025

    @rluvaton
    rluvaton
    on Oct 17, 2025
    Member

    It seems like that benchmarking pipeline is not in Jenkinsfile but was created in the UI

    this is the configuration with some parts trimmed Image

  8. alamb commented on Nov 30, 2025

    @alamb
    alamb
    on Nov 30, 2025
    ContributorAuthor

    I hacked up some basic scripts to run benchmarks on PRs via comments and a random GCP VM I have running

    Here is the documentation: https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/README.md

    TLDR is To request a benchmark run on a PR, add a comment to the PR with the following format:

    run benchmark <benchmark_name>

    To run the "standard" benchmarks run

    run benchmarks

    The available users and benchmarks are currently whitelisted. Initially, I added @rluvaton @alamb @Dandandan and @adriangb -- if it turns out to be widely used, I'll work on expanding the capacity and number of users who can run this

    Here is an example:

  9. Tpt commented on Jan 1, 2026

    @Tpt
    Tpt
    on Jan 1, 2026
    Contributor

    For cargo bench with criterion https://codspeed.io/ is very convenient: it instruments the benchmarks in Github CI using Valgrind and tracks changes in a nice UI. Happy to do a setup MR if useful.

  10. alamb commented on Jan 1, 2026

    @alamb
    alamb
    on Jan 1, 2026
    ContributorAuthor

    For cargo bench with criterion https://codspeed.io/ is very convenient: it instruments the benchmarks in Github CI using Valgrind and tracks changes in a nice UI. Happy to do a setup MR if useful.

    I have also heard good things about codspeed and would love to try it possible. Some potential issues:

    1. We don't have direct access to github admin capabilities (ASF infra does) so if we need to authorize apps etc that can be a non trivial process
    2. There are many benchmarks for DataFusion and effectively analyzing them is the hardest part

    If you want to try something out, I recommend starting with one benchmark, for example sql_planner

    cargo bench --bench sql_planner

    And seeing how that goes

  11. Tpt commented on Jan 1, 2026

    @Tpt
    Tpt
    on Jan 1, 2026
    Contributor

    Thanks! Will give it a try! OpenDAL has already setup then removed codpseed (apache/opendal#5626), I guess because the instrumentation is not always stable, especially when syscall are involved (I never got real issues in oxigraph outside of when syscalls are involved).

  12. andygrove commented on Jan 2, 2026

    @andygrove
    andygrove
    on Jan 2, 2026
    Member

    I have been working on implementing automated benchmarks for Comet.

    The approach I am using does not use any ASF infrastructure or GitHub runners.

    It is a simple approach (IMO) of running k8s jobs that clone a particular branch, run benchmarks, then post the results in a comment on the PR.

    Example: apache/datafusion-comet#2999 (comment)

    I currently run this on a k8s cluster at my home, but this could run in the cloud if someone wanted to pay for that. I plan on restricting access to committers only. I am considering using AI to check that the PRs are not malicious.

    I would be happy to share the code, perhaps in the datafusion-benchmarks repo, if this is of interest (it was all written by Claude Code).

  13. Omega359 commented on Jan 4, 2026

    @Omega359
    Omega359
    on Jan 4, 2026
    Contributor

    I currently run this on a k8s cluster at my home, but this could run in the cloud if someone wanted to pay for that. I plan on restricting access to committers only. I am considering using AI to check that the PRs are not malicious.

    You might want to check out something like https://fireactions.io/ and run the benchmarks in a firecracker vm for additional security.

  14. adriangb commented on Mar 26, 2026

    @adriangb
    adriangb
    on Mar 26, 2026
    Contributor

    We've now been using https://github.com/adriangb/datafusion-benchmarking for a bit which runs on GKE Autopilot Performance class pods. The results seem pretty stable.

  15. alamb commented on Mar 27, 2026

    @alamb
    alamb
    on Mar 27, 2026
    ContributorAuthor

    I agree -- let's call this complete

  16. alamb
    closed this as completedon Mar 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    Feature

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions