Skip to content

Add a datafusion benchmark for filter_pushdown #16729

Closed

Description

@alamb

Is your feature request related to a problem or challenge?

In order to enable filter_pushdown by default, we need to ensure it doesn't regress existing performance

However, it has been very hard to make forward progress on improving filter pushdown because all our benchmarks compare filter pushdown to not filter pushdown, so the bar for change is quite high.
Here is the most recent example:

Describe the solution you'd like

I would like a benchmark to measure for improvements to the filter pushdown are comparing when filter pushdown is already on. However, we don't have any such benchmark

There is one existing benchmark called parquet but it seems not to capture real world filter pushdown usecases and is not often run (I have never seen or heard of anyone running it)

Describe alternatives you've considered

  1. Remove the old "parquet" benchmark (that isn't used)
  2. Add a new clickbench_filter_pushdown benchmark that runs the same clickbench queries with filter pushdown enabled

Additional context

No response

Activity

  1. changed the title [-]Add a benchmark for `filter_pushdown`[/-] [+]Add a datafusion benchmark for `filter_pushdown`[/+] on Jul 9, 2025
  2. self-assigned this
    on Jul 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions