Repository navigation
Blog post about TopK filter pushdown #15513
Description
Is your feature request related to a problem or challenge?
@adriangb has a great PR to add dynamic filtering with TopK in DataFusion here:
Describe the solution you'd like
It would be really nice to write a blog post about this optimization in DataFusion (and maybe call out how it works)
Here are some related work
- Snowflake blog: Reimagining Top-K Aggregation Optimization at Snowflake
- Dynamic Filters in Starburst,
High level points
- Study DuckDB plan Make ClickBench Q23 Go Faster #15177 (hos much faster)
- Went back and forth to design something that could be general purpose
- Did benchmarking,
- Results are amazing, etc
Describe alternatives you've considered
No response
Additional context
No response
Activity
aaryyya commentedon Apr 6, 2025
take
alamb commentedon Apr 6, 2025
Thanks @aaryyya -- note we'll need to actually finish the work before we can publish a blog
Of course, blog driven development does work pretty well -- it is basically we @clflushopt @scsmithr and I did with another project (wrote the outline of the blog we wanted to see and then wrote the code to make it happen 😆 )
adriangb commentedon Jun 18, 2025
I think we're ready to publish a blogpost!
alamb commentedon Jun 18, 2025
Now all we need to do is find time to write one 😆
I think a great visual is to have is some chart showing the performance improvement with/without topk pushdown
Basically like this https://arrow.apache.org/blog/2023/08/05/datafusion_fast_grouping/
alamb commentedon Jul 2, 2025
Here is my suggestion for a blog / outline:
The goal is a technical evangelism piece. The reader should come away having learned something about columnar query engines (not just that DataFusion is great, which it is!)
Title: Using Dynamic Filterers and to make TopK / LIMIT queries much faster
Structure:
Intro
w/ some sort of summary performance chart
Running example:
A simple example query -- I think the clickbench Q23
SELECT * FROM hits ORDER BY time DESC LIMIT 10is a pretty good one as it is so simple but illustrates the point. More details can be summarized from #15177Background
Show the plan for Q23
Explain the existing topk optimization (that there is a heap)
Explain that the query does much more work than necessary because it decodes all rows just to throw all but 10 of them away
Introduce the notion of filter pushdown and point out that DataFusion does it at multiple phases
pushdown_filtersis on)Dynamic Filters
Explain that the topk operator knows the minimum time that could be emitted after the plan started -- basically like
WHERE time > (current min in top k).However the current min isn't know at plan time
Then describe the summary technical approach, highlighting that you made it general purpose to aslo support SIPs and other user defined dynamic filters; Also highlight you worked with the community to do this
Results
Show some sort of results if possible
Conclusion / Call to action
This will be released in DataFusion 49
Come help / join us / use DataFusion 🎣
adriangb commentedon Aug 17, 2025
I'd also like to list the "next steps" to 🎣 contributors:
adriangb commentedon Aug 19, 2025
Initial draft is up! apache/datafusion-site#102
aaryyya commentedon Oct 16, 2025
as this feature is complete is this issue still open for documentation?
adriangb commentedon Oct 16, 2025
If there are updates or improvements you'd like to make this blog post or DataFusion's documentation it's very welcome, just make a new issue / PR!
alamb commentedon Oct 16, 2025
The final URL is here: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/