Repository navigation
Blog post for the DataFusion 51.0.0 release #18548
Description
Activity
Not a very exciting topic, but if you want some info for the blog post on the
casework, here are some (unscientifically collected) numbers I got by running some queries on an SF10 lineitems parquet file.I can make some proper versions of the diagrams from #18075 (comment) if you want some more elaboration on what changed exactly in the implementation. In the end this is just memory shuffling reduction, so maybe "it's a bit faster now" is sufficient.
Just a case expression
select case l_returnflag when 'R' then date_part('year', l_shipdate) when 'A' then date_part('year', l_commitdate) when 'N' then date_part('year',l_receiptdate) else null end from lineitem;50.0.0: ~0.492 seconds
51.0.0: ~0.278 seconds*select * from lineitem50.0.0: 0.668 seconds
50.1.0: 0.668 seconds*+ a case expressionselect *, case l_returnflag when 'R' then date_part('year', l_shipdate) when 'A' then date_part('year', l_commitdate) when 'N' then date_part('year',l_receiptdate) else null end from lineitem;50.0.0: ~1.287 seconds ->
caseadds 0,619 seconds
51.0.0: ~0.850 seconds ->caseadds 0,182 secondsNot a very exciting topic, but if you want some info for the blog post on the
casework, here are some (unscientifically collected) numbers I got by running some queries on an SF10 lineitems parquet file.I am excited!
I can make some proper versions of the diagrams from #18075 (comment) if you want some more elaboration on what changed exactly in the implementation. In the end this is just memory shuffling reduction, so maybe "it's a bit faster now" is sufficient.
Thank you for the offer -- unless you already have one or would like to spend time writing up a more full features blog about how to optimize CASE evaluation, I think some unscientific numbers are sufficient
Speaking of a blog, I am not aware of any practical writeup of implementing optimized
CASE(conditional) evaluation in a vectorized query engine (aka like DataFusion). I would be willing to help write such a post, but if so I would want to make it a high quality one:- background of CASE / conditional
- Basic evaluation strategy
- Why it is hard to make CASE fast
- All the crazy stuff that we did to actually make it fast in DataFusion
I've placed a first draft written entirely by Claude AI at apache/datafusion-site#122 to get the ball rolling. It correctly captured the essence of the original issues and the changes the were made, but the post still needs quite some editing.
PR's in this release (approx based on dates) - https://github.com/apache/datafusion/pulls?q=is%3Apr+merged%3A2025-09-16..2025-11-08
Reacted by Andrew LambAnother source of data is https://github.com/apache/datafusion/blob/branch-51/dev/changelog/51.0.0.md
The blog post is now live: https://datafusion.apache.org/blog/2025/11/25/datafusion-51.0.0/
Is your feature request related to a problem or challenge?
51.0.0(Nov 2025) #17558Describe the solution you'd like
Let's tell the world what we have done since the DataFusion 50 release
Here is the last blog:
We can start with the features listed on the current release ticket (maybe there should be more)
50.0.0(Aug/Sep 2025) #16799Describe alternatives you've considered
No response
Additional context
No response