Repository navigation
Blog for DataFusion 46.0.0 #15053
Description
Is your feature request related to a problem or challenge?
- We are in the final phases of releasing 46.0.0: Release DataFusion
46.0.0#14123
It would be great to write a blog post about the release
Describe the solution you'd like
I would like a blog about DataFusion 46, similarly to how @Omega359 wrote
Except I think it should be easier to write such a post as 46 is only 1 month of work, rather than 8months
Describe alternatives you've considered
In terms of content, I think we can draw from
Changelog: https://github.com/apache/datafusion/blob/branch-46/dev/changelog/46.0.0.md
Some suggested things to point out:
Upgrade Guide for help updating to DataFusion 46.0.0
Performance related:
feat: Add ScalarUDF support in FFI crate #14579 (timsaucer)
Better Error Messages
Quality improvements
Community Highlights
Additional context
FYI @oznur-synnada
Activity
alamb commentedon Mar 6, 2025
FYI @oznur-synnada perhaps you are willing to try writing this post?
berkaysynnada commentedon Mar 7, 2025
Yes, she is. I'm reaching out her now
oznur-synnada commentedon Mar 7, 2025
Yes, @alamb, noted. I'm going to take this one
alamb commentedon Mar 7, 2025
Thank you! I will love to help write it too! Just let me know!
berkaysynnada commentedon Mar 16, 2025
Hi @alamb. We are considering highlighting the following items:
Major Items:
Add related source code locations to errors by eliaperantoni
Introduce unified
DataSourceExecfor provided datasources, removeParquetExec,CsvExec, etc by mertak-synnadafeat: Add ScalarUDF support in FFI crate by timsaucer
StatisticsV2: initial statistics framework redesign by Fly-Style
Feature: AggregateMonotonicity by mertak-synnada and Window Functions Order Conservation -- Follow-up On Set Monotonicity by berkaysynnada
GSoC Acceptance of Datafusion
Minor Mentions:
feat: Implement UNION ALL BY NAME by rkrishn7
perf: Improve
medianwith no grouping by 2X by 2010YOUY01Improve performance 10%-100% in
FIRST_VALUE/LAST_VALUEby not sort rows inFirstValueAccumulatorby blagininAdd
rangetable function by simonvandeloptimize performance of the repeat function (up to 50% faster) by zjregee
Would you like to make any update to this list?
alamb commentedon Mar 18, 2025
Thanks @berkaysynnada
In general I suggest emphasizing things that many users of the crate will see / appreciate and mentioning, but not too deeply, things that developers of the crate care more about
I also suggest that until we enable a feature by default we should not make a big deal about it. The rationale is that most users won't use it until it is on by default, and we can make a big deal about it when it is available
I suggest emphasizing:
I would also mention that we are working on statistics / monotonicity, but until there is something major we can point at that is user facing (e.g. an important SQL query that now goes faster with the monotonicity changes) I probably wouldn't emphasize them
I think these ones are worth mentioning but don't make a huge deal:
DataSourceExecfor provided datasources, removeParquetExec,CsvExec, etc #14224These ones are in progress so I would suggest we wait t:
I would suggesting highlighting the performance improvements
medianwith no grouping by 2X #14399 (2010YOUY01)FIRST_VALUE/LAST_VALUEby not sort rows inFirstValueAccumulator#14402 (blaginin)uuidUDF (40x faster) #14675 (simonvandel)Speedup to_hex (~2x faster) Speedup
to_hex(~2x faster) #14686 (simonvandel)chrUDF (~4x faster) #14700 (simonvandel)Also I suggest pointing people at the new upgrade guide: https://github.com/apache/datafusion/blob/main/docs/source/library-user-guide/upgrading.md#datafusion-4600
And maybe we can refer people to https://github.com/apache/datafusion/blob/main/dev/changelog/46.0.0.md for more details
alamb commentedon Mar 26, 2025
Completed in
Published: https://datafusion.apache.org/blog/2025/03/24/datafusion-46.0.0/