Repository navigation
Release DataFusion 48.0.0 (June 2025) #15771
Description
Is your feature request related to a problem or challenge?
Tracking ticket for next release, also a place to track desired inclusions
Previous release will be https://crates.io/crates/datafusion/47.0.0 (April, 2025) so next major release would be late May or Early June 2025
Steps:
- Update version and changelog: Prepare for 48.0.0 release: Version and Changelog #16238
- Test with DataFusion Python:
- Test with DataFusion Comet (chore: Prepare for DataFusion 48.0.0 datafusion-comet#1710)
- Test with delta.rs: chore: update to DataFusion
48.0.0/ arrow to 55.2.0 delta-io/delta-rs#3520 - Test with LakeSail: chore: Upgrade to DataFusion 48.0.0 lakehq/sail#518
- Write upgrade guide:
- Test with parquet viewer
- Test with datafusion-materialized-views (Upgrade DataFusion 48.0.0 datafusion-contrib/datafusion-materialized-views#61)
- Voting Thread: https://lists.apache.org/thread/b7r28j9bzk82cvgcoorxk2cz4c90lso1
- Create ticket for next release: Release DataFusion
49.0.0(July 2025) #16235
Prior release tickets:
Changes to add to upgrade guide
These PRs made changes that deserve a mention in the upgrade guide
- Feat: introduce
ExecutionPlan::partition_statisticsAPI #15852 - Change default SQL mapping for
VARCAHRfromUtf8toUtf8View#16142 - Reduce size of
Exprstruct #16207: Minor: Add upgrade guide forExpr::WindowFunction#16313
Features to mention in the blog (if they make it)
- Change mapping of SQL
VARCHARfromUtf8toUtf8View#15096 - feat: Add
datafusion-sparkcrate #15168 - feat: ORDER BY ALL #15772
- Reduce size of
Exprstruct #16207 - feat: Add
datafusion-sparkcrate #15168 - Introduce Async User Defined Functions #14837
- [EPIC] A collection of tickets for improving sorting larger than memory datasets / spilling sorts #15271
Bugs that would be good to fix
- Fix: fetch is missing in
replace_order_preserving_variantsmethod duringEnforceDistributionoptimizer #15808 - Fix: after repartitioning, the
PartitionedFileandFileGroupstatistics should be inexact/recomputed #15539 - Intermittent failures in CI in
test_files/limit.slt#16180
Activity
xudong963 commentedon Apr 21, 2025
I'll help to test DF48 by upgrading df in https://github.com/datafusion-contrib/datafusion-materialized-views
andygrove commentedon May 24, 2025
Since updating Comet to use latest DataFusion (pinned dependency), we have been seeing regular but intermittent CI failures that we are still trying to debug.
It may or may not be related to changes in DataFusion since 47.0.0 .. we do not know the root cause yet, but mentioning this here just in case others are seeing similar issues or have any thoughts on what may be happening.
apache/datafusion-comet#1786
alamb commentedon May 25, 2025
I am not sure if it is related, but we have also seen some intermittent failures in DataFusion CI
test_files/limit.slt#16180xudong963 commentedon May 28, 2025
@alamb, how about starting test next week?
alamb commentedon May 28, 2025
I think that would be a great idea. Thanks @xudong963
andygrove commentedon May 29, 2025
This issue was not related to upgrading to use the latest DataFusion.
xudong963 commentedon Jun 3, 2025
I plan to update version and changelog today and then test with MV tomorrow.
I hope we can start the vote process on Friday, and release early next week.
shehabgamin commentedon Jun 3, 2025
I plan to test over the next couple of days as well.
alamb commentedon Jun 3, 2025
I hope to begin working on the delta-rs upgrade tomorrow as well
xudong963 commentedon Jun 4, 2025
48.0.0 rc1 is ready: https://github.com/apache/datafusion/tree/48.0.0-rc1
andygrove commentedon Jun 4, 2025
Thanks @xudong963. I'll start testing this with Comet.
I don't seem to have received an email about the start of the vote for this release candidate.
xudong963 commentedon Jun 4, 2025
I was thinking of starting the vote after testing, maybe Friday. WDYT?
26 remaining items
xudong963 commentedon Jun 6, 2025
The voting process ⏳ https://lists.apache.org/thread/tqwcgz5567pwo8pfn2p2dvvd5wsbg9bx
timsaucer commentedon Jun 6, 2025
The one I was hoping to get in but didn’t make it before this vote started was #16170
This was the last of the breaking metadata changes. Without it we lose metadata when we operate on literals during optimization.
XiangpengHao commentedon Jun 6, 2025
No issue found from Parquet Viewer and LiquidCache!
andygrove commentedon Jun 6, 2025
Upgrading Comet to use rc2 causes tests to fail with a
attempt to subtract with overflowpanic. This did not happen with rc1. I have not debugged this yet to find the root cause.Tracking issue: #16308
andygrove commentedon Jun 6, 2025
I did confirm that reverting #16234 fixes the issue
alamb commentedon Jun 6, 2025
Update:
release-48: fix: [branch-48] Revert "Improve performance of constant aggregate window expression" #1630748.0.0/ arrow to 55.2.0 delta-io/delta-rs#3520So I think we need to make an rc3. Thank you for bearing with us @xudong963
I also plan to look at PR that @timsaucer and @paleolimbot suggest, but given that we are in the final testing phase unless it is without risk I would be inclined to postpone that until 49.0.0 (sorry 😭 )
alamb commentedon Jun 6, 2025
BTW I think we should cut any release candidate off the
release-48branch and main is now open for 49.0.0 featuresandygrove commentedon Jun 6, 2025
Here is the cherry-pick PR to backport #16170 into
branch-48. Once this is merged, we can cut rc3.#16315
xudong963 commentedon Jun 7, 2025
I'm cooking rc3!
xudong963 commentedon Jun 7, 2025
RC3 voting is out: https://lists.apache.org/thread/b7r28j9bzk82cvgcoorxk2cz4c90lso1
alamb commentedon Jun 11, 2025
We have released to crates.io so I think we are all good with this release
blaginin commentedon Jun 19, 2025
Do you think we need to do
48.0.1to include fix for #16444?alamb commentedon Jun 20, 2025
I think we can argue if that was a regression or not (it was a change of a default setting, though we are about to change it again in 49....)
I volunteer to do the release candidate / voting thing if others agree we should do it
@blaginin maybe you can make a new ticket to discuss the possibility of 48.0.1 explicitly?
Upgrade to DataFusion `48.0.1` (#2032)