Repository navigation
Release DataFusion 49.0.0 (July 2025) #16235
Description
Activity
I am thinking we should begin the 49.0.0 release next week. I'll publicize to the lists / etc
Reacted by xudong.wWe introduced new APIs in #16461 and more importantly removed the existing SchemaAdapter. I am not sure how upgrading other systems has gone but if they relied on custom SchemaAdapters they'll be broken. I think we should restore the default SchemaAdapter and make it possible to choose which path you use. To get there I think we need at least part of #15057 and to revert the change of removing SchemaAdapter in the original PR. I'm sorry if this causes a delay, I'll get to it ASAP on Tuesday.
Reacted by xudong.wWe introduced new APIs in #16461 and more importantly removed the existing SchemaAdapter. I am not sure how upgrading other systems has gone but if they relied on custom SchemaAdapters they'll be broken. I think we should restore the default SchemaAdapter and make it possible to choose which path you use. To get there I think we need at least part of #15057 and to revert the change of removing SchemaAdapter in the original PR. I'm sorry if this causes a delay, I'll get to it ASAP on Tuesday.
I'm okay to wait for these changes. It's always good to not break users.
Reacted by Andrew Lamb, Adrian Garcia Badaracco and Oleks VI'm okay to wait for these changes. It's always good to not break users.
Sounds good -- I added it to the checklist on the description of this ticket
I'm okay to wait for these changes. It's always good to not break users.
We're trying to minimize breakage! Thanks to you and everyone else for your your patience as we make these exciting but difficult improvements.
xpost: #15057 (comment)
I am hoping to start with some of the various release tasks tomorrow (like ensuring hte upgrade guide is in a good place) but I have been very busy with other things recently. Hopefully we can sort out #16791 soon and get the relese process started
Almost missed this thread. Will start testing when there is a
49branch!I am still behind -- i will likely start in earnest on this task tomorrow (Friday) and over the weekend and hopefully we'll be reado to cut a Release Candidate (RC) early next week
15 remaining items
Tested on Sail with commit afb9099. Everything looks good!
Reacted by Andrew Lamb and Andrew DuffyTesting on Comet, seeing some test failures
UPD: Moved to the latest DF RC branch and tracking progress apache/datafusion-comet#2040
One weird thing is the correctness issue on the queries with first/last with ignore null flag.
I can't find if Datafusion had changes for
ignore nullflag in 49 release 🤔UPD2: Looks like the cause is somewhere in
aggregate.rsorfirst_last.rsUPD3: @alamb I think it is good to go, the reason is the
ignore nullsresets the value if the grouping key lives in 2 separate parquet files. Same scenario works fine in DF, I assume some logic was moved from physical layer to logical causing Comet correctness issues when usingFirstValuefunction. I'm still investigating it, but again the same scenario works fine in DFUPD4: @alamb @ozankabak @berkaysynnada @mbutrovich Filed a PR https://github.com/apache/datafusion/pull/16918/files, appreciate for thoughts
Reacted by Andrew LambTest MV repo with DF49, datafusion-contrib/datafusion-materialized-views#75. Looks good
Reacted by Andrew LambI run into some compile issue in parquet viewer, still investigating...
Update 1: I have to disable the
encryptionfeature in Parquet to make it work: https://github.com/apache/datafusion/blob/main/Cargo.toml#L162It seems that it's not happy with
ring, but I also saw ring already have some special handling for wasmI ran my test suite against df 49, all passed. I did have to make an update for
TableParquetOptionsthat wasn't listed in the upgrade doc (new crypto field).Update 1: I have to disable the
encryptionfeature in Parquet to make it work: https://github.com/apache/datafusion/blob/main/Cargo.toml#L162Thanks @XiangpengHao -- do you think we should disable the crypto feature by default?
Update 1: I have to disable the
encryptionfeature in Parquet to make it work: https://github.com/apache/datafusion/blob/main/Cargo.toml#L162Thanks @XiangpengHao -- do you think we should disable the crypto feature by default?
@aoli-al figured out that this is an issue from my nix flake build, not a datafusion issue: XiangpengHao/parquet-viewer#50
(but I only see this error starting df49)
Now all tests passed with parquet viewer!
Reacted by Andrew LambThanks @XiangpengHao -- do you think we should disable the crypto feature by default?
It sounds like this is sorted but let me know if you'd like it disabled by default for the release. Corwin is on holiday but I can help with that if needed.
Reacted by Andrew LambThe 49.0.0 is released, and I've published it to crate.io
Reacted by Ian Wagner, Andrew Lamb and Adam GutglickThank you very much @xudong963 and everyone else who helped
I hope to publish the blog tomorrow:
- added a commit that references this issue
on Jul 28, 2025 - We are discussing a 49.0.1 patch: Release DataFusion
49.0.1(patch) #17036
- We are discussing a 49.0.1 patch: Release DataFusion
Is your feature request related to a problem or challenge?
Tracking ticket for next release, also a place to track desired inclusions
Previous release will be https://crates.io/crates/datafusion/47.0.0 (June, 2025) so next major release would be late June or Early July 2025
Steps:
49.0.0, add changelog #1682249.0.0delta-io/delta-rs#360350.0.0(Aug/Sep 2025) #16799main: [main] Update version to 49.0.0, add 49.0.0 changelog #16855TODOs
Prior release tickets:
48.0.0: Release DataFusion48.0.0(June 2025) #15771Changes to add to upgrade guide
These PRs made changes that deserve a mention in the upgrade guide
Features to mention in the blog (if they make it)
Bugs that need to be fixed
datafusion_expr::window_state::WindowAggState::update#16308pushdown_filtersis enabled (starting in 48.0.0) #16588Bugs that would be good to fix / investigate
49.0.0(July 2025) #16235 (comment)Community Wishlist
Date32to string given timestamp specifiers #15361