Repository navigation
Release DataFusion 43.0.0 #12470
Description
Activity
I'm keen to start the 43.0.0 release process as soon as we have upgraded to arrow-rs 53.1.0 since it will unblock apache/datafusion-ray#10
I'm keen to start the 43.0.0 release process as soon as we have upgraded to arrow-rs 53.1.0 since it will unblock apache/datafusion-ray#10
The arrow upgrade is all ready here: #12724
Since the arrow upgrade is a minor version, you won't have to wait for the datafusion upgrade to get the latest version of arrow-rs
Reacted by Andy GroveThanks @andygrove -- I think we should start planning. Here are some items I think we should include:
Required
- Bug fixes for metadata mismatches (@wiedld has a few more she will file over the next day or so): [EPIC] Schema metadata handling / bugs #12733
I would love to get these as well
- Implement special Groups for StringViews #12771 (should merge tomorrow)
- Performance: Add "read strings as binary" option for parquet #12788 (needs an arrow-rs upgrade which isn't scheduled until Nov: Release arrow-rs / parquet minor version
53.2.0(~November~ October 2024) arrow-rs#6341 -- though I could make a release sooner if you are willing to hold the DF release until next week)
Other than #12788 we are on track I think to be ready to release in the next day or two
BTW I am going to accelerate the timeline to release arrow 53.2.0 so we can potentially include it in the next datafusion release: apache/arrow-rs#6341
I hope to make the RC today
Reacted by Andy GroveThis came up on the sync call today
I would like consider doing this item before we release (as this check has caused significant pain downstream in InfluxDB 3.0 as well as delta-rs (see links on ticket):
If we did that I view everything else as nice to have, including
- [EPIC] Schema metadata handling / bugs #12733 (there are still some additional outstanding issues)
- Enable reading StringView by default from Parquet (
schema_force_string_view) by default #11682 as much as I would like to turn this on by default, I think it would be a bad idea to turn it on by default right before a release. It would be better to merge it to main and give it some bake time
There is a regression for CREATE TABLE #12864 (comment)
This should be resolved before release (or judged OK to break).There is a regression for CREATE TABLE #12864 (comment) This should be resolved before release (or judged OK to break).
Filed I have filed #13124 to track the
Reacted by Piotr FindeisenI am getting worried about the number of changes that are accumulating unreleased.
Here is my proposal:
- Add the "turn off the schema check" PR: Option to disable
Physical input schema should be the same as the one converted from logical schemaerror #13065 (any volunteers to help with this?) - Backport/make another 42.2 release with just that fix
- Create the 43 RC
Reacted by Andy Grove- Add the "turn off the schema check" PR: Option to disable
@andygrove what are your thoughts timingwise for 43? I feel like we are accumulating substantial API changes and it might be nice to release sooner rather than later
Do you need an assist with any portion of this?
Thank yoU @timsaucer ❤️
I am not sure -- I haven't heard from @andygrove -- perhaps we should ask him in Slack?
If he is willing, I think it would be great if you wanted to create the PR to update the versions and readme as documented in https://github.com/apache/datafusion/blob/main/dev/release/README.md#change-log
I am here (now). I can start the RC process this evening if we are not waiting on anything else
The release has been approved and published to crates.io
Minor follow on:
Tracking
44.0.0:Thanks again @andygrove
Is your feature request related to a problem or challenge?
Tracking ticket for next release, also a place to track desired inclusions
Last release was https://crates.io/crates/datafusion/42.0.0 September 17th, 2024 so next major release would be around October 20, 2024
Prior release tickets:
42.0.0#11902Desired Items that would be good to get into this release:
schema_force_string_view) by default #11682filter_pushdown) by default #3463Items to fix before release
CREATE TABLE AS SELECT... insertingVALUES#13124Physical input schema should be the same as the one converted from logical schemaerror #13065