Repository navigation
Release DataFusion 37.0.0 #9682
Description
Is your feature request related to a problem or challenge?
It has been ~4 weeks since the last release was published. so I plan on starting the release process for the next version later this week.
Describe the solution you'd like
Waiting on:
- Regression: Wrong result when there are 2 count(distinct) #9586
- Regression: incorrect result for aliased arguments in select list after TreeNode rewrite #9678
- Update Arrow/Parquet to
51.0.0, tonic to0.11#9613 - Support backwards compatible / un prefixed config options (
compressionrather thanparquet.compression) #9575 - Regression: All formatting options in COPY commands require
format.prefix, but did not in DataFusion 36.0.0 #9716 - COPY TO fails on passing options for format through cli #9707
- Rename
common_runtimedirectory tocommon-runtimefor consistency with other crate directory names #9696 - Parallel parquet writer can panic when max_row_group_size < execution.batch_size #9736
Describe alternatives you've considered
No response
Additional context
No response
Activity
alamb commentedon Mar 18, 2024
EPIC!
alamb commentedon Mar 19, 2024
I think we should hold this release for two regression bugs:
I expect solutions to both by the end of day today
leoyvens commentedon Mar 19, 2024
Any chance of including #9613? Seems desirable for DataFusion to tightly follow the latest arrow-rs dependencies.
alamb commentedon Mar 19, 2024
It makes sense to me. I'll go and get that PR ready for merge now
Update: merged
alamb commentedon Mar 19, 2024
Something else we may want to wait for is backwards compatible write configuration options
compressionrather thanparquet.compression) #9575 (I think @tinfoil-knight is working on this)format.prefix after Make COPY TO align with CREATE EXTERNAL TABLE #9604 (I am not sure there is a ticket tracking this yet)cc @metesynnada and @ozankabak and @devinjdangelo
alamb commentedon Mar 20, 2024
@wiedld has a PR up to fix #9678 ( #9685)
alamb commentedon Mar 20, 2024
I have added the list of things I think we should wait on to the description of this ticket
alamb commentedon Mar 23, 2024
I believe all outstanding items from my perspective are complete
alamb commentedon Mar 28, 2024
@andygrove unless you or someone else does it first, I plan to make a release candidate tomorrow, Friday March 29. 2024
alamb commentedon Mar 30, 2024
Vote thread is up: https://lists.apache.org/thread/lwdflob72q7t7mqqxnqcobjzqjbc218o
alamb commentedon Mar 30, 2024
@wiedld has found another regression in 37.0.0: #9870 (see details in #9870 (comment))
Update: I reviewed the PR to fix it #9871 and it looks good to go to me
alamb commentedon Apr 1, 2024
Has been merged
alamb commentedon Apr 1, 2024
RC2 voting thread: https://lists.apache.org/thread/mzjt0l4tzkm3y7hxsv8lk35x0xxq1q5b
comphead commentedon Apr 4, 2024
Are we still planning to release 37.0.0?
alamb commentedon Apr 4, 2024
I think we are now ready to go: the RC has been approved and 72 hours has elapsed.
https://lists.apache.org/thread/mzjt0l4tzkm3y7hxsv8lk35x0xxq1q5b
If @andygrove doesn't get a chance to publish to crates.io I can do it later today
alamb commentedon Apr 4, 2024
The release is available here:
https://dist.apache.org/repos/dist/release/arrow/arrow-datafusion-37.0.0
The release has also been published to crates.io:
https://crates.io/crates/datafusion/37.0.0
Thank you all for your help!
Andrew
alamb commentedon Apr 4, 2024
I made some small improvements based on my experience this time, but otherwise I think this is now done
alamb commentedon Apr 18, 2024
BTW we are making a 37.1.0 release -- see #9904
alamb commentedon Apr 18, 2024
Note the failing CI test (security_audit) is explained here #10128 (comment).