Repository navigation
Release DataFusion 50.3.0 (minor) #18072
Description
Is your feature request related to a problem or challenge?
- Follow on to Release DataFusion 50.2.0 (September 2025) Release DataFusion
50.2.0(minor) #17849
There are some things that we may way to backport into the 50.0.0 line, and this ticket tracks doing so.
Steps
- Update version and changelog
- Test with DataFusion Python
- Test with DataFusion Comet Release DataFusion
50.3.0(minor) #18072 - Test with delta.rs:
- Test Vortex
- Test with iceberg-rust:
- Test with LakeSail:
- Test with parquet viewer
- Test with datafusion-materialized-views
- Voting Thread
- Publish to crates.io
- Create ticket for next release:
Potential Content:
Minor Features
- feat: support Spark
concatstring function #18063 + chore: useNullBuffer::unionfor Sparkconcat#18087 @comphead
Bug Fixes
- AmbiguousReference on project following join when expression name matches left or right column name #17294 / Fix ambiguous column names in substrait conversion as a result of literals having the same name during conversion. #17299 @xbailey
- LimitPushPastWindows returns incorrect results for queries with
lead()#18028 / Fix bug in LimitPushPastWindows #18029 @avantgardnerio - Semantic equality check for DataType::Timestamp doesn't exist #17776 / Extend datatype semantic equality check to include timestamps #17777 @shivbhatia10
-
change_redundant_columnlossy approach breaks logical optimizer and physical planner #17405 / fix(SubqueryAlias): use maybe_project_redundant_column #17478 @notfilippo -
DateAdd/DateSubdo not perform overflow checks in release builds datafusion-comet#2539 / fix: Add overflow checks to SparkDateAdd/Sub to avoid panics #18013 @andygrove - Datafusion 50 Performance Regression (array_has style filter/join for Parquet data set) #18070
Backport PR list:
- [branch-50]: chore: cherry pick
concatto 50.3.0 #18128 - [branch-50] Fix ambiguous column names in substrait conversion #17299 #18077
- [branch-50] Backport Fix bug in LimitPushPastWindows (#18029) #18107
- [branch-50] Extend datatype semantic equality check to include timestamps (#17777) #18129
- [branch-50] fix(SubqueryAlias): use maybe_project_redundant_column (#17478) #18130
- [branch-50]: fix: Add overflow checks to SparkDateAdd/Sub to avoid panics (#18013) #18131
- [branch-50] perf: Fix NLJ slow join with condition
array_has(#18161) #18179
Metadata
Metadata
Assignees
Labels
No labels
Activity
[-]Release DataFusion 50.3.0 (minor)[/-][+]Release DataFusion `50.3.0` (minor)[/+]alamb commentedon Oct 15, 2025
FYI @xudong963 -- I can do this one too, but wanted to make sure you were aware
Thank you @hareshkh for filing the issue
comphead commentedon Oct 15, 2025
Thanks folks
concatstring function #18063Please allow me couple of hours to test it out
alamb commentedon Oct 15, 2025
Another potential candidate:
andygrove commentedon Oct 15, 2025
#18013 would be nice to have
avantgardnerio commentedon Oct 17, 2025
I created #18107 . Sorry for the delay!
xudong963 commentedon Oct 17, 2025
@alamb, I can be your backup in case you have something urgent at that time.
alamb commentedon Oct 17, 2025
I filed an issue about examples CI failing
Rust / cargo examples (amd64) (pull_request)failing on branch-50 #18135[branch-50] Backport Fix bug in LimitPushPastWindows (#18029) (#18107)
alamb commentedon Oct 19, 2025
#18070 turns out to be related to the new NLJ implementation, so we are not going to wait for that one to make the branch-50 release
[branch-50] fix(SubqueryAlias): use maybe_project_redundant_column (#…
[branch-50]: fix: Add overflow checks to SparkDateAdd/Sub to avoid pa…
[branch-50] Fix ambiguous column names in substrait conversion #17299 (…
[branch-50] Extend datatype semantic equality check to include timest…
10 remaining items
jackkleeman commentedon Oct 24, 2025
looks like I missed out on this one, but if we do another patch release before 51.0, it would be really good to backport #17577
xudong963 commentedon Oct 24, 2025
Sorry, I forgot to add you last release, just sent the invitation
alamb commentedon Oct 24, 2025
Thank you. I got it and successfully uploaded the remaining crates
Thanks again everyone!
alamb commentedon Oct 24, 2025
@jackkleeman -- indeed. I can run another release if you want, though I suspect we are getting pretty close to 51 too
alamb commentedon Oct 27, 2025
@colinmarc mentioned to me they had several things he would like backported too -- I would be willing to run another release if someone else makes a release ticket / does the backport prs, etc
jackkleeman commentedon Oct 27, 2025
if colin has some, i can have a go at making the ticket and the prs, but also happy to wait for 51
colinmarc commentedon Oct 27, 2025
This is our list (but also happy to wait for 51):
akhramov commentedon Oct 28, 2025
Uh, can I also please piggyback with #17688 if 50.4.0 takes place?
xudong963 commentedon Oct 28, 2025
FYI, I plan to start the 51.0.0 release next week
[branch-50] Prepare 50.3.0 release version number and README (apache#…
[branch-50] Prepare 50.3.0 release version number and README (apache#…
[branch-50] Extend datatype semantic equality check to include timest…
[branch-50] perf: Fix NLJ slow join with condition `array_has` (apach…
[branch-50] Prepare 50.3.0 release version number and README (apache#…