Repository navigation
Release DataFusion 54.1.0 (minor/patch) Release #22547
Description
Is your feature request related to a problem or challenge?
- Part of [EPIC] DataFusion Releases #19783
- Follow on to Release DataFusion
54.0.0(Apr 2026 / May 2026) #21080
Let's plan a 54.0.1 or 54.1.0 patch release to fix any bugs / issues found in 54.0.0.
Steps:
- Update version and changelog
- Test with DataFusion Python
- Test with DataFusion Comet
- Test with delta.rs:
- Test vortex:
- Test with iceberg-rust:
- Test with LakeSail:
- Test with parquet viewer
- Test with datafusion-materialized-views
- Voting Thread: https://lists.apache.org/thread/7c9j2xc630oyo0xx8v108s2fwrqk9kcm
- Publish to crates.io
Potential Content:
Bug Fixes
- logical JoinNode proto missing null_aware field (follow-up to #19635) #22065 (backport [branch-54] fix: preserve null_aware on logical JoinNode proto round-trip (backport #22104) #22785)
- fix: Avoid panicing when stats are not available for a file group split #23277 (backport [branch-54] fix: Avoid panicing when stats are not available for a file group split (backport #23277) #23340)
- bugfix: changed return type of spark's width_bucket to i64 #22811 (backport [branch-54]: backport #22811 (bugfix: changed return type of spark's width_bucket to i64) #23087)
- fix:
array_compacthandle edge case with NULLs #23192 (backport [branch-54] backport #23192array_compacthandle edge case with NULLs #23196) - fix: handle
IS TRUEcorrectly inEliminateOuterJoin#22444 (backport [branch-54] fix: handleIS TRUEcorrectly inEliminateOuterJoin(backport #22444) #23580) - fix: preserve no-filter SMJ matches across pending outer batches #23049 (backport [branch-54] fix: preserve no-filter SMJ matches across pending outer batches (backport #23049) #23574)
- fix: NestedLoopJoinExec emits spurious unmatched-left rows with multiple probe partitions #22791 (backport [branch-54] fix: NestedLoopJoinExec emits spurious unmatched-left rows with multiple probe partitions (backport #22791) #23577)
- fix: regex simplification of anchored patterns produces wrong results #22727 (backport [branch-54] fix: regex simplification of anchored patterns produces wrong results (backport #22727) #23578)
- fix: approx_distinct over-counts for utf8view #22815 (backport [branch-54] fix:
approx_distinctover-counts for utf8view (backport #22815, adapted) #23576) - fix: isolate anonymous file statistics cache #22950 (backport [branch-54] fix: isolate anonymous file statistics cache (backport #22950, adapted) #23573)
- Fix:22477 any all schema error #22915 (backport [branch-54] fix:
= ANY (SELECT ...)/<> ALL (SELECT ...)schema error (backport #22915) #23575) - fix: Correctly compute nullability in recursive CTE schemas #22552 (backport [branch-54] fix: Correctly compute nullability in recursive CTE schemas (backport #22552) #23579)
- fix: don't duplicate volatile expressions when pushing projection into file scan #23395 (backport [branch-54] fix: don't duplicate volatile expressions when pushing projection into file scan (backport #23395, adapted) #23585)
- fix(lambda): only push referenced params into the merged batch #22853 (backport [branch-54] Fix evaluation of lambda with multiple parameters when some are unused (backport of #22853, adapted) #23660)
- Handle nulls in type coercion of higher-order UDFs, map_extract, spark array_repeat #23071 (backport [branch-54] Handle nulls in type coercion of higher-order UDFs, map_extract, spark array_repeat (backport #23071) #23629)
- Resolve lost wakeup in SpillPoolReader with multiple concurrent SpillPoolWriters #23522 (backport [branch-54] Resolve lost wakeup in SpillPoolReader with multiple conc… #23654)
Performance
- Skip loading Parquet page index when row-group statistics already prove it cannot prune #22857 (backport [branch-54] backport #22857 (Skip loading Parquet page index when row-group statistics already prove it cannot prune) #23088)
- perf: avoid intermediate slice allocation in Spark slice function #23481 (backport [branch-54] perf: avoid intermediate slice allocation in Spark slice function (backport #23481) #23582)
Other
- [branch-54] chore: fix cargo audit #23607 (fix
cargo auditdirectly onbranch-54, no backport needed)
Minor Features:
- TBD
Metadata
Metadata
Assignees
Labels
No labels
Activity
[-]Release DataFusion 54.1.0 (minor/patch) Release[/-][+]Release DataFusion `54.1.0` (minor/patch) Release[/+]mithuncy commentedon Jun 6, 2026
The backport for #22065 is up as #22785 (clean cherry-pick of #22104, targeting
branch-54). Could this be added to the 54.1.0 Bug Fixes list? Thanks!alamb commentedon Jun 8, 2026
Note that @mbutrovich also saw some performance concerns when testing 54.0.0 with Comet:
alamb commentedon Jun 8, 2026
Thanks @mithuncy -- added
[branch-54] fix: preserve null_aware on logical JoinNode proto round-…
gstvg commentedon Jun 22, 2026
There are two lambda-related fixes which I believe qualify to backporting after they got merged into main
#22853 by @Adam-Alani
#23071
mbutrovich commentedon Jun 22, 2026
I will open a backport for this today since it originated from Comet trying to upgrade to DF 54.0:
#22811
mbutrovich commentedon Jun 22, 2026
I will also try to backport #22857 since Comet also encountered this while trying to upgrade to DF 54.0.
mbutrovich commentedon Jun 22, 2026
Backport PRs:
#23088
#23087
comphead commentedon Jun 25, 2026
I would like to backport #23192, the PR fixes correctness issue, no breaking changes
#23196 for backport
mkleen commentedon Jul 6, 2026
Backport PR:
#23340
[branch-54] fix: Avoid panicing when stats are not available for a fi…
andygrove commentedon Jul 11, 2026
If we are open to back porting performance fixes, #23481 could potentially help Comet.
mbutrovich commentedon Jul 11, 2026
I think small blast radius performance PRs (easy to cherry pick) are fair game as long as there are no behavior or API changes. I’d approve such backports.
alamb commentedon Jul 12, 2026
I agree re performance PRs -- in my mind as long as there is little risk to stability backports are fine and should be largely driven by people's demonstrated interest (by creating backports)
21 remaining items
alamb commentedon Jul 15, 2026
@gstvg do you still think we should backport these? If so, would you be willing to make the backport PRs?
gstvg commentedon Jul 16, 2026
@alamb opened #23629 for #23071. #22853 now contains breaking changes, I'll try to adapt a non-breaking backport
[branch-54] Resolve lost wakeup in SpillPoolReader with multiple conc… (
gstvg commentedon Jul 17, 2026
@alamb I opened the adapted backport of #22853 at #23660
alamb commentedon Jul 17, 2026
Thanks @gstvg -- I took a look at that PR but I don't think it is ready to merge into branch-54 -- see #23660 (comment)
alamb commentedon Jul 17, 2026
@mbutrovich from my review I think we are ready to make a 54.1.0 release candidate. I suggest we don't hold the release for the resolution of #23660 as we already have many changes waiting
instead of blocking 54.1.0, we can schedule a 54.2.0 release if needed
mbutrovich commentedon Jul 17, 2026
Sound good, I will get the process going.
gstvg commentedon Jul 18, 2026
@alamb agreed on not blocking the release, 54.2.0 is perfecly fine for that backport
mbutrovich commentedon Jul 18, 2026
Version bump and changelog PR #23689.
mbutrovich commentedon Jul 18, 2026
Voting thread is now open: https://lists.apache.org/thread/7c9j2xc630oyo0xx8v108s2fwrqk9kcm
I will try to test with Comet and Iceberg Rust on Monday.
mbutrovich commentedon Jul 21, 2026
Crates are published, last PR to port the changelog and version number to
main#23764
chore: Update version 54.1.0, add changelog (apache#23689) (apache#23764
mbutrovich commentedon Jul 22, 2026
Changelog is in main, DataFusion Comet and Iceberg Rust have already successfully bumped (apache/datafusion-comet#4996 and apache/iceberg-rust#2872) so I am going to close off this issue. I filed a followup issue for 54.2.0 if we can start bringing any outstanding backport candidates to that issue.
#23805
chore: Update version 54.1.0, add changelog (apache#23689) (apache#23764