Repository navigation
Release DataFusion 48.0.1 #16486
Description
Activity
posting here @alamb's comment for transparency:
I think we can argue if that was a regression or not (it was a change of a default setting, though we are about to change it again in 49....)
That sounds like a valid point to me. Nothing has seriously broken, and if it's too slow, users can tweak the config.
Not sure if we want to fix the topk pushdown issues? Given that our own test suite is failing, can we sometimes produce incorrect results for users?
I think the topK stuff was not included in 48.0.0 so there is no need to backport any fixes
I volunteer to run the release process.
In terms of fixing the regression in clickbench performance (which is due to statistics not being gathered by default for dataframes) with I think we have two options
- backport the change of default value here: Set the default value of
datafusion.execution.collect_statisticstotrue#16447 (will align with what 49.0.0 release) - revert the change in Make
SessionContext::register_parquetobeycollect_statisticsconfig #16080 which made the defaults consistent
- backport the change of default value here: Set the default value of
I think this bug is enough to make me want to do s 48.0.1 release
I will begin work on such a release later today
Reacted by Dmitrii Blaginin- changed the title
[-][Potentially] Release DataFusion `48.0.1`[/-][+]Release DataFusion `48.0.1`[/+]on Jul 2, 2025 The process will be:
- We will create backport PRs for fixes, following the instructions here https://github.com/apache/datafusion/blob/main/dev/release/README.md#how-to-backport-add-changes-to-branch--branch
- I'll plan to make an Release Candidate from
branch-48in the next day or two
I will handle backporting
- fix: column indices in FFI partition evaluator #16480
- Make
SessionContext::register_parquetobeycollect_statisticsconfig #16080 - Fix parquet filter_pushdown: respect parquet filter pushdown config in scan #16646
If anyone else has fixes or changes they would like to include in 48.0.1, please create a backport PR (targeting the
branch-48branch)Reacted by Dmitrii Blaginin@xudong963 do you have additional suggestions for issues that should be backported?
I have made some backport PRs:
- [branch-48] Fix parquet filter_pushdown: respect parquet filter pushdown config i… #16656
- [branch-48] fix: column indices in FFI partition evaluator (#16480) #16657
I'll wait another day or two see if there is any more content needed for this release
Ok, I have backported the three identified issues and I will now create a release candidate
Reacted by Dmitrii Blaginin and Shehab AminWe have begun voting: https://lists.apache.org/thread/xobyfmpcvtcclcbgo0wz6c74b35mxcdx
@xudong963 do you have additional suggestions for issues that should be backported?
Sorry, just come back, lgtm.
Reacted by Andrew Lambreleased to crates.io and all is good
Reacted by Douglas Anderson, Shehab Amin and Dmitrii Blaginin
Previous release: #15771
Creating this issue to discuss if we want to do a patch release to
pushdown_filtersis enabled (starting in 48.0.0) #16588