Repository navigation
RFC: Should main be patched to latest datafusion periodically? #1399
Description
A question came up in the discord server that would be useful to get wider audience on. We have two PRs that are nearly ready to merge, but they rely on features that will be in the next release:
The question is: Should we allow a [patch.crates-io] section in Cargo.toml?
Reasons I know of why we should patch on main:
- It will allow us to reduce the time between when DataFusion upstream release and when we release a Python version by keeping us more in line with changes upstream as they come.
- It will possibly reduce churn on developers who write a feature for the next release and then have to do rebasing on their PR to get it back to working when the upstream finally releases.
- It's nice for the developer to get all their work in faster while it's fresh in their mind.
Reasons I know of why we should not patch on main:
- It pushes additional work onto whomever wants to update to latest which involves unrelated changes as
mainupdates upstream. - Sometimes it creates additional churn as upstream changes are not always stable until the the release happens. There is definite potential for rework.
- We cannot release to crates.io with a patch section. It will be ignored. We would need to add a CI check to our release workflow to ensure there are no patches.
- We would need to ensure the patches point to a specific commit and that commit should be on
mainupstream. This isn't a reason against it per se, but it is something we would want to remain cognizant of.
Personally, I don't have a very strong opinion on which is the better route, which is why I'm requesting comments.
Activity
timsaucer commentedon Feb 26, 2026
Tagging @kosiew @mesejo @milenkovicm @nuno-faria @davisp in case you have opinions.
milenkovicm commentedon Feb 26, 2026
I don’t have a strong opinion @timsaucer but it seems like it would put more pressure on maintainers.
ion-elgreco commentedon Feb 26, 2026
If you make Python part of the core repo, you can always build on main :) downside, now everyone has that burden of maintaining the Python wheels, upside now everyone is also a maintainer of Python wheels 😂
nuno-faria commentedon Feb 26, 2026
I tend to agree with periodic patches, as I think this could reduce the lag between the major versions of
datafusionin Rust and Python.kosiew commentedon Mar 2, 2026
Thanks @timsaucer for the ping.
I lean towards not allowing patch.crates-io on
main. When I use it, it's mostly to test and validate local fixes.I suggest allowing it on
mainonly if:timsaucer commentedon Mar 2, 2026
Case in point: @nuno-faria has this nice PR to prepare for the next release: #1402
That's a big change. It would be nice to merge it in early so that we can stabilize any new work that goes in between now and when 53 releases. One pressure I feel frequently is community members asking when df-python will be available once the upstream repo releases. Sometimes we get the question on the same day! I know of at least 3 repos that use both datafusion internally and datafusion-python as well. These repos get held up on the delay.
We could go as @ion-elgreco suggests and move towards merging into the datafusion upstream repository. I suspect that would not be desired by many of the maintainers of the upstream repo to take on the additional responsibility, but I honestly don't know what the temperature of that room is.
ion-elgreco commentedon Mar 2, 2026
@timsaucer one argument would be that pushing out fixes earlier through the python community by building on main, has at least been beneficial in delta-rs to catch bugs early before they end up in a rust release