Repository navigation
build: allow the whole DataFusion 54 line - #3033
Conversation
|
@CTTY @xanderbailey could you take a look when you have a chance? The exact 54.1.0 requirement means any project still on DataFusion 54.0.x cannot depend on iceberg-datafusion at all. This widens it to >=54.0.0, <55; the lockfile still resolves 54.1.0, so CI builds are unchanged. Happy to follow a different pinning policy if the integration crates have one. |
xanderbailey
left a comment
xanderbailey
left a comment
There was a problem hiding this comment.
Change makes sense to me. Still semver compatible
dannycjones
left a comment
dannycjones
left a comment
There was a problem hiding this comment.
The change seems reasonable, although I'm wondering what we may lose.
@mbutrovich bumped us from .0 to .1 in #2872. What was the motivation - testing? If so, should we have instructions to bump the DataFusion dependency in the lock file only in future?
| datafusion = "54.0.0" | ||
| datafusion-cli = "54.0.0" | ||
| datafusion-ffi = "54.0.0" | ||
| datafusion-sqllogictest = "54.0.0" |
There was a problem hiding this comment.
Perhaps we should switch to this, so we don't find ourselves in this situation again?
| datafusion = "54.0.0" | |
| datafusion-cli = "54.0.0" | |
| datafusion-ffi = "54.0.0" | |
| datafusion-sqllogictest = "54.0.0" | |
| datafusion = "54" | |
| datafusion-cli = "54" | |
| datafusion-ffi = "54" | |
| datafusion-sqllogictest = "54" |
There was a problem hiding this comment.
@dannycjones Taking the suggestion, thanks — "54" says the same thing and leaves nothing to bump.
On #2872: it reads as a routine follow-the-release bump, covered by existing tests — nothing needed 54.1 specifically. So yes, lockfile-only seems right for those.
plusplusjiajia
changed the title
|
I'm curious if we should follow the same guide for other crates as well (aka, using major version, but not patch version) |
+1. We should write them in AGENTS.md or add some ci check. |
6 participants
What changes are included in this PR?
datafusion = "54.1.0"resolves to^54.1.0, which excludes the 54.0.x line. Downstream projects that must stay on 54.0 therefore cannot depend oniceberg-datafusionat all.The concrete case: the
datafusion-fficapsule layout is not stable across minor versions, so a project embedding DataFusion next to adatafusion-pythonhost has to match its version — and PyPI'sdatafusion-pythoncurrently tops out at 54.0.This widens the requirement to
>=54.0.0, <55. The lockfile still resolves to 54.1.0, so CI builds are unchanged; only the range downstreams may resolve within gets wider. Verified thaticebergandiceberg-datafusioncompile and pass their consumer's tests against DataFusion 54.0.0.