Repository navigation
[DISCUSSION] 2024 Q4 / 2025 Q1 Roadmap #13274
Description
Activity
- pinned this issue
on Nov 6, 2024 BTW my personal plans over the next few months are likely going to be focus on consolidating some of the gains / improvements we have made recently. That includes:
- External communication / documentation
- with @SamSynnada like Enhancing DataFusion's Community Engagement and Visibility #13049
- Blog post with DataFusion July - Sep 2024 #11631
Improve the project's documentation
- [EPIC] Improve examples to make them easier to navigate #11172
- Document DataFusion Threading / tokio runtimes (how to separate IO and CPU bound work) #12393
- [Epic] Better / Improved Documentation, Tutorials and Examples #7013
Performance wise I plan to
- help work on more advanced parquet predicate pushdown with @XiangpengHao Enable parquet filter pushdown (
filter_pushdown) by default #3463 - Continue to flesh out the grouping code: [EPIC] Improvements to GroupColumn multi-column aggregation performance #12680
Reacted by Jonathan Chen, kamille, Yongting You, Andy Grove, Xiangpeng Hao, Austin Liu, Oleks V, Douglas Anderson and Dmitri B.I am not sure if this is the place for it but I have been putting a lot of work into
dftand plan on doing a release before end of year.Reacted by Andrew Lamb, Rohith Reddy Kota, Bruce Ritchie, Oleks V, Tai Le Manh, Yongting You and Jax LiuI am not sure if this is the place for it but I have been putting a lot of work into
dftand plan on doing a release before end of year.For anyone else following along,
dftis https://github.com/datafusion-contrib/datafusion-dftI am not sure if this is the place for it but I have been putting a lot of work into
dftand plan on doing a release before end of year.I may want to help delta / iceberg integration, I think they are quite important. But I will work on performance task first
Reacted by Andrew Lamb, Dharan Aditya and Mike RitchieI may want to help delta / iceberg integration, I think they are quite important. But I will work on performance task first
@jayzhan211 I agree, they are very important. Unfortunately, we have been held up because of the crates using different versions of datafusion. The idea was to converge on 42 - which iceberg and hudi currently use but deltalake (which we already have an integration for) is on 41 and hasnt been able to upgrade yet. It looks like they are skipping version 42 now and will use 43 - so hopefully this is resolved soon.
Here is some relevant work
Reacted by Jay Zhan@jayzhan211 and to be more explicit on my release plans, i did not plan on releasing until iceberg and hudi were added.
LogicalType is important too #12622
Reacted by Jax Liu, Andrew Lamb, Dmitri B., Filippo and 张林伟I may want to help delta / iceberg integration, I think they are quite important. But I will work on performance task first
@jayzhan211 I agree, they are very important. Unfortunately, we have been held up because of the crates using different versions of datafusion.
I think another potentially very interesting approach here will be to use the
FFIbindings from @timsaucer:The idea there would be to wrap the delta / iceberge in a stable ABI (aka the FFI bindings) so we could call delta.rs / iceberg which used a different version of DataFusion from
dft.On the python side, getting better integration with the python delta-rs package was the entire reason for pushing for the FFI bindings. I have branches ready to go for
datafusion-pythonanddelta-rsas soon as 43.0.0 releases. I also have tested it with a few of the other table providers indatafusion-contrib.For the pure rust implementations, I think it would be best to not cross the
unsafeFFI boundary if you don't have to. Unfortunately that does put additional dependencies on the other crates updating at a reasonable pace.Reacted by Andrew LambI think as soon as DataFusion 43.0.0 is released we'll be able to test it out:
- Update
dftto DataFusion 43 - Implement a crate binding (in
dftto delta-rs with older datafusion version)
It should be quite sweet
- Update
I don't know if this discussion is the place we want to track work in the other related projects, but my top goals for 2024 Q4 are:
- Get the
datafusion-pythonanddelta-rsintegration over the line using FFI Add foreign table providers datafusion-python#921 - Adding a user tutorial for
datafusion-pythonRFC: Ideas for what to include in a user tutorial datafusion-python#842 - Evaluating cuDF integration either in
datafusion-pythonordatafusionSpike: evaluate if cuDF can be used with datafusion-python datafusion-python#936 - this might be Q1 2025 - Adding
QueryPlannertodatafusion-pythonand implementing a custom query planner indatafusion-rayso that we can run all datafusion commands that performexecute()on the dataframe and it just knows to how to run them distributed - Probably looking to expose more of the
SessionContextvia FFI
Reacted by Andrew Lamb, Jonathan Chen and Jay Zhan- Get the
One thing thats not clear to me with the FFI approach is who the intended owner of the bindings are - should it be
dftas i dont want to worry about my deps being on different datafusion versions or is it more for the table providers crates (iceberg, deltalake, hudi, etc)?Of course in the short term it could be prototyped in
dftand contributed back to those repos but im asking more in the target state where the appropriate home would be.One thing thats not clear to me with the FFI approach is who the intended owner of the bindings are - should it be
dftas i dont want to worry about my deps being on different datafusion versions or is it more for the table providers crates (iceberg, deltalake, hudi, etc)?Of course in the short term it could be prototyped in
dftand contributed back to those repos but im asking more in the target state where the appropriate home would be.The version of DataFusion used in the bindings has to match the client program (
dftin this case) so I don't think they can go in the delta/iceberg cratesOne thing we might be able to do is have a separate crate like
datafusion-delta-table-providerthat has different feature flags for different DataFusion versions 🤔 -- but I am now more or less wildly speculatingReacted by Matthew Turner@matthewmturner Do you have anything in mind moving forward for integrating the rest of the data lakes? (such as a list of what needs to be done moving forward)
@matthewmturner Do you have anything in mind moving forward for integrating the rest of the data lakes? (such as a list of what needs to be done moving forward)
Yes, now that DataFusion v43 has been released I am hoping that the rust implementations of the three main data lake formats (Deltalake / Iceberg / Hudi) update to that version. Then I will:
- Update the existing Deltalake integration
- Refresh the current Hudi PR
- Add Iceberg integration which should be pretty easy now that it implements
TableProviderFactory
I am interested in the FFI bindings but I don't anticipate working on that prior to the current release I am planning.
Reacted by Austin LiuReacted by Jonathan ChenI have had a few days to reflect , and I personally think making it easy to integrate DataFusion into the "open data lake" stack might be my top priority over the coming months
@julienledem wrote up a very nice piece descsribing this The advent of the Open Data Lake
In my mind, the specific work this entails stuff like
- Making it easier to use iceberg/delta/hudi with DataFusion
- Document different tokio runtimes
- Make parquet reader in arrow-rs faster/better on remote object stores
More to come
Reacted by Austin Liu, Jay Zhan, jjyeh, Bruce Ritchie and Mike RitchieReacted by Austin Liu and Eric FredineReacted by Austin Liu, Jonathan Chen, Yongting You, jjyeh, Dmitri B. and Landon Gingerich@alamb can you expand on the different runtimes point? Are referring to having a dedicated tokio runtime for CPU bound work? I actually have a ticket open for that - if that is a very important item for you I can add it to the list to do before releasing.
This is what I had in mind:
Thanks for the link to the dft one. That is a good one
@alamb i will work on that next. will ping you when ready for review.
Reacted by Andrew LambMore to come
I filed
to try and organize my thoughts here better
- added a commit that references this issue
on Nov 19, 2024 Cross Posting:
- Discussion about making it easier to use DataFusion [DISCUSSION] Making it easier to use DataFusion (lessons from GlareDB) #13525
- unpinned this issue
on Feb 10, 2025 I filed a ticket for the next quarter:
Is your feature request related to a problem or challenge?
The last roadmap discussion we had seems to have worked out well to galvanize and get us organized around some common goals
Describe the solution you'd like
Let's collect any projects that people think they are likely to spend time on or projects that the broader community would really like to see done and write them down!
Describe alternatives you've considered
No response
Additional context
No response