Repository navigation
DataFusion weekly project plan (Andrew Lamb) - Nov 27, 2023 #8329
Description
The idea of this ticket is make my plans for DataFusion visible, largely for my own personal organizational needs, but also to:
- Try some different ways to communicate / coordinate in the community
- Help provide an interesting summary of what is happening in DataFusion this week
It would be great if anyone else who has plans like this for DataFusion could try to make them visible somehow as well
Plans for this week personally:
- sqlparser release: Release version
0.40.0datafusion-sqlparser-rs#1030 - Epic: Statistics improvements #8227 (specifically, Consolidate statistics aggregation #8229 as a step towards Introduce a way to represent constrained statistics / bounds on values in Statistics #8078)
- Final edits + draft submission for Write DataFusion paper for (SIGMOD / VLDB / ICDE) #6782 with @ozankabak @Dandandan @yjshen and @JayjeetAtGithub @sunchao and @viirya
- File follow on tickets towards making function packages more modular: [EPIC] Unify Function Interface (remove
BuiltInScalarFunction) #8045
My plans for next week
- FIFO
JoinHashMapforHashJoin#8130, in support of memory explosion in HashJoin outputs:HashJoinStreammemory tracking insufficient #7848 with @korowa
Major projects I intend to help review and coordinate
- Next steps towards function packages more modular (make all BuiltInScalarFunctions ScalarUDFs): [EPIC] Unify Function Interface (remove
BuiltInScalarFunction) #8045 with @2010YOUY01 -
ARRAYfunction hardening: [Epic] General ticket for the concept of the practical implementation ofARRAY#6980 / https://github.com/apache/arrow-datafusion/issues/7988with @jayzhan211 @Veeupup @edmondop and @xudong963 - Review any new ticket / bug reports
Algorithm for (my) prioritizing PRs
Note there are many committers who can review and merge PRs as well, so this is not the priorities of the project as a whole, just the approximate algorithm I am using
Priority:
- Bug fixes (where something is just incorrect), especially regressions (where it used to work and now does not)
- Improvements directly related to stuff needed for InfluxDB (as I am employed by them)
- Documentation and test improvements (I view these as very strategically important)
- PRs that I think are strategically important
- Other new features / additions to functionality
The current strategically important projects in my head are:
- Support User Defined Table Functions / Table Value Functions #7926
- [EPIC] Unify Function Interface (remove
BuiltInScalarFunction) #8045 - Improving performance of high cardinality aggregation
Note that adding new array functionality (e.g. #6980 and its children) is very low on my priority list, not because I don't think they are valuable, but because I think they are less valuable than the other projects going on
Thus, if you are interested in contributing to DataFusion and are interested in a fast turn around time I would recommend looking into bug fixes / test improvements.
If you propose adding new functionality, the review cycle will be longer. You can make it a shorter cycle by looking at the comments on other recent PRs and following the same model (e.g. ensure there are tests in sqllogictest for example)
Activity
alamb commentedon Nov 27, 2023
Arrow-rs
DataFusion
sqlparser-rs
syn 2.0datafusion-sqlparser-rs#1040alamb commentedon Nov 27, 2023
I am very backed up on reviews. I will try and work through them over the next few days
alamb commentedon Nov 28, 2023
DataFusion
aggregates.rsto sqllogictest #8316csv_files.rsto sqllogictest #8251DataFusion: As time permits (lower priority)
alamb commentedon Nov 28, 2023
I also want to make my algorithm for prioritizing PR reviews more visible, so adding it here (and will add to descriptions too). I want to point out we have many committers who can review and merge PRs as well, so this is not the priorities of the project as a whole, just the approximate algorithm I am using
Priority:
The current strategically important projects I am tracking are listed in the description of this ticket
Thus, if you are interested in contributing to DataFusion and are interested in a fast turn around time I would recommend looking into bug fixes / test improvements.
If you propose adding new functionality, the review cycle will be longer. You can make it a shorter cycle by looking at the comments on other recent PRs and following the same model (e.g. ensure there are tests in sqllogictest for example)
alamb commentedon Nov 29, 2023
Arrow
DataFusion
quoteandescapeattributes to create csv external table #8351 (review)DataFusion: As time permits (lower priority)
alamb commentedon Nov 30, 2023
DataFusion
LargeListinarray_empty#8321 (review)ntilework in some corner cases #8371 (review)DataFusion: As time permits (lower priority)
alamb commentedon Dec 1, 2023
DataFusion
expr::window_function::WindowFunctiontoWindowFunctionDefinition, make structure consistent with ScalarFunction #8382 (review)ScalarFunctionDefinition#8387array_ndimsto fix List(Null) handling #8320 (review)DataFusion: As time permits (lower priority)
array_replace#8337jackwener commentedon Dec 4, 2023
Thanks @alamb !
Many people have asked me how to get involved in the DataFusion community and what they can do for the project (Because I've previously promoted DataFusion on the Chinese social media). This summary is very helpful for them.
I have pinned this issue to make it easy for everyone to view.
Express gratitude once again for all that you have done for the community.
alamb commentedon Dec 5, 2023
Thank you for this feedback. It is very helpful. Other than writing up more detailed tickets, do you know of other things that would help?
I saw this -- that was very cool. Thank you. I will pin future such issues.
Thank you very much @jackwener - I am always amazed by all the work we do together.
alamb commentedon Dec 5, 2023
Next Week: #8420