Repository navigation
DataFusion weekly project plan (Andrew Lamb) - Jan 8, 2024 #8786
Description
Follow on to Jan 1, 2024
Boilerplate Overview
The idea of this ticket is make my plans for DataFusion visible, largely for my own personal organizational needs, but also to:
- Try some different ways to communicate / coordinate in the community
- Help provide an interesting summary of what is happening in DataFusion this week
It would be great if anyone else who has plans like this for DataFusion could try to make them visible somehow as well 🙏 (feel free to copy / modify the format)
My (personal) plans for this week
- Work on more advanced pruning predicates Don't error on unknown column when pruning if predicate can still be proven false #7869 (comment)
- Work on a blog post about DataFusion: [Website]: DataFusion 26-34 blog post arrow-site#457
- Prepare next steps for making function packages more modular [EPIC] Unify Function Interface (remove
BuiltInScalarFunction) #8045
Project Queue (list of future projects)
- Statistics: Epic: Statistics improvements #8227 (specifically, Introduce a way to represent constrained statistics / bounds on values in Statistics #8078)
Projects I plan to prioritize reviewing / helping
- Memory explosion in HashJoin outputs:
HashJoinStreammemory tracking insufficient #7848 from @korowa - Improve planning speed [Epic] A collection of issues to improve planning performance / speed / efficiency #5637 with @matthewmturner
- Making function packages more modular: [EPIC] Unify Function Interface (remove
BuiltInScalarFunction) #8045 (looking for help)
Algorithm for (my) prioritizing PR reviews
Note there are many committers who can and do review and merge PRs, so this is not the priorities of the project as a whole, just the approximate algorithm I am using
Priority:
- Bug fixes (where something is just incorrect), especially regressions (where it used to work and now does not)
- Improvements directly related to features needed for InfluxDB (my employer)
- Documentation and test improvements (I view these as very strategically important)
- PRs that I think are strategically important
- Other new features / additions to functionality
The current strategically important projects in my head are:
- Making function packages modular and easier to mix/match: [EPIC] Unify Function Interface (remove
BuiltInScalarFunction) #8045 - Getting started guide for new users (who want to use DataFusion in their project)
- Improving performance of aggregation - especially high cardinality groupings, where we still lag state of the art by 2x
Thus, if you are interested in contributing to DataFusion and are interested in a fast turn around time I would recommend looking into bug fixes / test improvements / documentation / etc.
If you propose adding new functionality, the review cycle will likely be longer. You can make it a shorter cycle by looking at the comments on other recent PRs and following the same model (e.g. ensure there are tests in sqllogictest for example, the CI passes, includes documentation, etc)
Activity
alamb commentedon Jan 8, 2024
DataFusion
0.9.0and arrow50.0.0#8758expr.rsto sqllogictests. Part1 #8773matthewmturner commentedon Jan 8, 2024
@alamb FYI I am away for a few days and then will get back into planning performance work.
Let me know if there's any area in particular you would like to tackle next.
matthewmturner commentedon Jan 8, 2024
I had in mind checking for more places to replace results with boolean and then seeing if I could extract some of the optimization ideas from @zeodotr on #7698. Or are you keen to make the DFSchema cheaper to copy?
alamb commentedon Jan 8, 2024
I think that is the best idea. I would personally suggest focusing on the planning benchmarks and trying to improve whatever they show as the tall pole
matthewgapp commentedon Jan 9, 2024
@alamb just bumping the recursive CTE PR :) #7581
I fixed an issue that was raised and brought it up to date with main.
alamb commentedon Jan 9, 2024
DataFusion
CompressionTypeVariantencoding in protobuf #8785array_resize#8744alamb commentedon Jan 9, 2024
Thank you @matthewgapp -- I keep hoping someone else can help with that review too that has an interest in recursive CTEs. I'll try and take a look later today
alamb commentedon Jan 10, 2024
DataFusion:
maruschin commentedon Jan 11, 2024
@alamb hi, сould you suggest an area where to direct my attention. I spend a lot of time looking for issue to start working on.
alamb commentedon Jan 11, 2024
DataFusion
array_resize#8744alamb commentedon Jan 11, 2024
Thanks @maruschin !
If you have time, looking into the bug report here #8819 would be super helpful
Debugging this report #8702 would also be helpful (as would in general making the datafusion-cli output generation code simpler)
There is also supporting partitioned writing that seems like it would be great to implement: #8493
alamb commentedon Jan 12, 2024
Review Queue:
DataFusion
group_by.rsto sqllogictest #8834CASE WHENclause #8833NTH_VALUEaggregation, make reverseARRAY_AGGmore efficient #8841Transformedenum #8835array_resize#8744alamb commentedon Jan 14, 2024
Done in #8864