Repository navigation
DataFusion weekly project plan (Andrew Lamb) - Nov 13, 2023 #8151
Description
The idea of this ticket is to log plans I have each week for DataFusion related items, largely for my own personal organizational needs, but also trying to
- Try some different ways to communicate / coordinate int he community
- Help provide an interesting summary of what is happening in DataFusion this week
It would be awesome if anyone else who has definite DataFusion plans could try something like this (maybe tickets are the best place 🤔 )
Major items I plan for personally this week:
- Final week of "major" edits to SIGMOD paper (I plan to revise / improve diagrams and some wordig): Write DataFusion paper for (SIGMOD / VLDB / ICDE) #6782 with @ozankabak @Dandandan @yjshen and @JayjeetAtGithub @sunchao and @viirya
- Write up ideas of "growing and sustaining the DataFusion community"
- Improve representation of statistics to support better optimizatons: Introduce a way to represent constrained statistics / bounds on values in Statistics #8078
Projects I am watching / trying to help review and coordinate if needed
- Memory explosion in HashJoin outputs:
HashJoinStreammemory tracking insufficient #7848 with @korowa - Disconnecting ListingTable (object store) from streaming usecase, add
StreamingTablelike API Decouple Streaming Use-Case from File IO Abstractions #7994 with @metegenez and @ozankabak -
ARRAYfunction hardening: [Epic] General ticket for the concept of the practical implementation ofARRAY#6980 / https://github.com/apache/arrow-datafusion/issues/7988with @jayzhan211 @Veeupup @edmondop and @xudong963 (e.g. Minor: Supportnullsinarray_replace, avoid a copy #8054) - Make function packages more modular (make all BuiltInScalarFunctions ScalarUDFs): [EPIC] Unify Function Interface (remove
BuiltInScalarFunction) #8045 with @2010YOUY01 (get initial PR merged, help @2010YOUY01 lay the foundation, prepare first function extractions) - Sort based optimization work started in Combine Equivalence and Ordering equivalence to simplify state #8006 (encapsulating more structs, e.g Encapsulate
EquivalenceClassinto a struct #8034 and reviews) with @ozankabak and @mustafasrepo - Review any new ticket / bug reports
Metadata
Metadata
Assignees
Labels
No labels
Activity
alamb commentedon Nov 14, 2023
Also, help user defined types / LogicalType intoduction with @yukkit : #8143
alamb commentedon Nov 14, 2023
Wrote up #8152 as part of communty growth
edmondop commentedon Nov 15, 2023
I was considering this issue #7955 @alamb do you think it's something that makes sense given your plan?
alamb commentedon Nov 15, 2023
I think this is a complicated feature and unfortunately don't have time at the moment to help it through the process. Perhaps other contributors such as @Dandandan or @sunchao / @viirya would have time to help shepherd / work with you on the implementation.
edmondop commentedon Nov 15, 2023
any simpler, more immediate issues in mind? I tried to search issues tagged with good-first-issue label but many of them had "some history" so maybe there are some of them that come to your mind that are higher priority/more useful
https://github.com/apache/arrow-datafusion/issues?page=2&q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+label%3A%22good+first+issue%22
alamb commentedon Nov 15, 2023
Thanks for asking @edmondop -- I don't know of any that are great candidates at this time (It takes a non trivial amount of time to write them up, and then they get done very quickly :)) -- I will try and write up some more shortly.
I filed some more issues under #6195 that might be interesting to do if you have time and inclination
alamb commentedon Nov 20, 2023
#8274 for this week