Weekly Plan (Andrew Lamb) March 10, 2025 #15121

This is an attempt to organize myself and make what I plan to work on more visible

Weekly High Level Goals

• Complete board report: March 2025 ASF Board Report (March 12) #13713
• Make arrow release: Release arrow-rs / parquet minor version 54.3.0 (Mar 2025) arrow-rs#7107
• Complete draft blog post about the DataFusion Optimizer
• Get v1 of tree explain plans [EPIC] Complete SQL EXPLAIN Tree Rendering #14914 ready for wider review with @irenjj

(BTW if anyone wants to help with writing or releases that would be 🤗 )

Other projects I think are strategically important

These are the ones I plan to look at / review PRs for first

• Anything performance related (though really I am interested in things that help many/all DataFusion users)
• pushing expressions down into scans: Support Push down expression evaluation in TableProviders #14993
• Spark function feat: Add datafusion-spark crate #14392 (comment) (e.g. this with @shehabgamin)
• Productionizing / hardening external sorts: A complete solution for stable and safe sort with spill #14692 with @2010YOUY01

Weekly Stretch / Nice to have goals

• Figure out next steps for Document DataFusion Threading / tokio runtimes (how to separate IO and CPU bound work) #12393
• Figure out where next to improve our performance
• Try to move an example or two from datafusion-examples

Background

I am putting this list on github because:

1. I like how github renders checklists w/ PR titles so it is easy to track (I currently have a local text file...)
2. I thought others might be interested from seeing what I am doing / planning to do
3. It makes me feel better that I don't have time to review all the PRs 😭

The way I prioritize PRs is in the following order

1. Bug fixes
2. Documentation / UX / API improvements (things that make DataFusion easier/better to work with)
3. Performance improvements
4. New features with wide appeal
5. New functions

Note new features and functions are deliberately at the bottom