Weekly Plan (Andrew Lamb) March 3, 2025 #14978

This is an attempt to organize myself and make what I plan to work on more visible

Weekly High Level Goals

• Help @xudong963 to get Release DataFusion 46.0.0 #14123 ready (mostly testing / herding bugs)
• Complete a sql-parser-rs release Release sqlparser-rs version 0.55.0 datafusion-sqlparser-rs#1671
• object_store releaase: Release object_store 0.12.0 (API breaking) Around Feb 30 2025 arrow-rs-object-store#28
• write a new "this week in datafusion": March 4, 2025: This week(s) in DataFusion #15005

(BTW if anyone wants to help with writing or releases that would be 🤗 )

Other projects I think are strategically important

These are the ones I plan to look at / review PRs for first

• Anything performance related (though really I am interested in things that help many/all DataFusion users)
• Productionizing / hardening external sorts: A complete solution for stable and safe sort with spill #14692 with @2010YOUY01
• Pretty tree explain plans [EPIC] Complete SQL EXPLAIN Tree Rendering #14914 (thanks to @irenjj for starting us off!)
• Splitting out datasources with @AdamGS and @logan-keede [Epic] Split datasources out from datafusion crate (datafusion/core) #14444
• pushing expressions down: Support Push down expression evaluation in TableProviders #14993

Weekly Stretch / Nice to have goals

• Figure out next steps for Document DataFusion Threading / tokio runtimes (how to separate IO and CPU bound work) #12393
• Figure out where next to improve our performance
• Try to move an example or two from datafusion-examples

Background

I am putting this list on github because:

1. I like how github renders checklists w/ PR titles so it is easy to track (I currently have a local text file...)
2. I thought others might be interested from seeing what I am doing / planning to do
3. It makes me feel better that I don't have time to review all the PRs 😭

The way I prioritize PRs is in the following order

1. Bug fixes
2. Documentation / UX / API improvements (things that make DataFusion easier/better to work with)
3. Performance improvements
4. New features with wide appeal
5. New functions

Note new features and functions are deliberately at the bottom