Repository navigation
Project Ideas for GSoC 2025 (Google Summer of Code) #14478
Description
Context
Apache DataFusion is putting together an application for GSoC 2025. For those who do not know, GSoC is a Google-sponsored program for students passionate about open-source to come in and contribute to cool projects for a duration of 12 weeks over the summer. The main body of this issue will serve as a repository for project ideas for prospective students, should we become a mentoring organization.
Process
As an ex-GSoC student who has been through the process (time flies!), I will be coordinating our application to the program, and oversee the progress on behalf of the PMC. I do not know how many student slots we will get, but in addition to myself, @jayzhan-synnada and @berkaysynnada are volunteering to be mentors to GSoC students and help with their projects. If there is a need for more mentors, I will ask the committers and the PMC for help (and I'm sure we will get as many mentors as necessary 🙂 ), but we have many interesting project ideas and certainly enough people to mentor the students.
Idea Proposals
Let's use the comments section of this issue to discuss and ideate about possible 12-week projects for GSoC students. Note that our aim is to answer the following questions positively as we graduate students from the program:
- Did the student learn new skills for building cutting-edge data systems? This involves things like (a) improving their Rust proficiency, (b) extending their knowledge on schedulers and sync/async computations, (c) developing their ability to grasp and implement novel ideas from recent academic papers, and many other practical computer science skills.
- Was the student able to complete a non-trivial project that actually integrates with (and is useful within) Apache DataFusion?
- Will the student leave the program with enthusiasm to become an active member of the Apache DataFusion community, or at least of another open-source community?
Ideas
TODO: I will be adding some ideas here during the course of this week, and hope to draw inspiration from the community feedback. Let's go! 🚀
Here is a quick summary of the project ideas, along with links to more details:
| Project | Category | Difficulty Level | Possible Mentor(s) and/or Helper(s) | Skills | Expected Project Size |
|---|---|---|---|---|---|
| Implement Continuous Monitoring of DataFusion Performance | Tooling | Medium | @alamb and @mertak-synnada | DevOps, Cloud Computing, Web Development, Integrations | 175 to 350 hours* |
| Supporting Correlated Subqueries | Core | Advanced | @jayzhan-synnada and @xudong963 | Databases, Algorithms, Data Structures, Testing Techniques | 350 hours |
| Improving DataFusion DX (e.g. 1 and 2) | DX | Medium | @eliaperantoni and @mkarbo | Software Engineering, Terminal Visualizations | 175 to 350 hours* |
| Robust WASM Support | Build | Medium | @alamb and @waynexia | WASM, Advanced Rust, Web Development, Software Engineering | 175 to 350 hours* |
| High Performance Aggregations | Core | Advanced | @jayzhan-synnada and @Rachelint | Algorithms, Data Structures, Advanced Rust, Databases, Benchmarking Techniques | 350 hours |
| Improving Python Bindings | Python Bindings | Medium | @timsaucer | APIs, FFIs, DataFrame Libraries | 175 to 350 hours* |
| Optimizing DataFusion Binary Size | Core/Build | Medium | @comphead and @alamb | Software Engineering, Refactoring, Dependency Management, Compilers | 175 to 350 hours* |
| Ergonomic SQL Features | SQL FE | Medium | @berkaysynnada | SQL, Planning, Parsing, Software Engineering | 350 hours |
| Advanced Interval Analysis | Core | Advanced | @ozankabak and @berkaysynnada | Algorithms, Data Structures, Applied Mathematics, Software Engineering | 350 hours |
| Spark-Compatible Functions Crate | Extensions | Medium | @alamb, @andygrove | SQL, Spark, Software Engineering | 175 to 350 hours* |
| SQL Fuzzing Framework in Rust | Extensions | Advanced | @2010YOUY01 | SQL, Testing Techniques, Advanced Rust, Software Engineering | 175 to 350 hours* |
- There is enough material to make this a 350-hour project, but it is granular enough to make it a 175-hour project as well.
Implement Continuous Monitoring of DataFusion Performance
DataFusion lacks continuous monitoring of how performance evolves over time -- we do this somewhat manually today. Even though performance has been one of our top priorities for a while now, we didn't build a continuous monitoring system yet. This linked issue contains a summary of all the previous efforts that made us inch closer to having such a system, but a functioning system needs to built on top of that progress. A student successfully completing this project would gain experience in building an end-to-end monitoring system that integrates with GitHub, scheduling/running benchmarks on some sort of a cloud infrastructure, and building a versatile web UI to expose the results. The outcome of this project will benefit Apache DataFusion on an ongoing basis in its quest for ever-more performance.
Supporting Correlated Subqueries
Correlated subqueries are an important SQL feature that enables some users to express their business logic more intuitively without thinking about "joins". Even though DataFusion has decent join support, it doesn't fully support correlated subqueries. The linked epic contains bite-size pieces of the steps necessary to achieve full support. For students interested in internals of data systems and databases, this project is a good opportunity to apply and/or improve their computer science knowledge. The experience of adding such a feature to a widely-used foundational query engine can also serve as a good opportunity to kickstart a career in the area of databases and data systems.
Improving DataFusion DX
While performance, extensibility and customizability is DataFusion's strong aspects, we have much work to do in terms of user-friendliness and ease of debug-ability. This project aims to make strides in these areas by improving terminal visualizations of query plans and increasing the "deployment" of the newly-added diagnostics framework. This project is a potential high-impact project with high output visibility, and reduce the barrier to entry to new users.
Robust WASM Support
DataFusion can be compiled today to WASM with some care. However, it is somewhat tricky and brittle. Having robust WASM support improves the embeddability aspect of DataFusion, and can enable many practical use cases. A good conclusion of this project would be the addition of a live demo sub-page to the DataFusion homepage.
High Performance Aggregations
An aggregation is one of the most fundamental operations within a query engine. Practical performance in many use cases, and results in many well-known benchmarks (e.g. ClickBench), depend heavily on aggregation performance. DataFusion community has been working on improving aggregation performance for a while now, but there is still work to do. A student working on this project will get the chance to hone their skills on high-performance, low(ish) level coding, intricacies of measuring performance, data structures and others.
Improving Python Bindings
DataFusion offers Python bindings that enable users to build data systems using Python. However, the Python bindings are still relatively low-level, and do not expose all APIs libraries like Pandas and Polars with a end-user focus offer. This project aims to improve DataFusion's Python bindings to make progress towards moving it closer to such libraries in terms of built-in APIs and functionality.
Optimizing DataFusion Binary Size
DataFusion is a foundational library with a large feature set. Even though we try to avoid adding too many dependencies and implement many low-level functionalities inside the codebase, the fast moving nature of the project results in an accumulation of dependencies over time. This inflates DataFusion's binary size over time, which reduces portability and embeddability. This project involves a study of the codebase, using compiler tooling, to understand where code bloat comes from, simplifying/reducing the number of dependencies by efficient in-house implementations, and avoiding code duplications.
Ergonomic SQL Features
DuckDB has many innovative features that significantly improve the SQL UX. Even though some of those features are already implemented in DataFusion, there are many others we can implement (and get inspiration from). This page contains a good summary of such features. Each such feature will serve as a bite-size, achievable milestone for a cool GSoC project that will have user-facing impact improving the UX on a broad basis. The project will start with a survey of what is already implemented, what is missing, and kick off with a prioritization proposal/implementation plan.
Advanced Interval Analysis
DataFusion implements interval arithmetic and utilizes it for range estimations, which enables use cases in data pruning, optimizations and statistics. However, the current implementation only works efficiently for forward evaluation; i.e. calculating the output range of an expression given input ranges (ranges of columns). When propagating constraints using the same graph, the current approach requires multiple bottom-up and top-down traversals to narrow column bounds fully. This project aims to fix this deficiency by utilizing a better algorithmic approach. Note that this is a very advanced project for students with a deep interest in computational methods, expression graphs, and constraint solvers.
Spark-Compatible Functions Crate
In general, DataFusion aims to be compatible with PostgreSQL in terms of functions and behaviors. However, there are many users (and downstream projects, such as DataFusion Comet) that desire compatibility with Apache Spark. This project aims to collect Spark-compatible functions into a separate crate to help such users and/or projects. The project will be an exercise in creating the right APIs, explaining how to use them, and then telling the world about them (e.g. via creating a compatibility-tracking page cataloging such functions, writing blog posts etc.).
SQL Fuzzing Framework in Rust
Fuzz testing is a very important technique we utilize often in DataFusion. Having SQL-level fuzz testing enables us to battle-test DataFusion in an end-to-end fashion. Initial version of our fuzzing framework is Java-based, but the time has come to migrate to Rust-native solution. This will simplify the overall implementation (by avoiding things like JDBC), enable us to implement more advanced algorithms for query generation, and attract more contributors over time. This project is a good blend of software engineering, algorithms and testing techniques (i.e. fuzzing techniques).
Activity
Omega359 commentedon Feb 4, 2025
I do not know what level students would be for GSoC however @alamb had some ideas for CMU class projects for university level
ozankabak commentedon Feb 4, 2025
IIRC, GSoC students can be undergraduates or graduate students. The main difference between CMU class projects and GSoC projects is that the former will be optimizer-focused, while the latter can be anything. Probably we will collect optimizer-related ideas in the CMU bucket, with the remaining falling in the GSoC bucket.
mkarbo commentedon Feb 4, 2025
We are happy to provide input and feedback to anyone looking to get involved with #14429
ozankabak commentedon Feb 4, 2025
Thank you @mkarbo! Can I assume @eliaperantoni would help too?
Some other CTA:
eliaperantoni commentedon Feb 4, 2025
Absolutely :) I'm actively working on this or related features e.g. #14439
timsaucer commentedon Feb 4, 2025
@ozankabak Yes, I will always try to make time for students.
ozankabak commentedon Feb 4, 2025
@mertak-synnada has been working on benchmarking Synnada's fork, so he may be a good co-mentor (along with someone from InfluxData) for the continuous monitoring project.
ozankabak commentedon Feb 5, 2025
@comphead, would you be willing to mentor a student on a project to study our codebase and dependencies to reduce DF binary size?
berkaysynnada commentedon Feb 5, 2025
I got three posts from duckdb related with these ideas. Perhaps they can provide some insights:
Project #5504 -> https://duckdb.org/2024/06/26/benchmarks-over-time.html
Project https://github.com/apache/datafusion-python -> https://duckdb.org/2023/07/07/python-udf.html
Project #5483 -> https://duckdb.org/2023/05/26/correlated-subqueries-in-sql.html
comphead commentedon Feb 5, 2025
Hi @ozankabak I would love to.
alamb commentedon Feb 5, 2025
This is a great list -- thank you @ozankabak and everyone for the ideas
Idea proposal:
I would like to suggest an additional goal which is:
Projects I would be willing (and happy!) to help mentor:
Probably Not: Aggregation Performance
I would probably decline advising aggregation performance as an intern project as I don't think it would be a good intern experience (unless it was an exceptional intern -- see below). The code is already quite highly optimized and I don't have any simple ideas to make it faster (though maybe @Rachelint does). Any changes here must be made quite carefully to avoid regressions
The ideal candidate would be someone already with very strong Rust and low level optimization experience (we can teach them the needed database internals).
Probably Not: Correlated Subqueries
For this project:
For this project, I think a successful candidate (and the kind of person I would be happy to help mentor) is graduate students who have a background in query optimizers (for example, can explain clearly what Unnesting Arbitrary Queries is about)
This isn't a great project for someone who doesn't already have a deep understanding on queries, join graphs, and subqueries as it will take most of the summer just to understand what we are trying to do
alamb commentedon Feb 5, 2025
Another potential project that I think would be huge and very much in the realm is Spark Functions
This set of functions is critical to many DataFusion users and the code already largely exists. This would be an exercise in creating the right APIs, explaining how to use them, and then telling the world about them
ozankabak commentedon Feb 5, 2025
@alamb, thank you for sharing your thoughts. I added a difficulty column to guide students as they browse the projects. I agree that the "Advanced" level projects are either for graduate students or students with strong enthusiasm in this area. At least in my time, the student body included both kinds of students, in addition to more junior ones.
I updated the table to list you as a mentor for the projects you mentioned in your first comment. I will add your proposed project to the list as well. Do you have any recommendations for possible mentors for the correlated subquery project? We can help, but it would be better to find a mentor (or a co-mentor) for it.
@Rachelint, would you be willing to mentor a student who wants to work on high performance aggregations?
32 remaining items
XiangpengHao commentedon May 9, 2025
I'm happy to help!
alamb commentedon May 9, 2025
I likewise will be very happy to help
Rachelint commentedon May 9, 2025
Yes, really happy to help!
#15591 may be close to be ready, and it may be necessary to support spilling on it.
oznur-synnada commentedon May 12, 2025
Hi all, we've registered @Rachelint as the co-mentor on the GSoC portal since they've already agreed to the program rules. However @XiangpengHao could be the unofficial 3rd mentor here if need be.
@alamb since you're already the co-mentor of another project we feel focusing solely on that would be more manageable. We're being cautious as we're not sure how much effort and dedication mentoring will require :) Thank you all for volunteering!
alamb commentedon May 12, 2025
I think subqueries with @irenjj will require a substantial mentoring effort so this plan sounds good to me
Rachelint commentedon May 12, 2025
@waynexia seems also interested about
oznur-synnada commentedon May 12, 2025
I'm happy to register them as well!
2010YOUY01 commentedon May 13, 2025
We'll be using Discord for part of the discussion. Feel free to jump in @Rachelint @waynexia and anyone else!
https://discord.com/channels/885562378132000778/1371869062883184730
alamb commentedon Jan 10, 2026
There appears to be some excitement and people showing up in the Discord channel thinking about Google Summer of Code 2026. Does anyone want to organize / coordinate DataFusion participation this year?
mtauha commentedon Jan 10, 2026
Hi @ozankabak and @timsaucer!
I'm a student interested in participating in GSoC 2026, specifically the "Improving Python Bindings" project. I've been learning DataFusion and would love to start contributing.
I have experience with Python for quite a while now but it's my first time contributing to open source.
I'm already setting up my development environment and going through the examples.
Is there a particular area of the Python bindings that needs attention?
I'd love to start with some smaller contributions to get familiar with the codebase.
Looking forward to being part of the community!
timsaucer commentedon Jan 10, 2026
@mtauha This issue is related to last year's GSoC participation. We are still considering if we are going to participate in the program this year. You're always welcome to join in on contributing to one of the datafusion projects by working on the open issues.
mtauha commentedon Jan 10, 2026
Sure! I'd love to help.
timsaucer commentedon Jan 27, 2026
Closing since 2025 event is over.
alamb commentedon Jan 27, 2026
Here is the discussion for 2026: