Repository navigation
Blog post about processing larger than memory queries #17089
Description
Is your feature request related to a problem or challenge?
- part of [Epic] DataFusion Blogs #14836
As part of trying to help people understand how to tune memory related settings in #17069, @2010YOUY01 pointed out #17069 (comment)
Let's make it concise now, I think adding a few more sentences of explanation might actually confuse those without the background knowledge. A tutorial-style doc is still needed to describe the full picture.
I agree and I think a blog post explaining how DataFusion processes larger than memory data sets would be really helpful to give people the broader context
Describe the solution you'd like
I suggest a blog on https://datafusion.apache.org/blog/ (make a PR to https://github.com/alamb/datafusion-site)
Describe alternatives you've considered
Some ideas
- Background on memory usage in DataFusion (the memory manager model and what takes large amounts of memory) -- can probably copy/paste from https://docs.rs/datafusion/latest/datafusion/execution/memory_pool/trait.MemoryPool.html#memory-management-overview
- Memory Consumers Describe at a high level what consumes most memory in a plan (group by hash table, sorting, etc)
- Optimizations that DataFusion does to try and avoid needing memory (e.g. take advantage of pre-xisting sort orders, topk, etc)
- Spilling Sort: Provide an overview of the main spilling sort algorithm (sort in memory, spill to disk, merge pre-sorted runs, etc)
- Using Spilling Sort in Group By: Explain that the grouping operation uses the same underlying building block
- Call for help: 🎣 for people to figure out how to add spilling for joins and make the spilling hash faster
Additional context
I am very happy to help write this
Activity
alamb commentedon Aug 8, 2025
@ding-young maybe you want to try out writing blogs as part of GSOC ?
cc @2010YOUY01
2010YOUY01 commentedon Aug 8, 2025
Thank you for posting the issue for blog idea!
I wish this blog to be more of explaining how spilling execution work internally, and add more example codes to help potential users get started.
Although @rluvaton @ding-young and others has done a lot to fix existing bugs, we still have a long way to go to stabilize this feature (i.e. less buggy, make it faster), we should definitely admit that and 🎣 for help in this 1st blog.
I plan to work with @ding-young for this blog post.
alamb commentedon Aug 8, 2025
In my opinion the best blog posts follow the following formula, for your consideration
That results in a wider audience than something that only focuses on DataFusion (and therefore also brings more people into the DataFusion comunity)
kiran-4444 commentedon Mar 16, 2026
Hey, @alamb I'd love to take this up. As someone trying to go through the internals of datafusion with passion for database technologies, I feel that this issue would be a great starting point. This will help me (and motivate me) to learn about how datafusion works under the hood. The process of formulating the blog definitely cracks open my knowledge gaps, thus pushing me to learn the concepts better. I'm actually quite excited!
alamb commentedon Mar 16, 2026
Sounds good -- thanks @kiran-4444 -- hopefully we can also get some of the other contributors to help too
kiran-4444 commentedon Mar 17, 2026
take