Repository navigation
Proposal: DataFusion Cookbook #24569
Description
A cookbook is a collection of short, self-contained, task-framed recipes ("I want to do X → here's how, here's why, here's the gotchas") — as opposed to reference docs (API surface) or a linear tutorial (one path through the basics). It's a proven format once a technology has enough surface area that no single tutorial covers it well.
Why it'd help DataFusion
DataFusion is powerful enough that the hard part for new adopters isn't "does this feature exist" but "how do I wire it together for my use case" — custom TableProvider, which UDF/UDAF/UDWF/UDTF variant to use, pushdown, catalog integration, remote object stores. A cookbook is the standard way projects close that gap.
Examples of cookbooks elsewhere
Published books:
- AWS Cookbook (O'Reilly) — https://www.amazon.com/AWS-Cookbook-Recipes-Success/dp/1492092606
- Terraform Cookbook (O'Reilly) — https://www.amazon.com/Terraform-Cookbook-Kerim-Satirli/dp/1804613477
- Django Web Development Cookbook (Packt, 4th ed.) — https://www.amazon.com/Django-Web-Development-Cookbook-development/dp/1838987428
- Polars Cookbook (Packt) — https://www.amazon.com/Polars-Cookbook-practical-transform-manipulate/dp/1805121154
Same-ecosystem, in-project precedents:
- Rust Cookbook — https://github.com/rust-lang-nursery/rust-cookbook
- ADBC Python Cookbook, embedded in the Arrow docs — https://arrow.apache.org/adbc/current/python/recipe/index.html
Relationship with examples
datafusion-examples (https://github.com/apache/datafusion/blob/main/datafusion-examples/README.md) already has ~50 runnable, tested examples across 11 groups: UDF/UDAF/UDWF/UDTF, custom TableProvider, custom file formats, Parquet encryption (incl. KMS), custom optimizer/analyzer rules, Flight SQL, querying S3, remote catalogs, and more.
These examples can be turned into recipe for a cookbook by rewriting each as a short step-by-step recipe — problem statement, code, explanation, pitfalls.
Probably 50 recipes are excessive, a good number would be in the 15-20, at least to start.
Organization options
Existing cookbooks adopt both the colocated and the separate repo approach, so we would need to decide between these two options:
- As a new section of the existing docs site, next to
datafusion-examples, versioned with the code (closest precedent: ADBC's cookbook, which lives directly in the Sphinx docs). - A separate repo, lower barrier for outside contributors, easier to restructure freely, but risks drifting from releases (precedent: Rust Cookbook, Spice.ai's cookbook repo).
Activity
edmondop commentedon Aug 21, 2026
Cc @alamb for visibility
Jefffrey commentedon Aug 22, 2026
slightly related but i found this website which has a good proposed way for organizing documentation: https://diataxis.fr/
in this case, cookbooks are equivalent to the how-to guides i believe 👍
2010YOUY01 commentedon Aug 23, 2026
Great proposal, +1 for this idea.
One thing I've been thinking about is how we should write tutorials differently in the age of LLMs. AI can explain things and often give shockingly good explanations, so perhaps tutorials should become more concise and focus more on intuition and the key ideas, rather than trying to be complete. 🤔
Another related issue:
alamb commentedon Aug 24, 2026
This is my vote
alamb commentedon Aug 24, 2026
Yes -- I have been dreaming about this too and I have a bunch of ideas that I want to try and help out with. I think this datafusion-cookbook should be primarily targeted at people building software with LLMs / coding tools
For example, I am imagining a user providing a prompt like this
I tried something like this at a seminar I was at recently. While claude was able to implement it, I found it took a while and took a bunch of iteration and instruction. I am feeling if it was complicated for me to do it, the barrier is still way too high
So what I imagine is "testing" this repo regularly with some pre-cooked prompts and then figuring out what is missing to make the process faster and more efficient.
For example, one thing that is complicated, is figuring out versions and how to integrate them. One thing I could see is a table / listing of all the various datafusion "plugins" that exist (e.g. datafusion distributed, flight, etc) as well as pointers to version compatibility and installation instructions / examples
alamb commentedon Aug 24, 2026
If there are no objections, how about I create
datafusion-contrib/datafusion-cookbookand we can start working on a draft there?alamb commentedon Aug 25, 2026
It turns out that @timvw had actually started this 4 years ago
However, it has had no activity for a while, so I also added @edmondop as an admin too.
Maybe we can brush it up, and we can get something started to start with
alamb commentedon Aug 27, 2026
See discussion on