Repository navigation
Getting started guide for new users (who want to use DataFusion in their project) #7014
Description
Is your feature request related to a problem or challenge?
If we want to have DataFusion used as the core of many new systems, we need it to be as easy as possible for someone to get their idea working on top of DataFusion.
The current user guide I think helps setup the basics of the project and get a "hello world" style program going but then kind of leave the reader in a "now what" type situation: https://arrow.apache.org/datafusion/user-guide/example-usage.html
Describe the solution you'd like
I would like a document, perhaps similar in style to the polars user guide: https://pola-rs.github.io/polars-book/user-guide/
This User Guide is an introduction to the Polars DataFrame library. Its goal is to introduce you to Polars by going through examples and comparing it to other solutions. Some design choices are introduced here. The guide will also introduce you to optimal usage of Polars.
Basically I am thinking of something that would have helped @bubbajoe get up to speed
The examples directory holds a bunch of examples: https://github.com/apache/arrow-datafusion/tree/main/datafusion-examples
Potential outline:
- Library Guide: Add SQL level user guide: #7302
- Library Guide: Add Working with Exprs #7304
- Library Guide: Add Using the DataFrame API #7305
- Library Guide: Building LogicalPlans #7306
- Catalogs: tables / schemas / tables / catalogs (in Add library guide for table provider and catalog providers #7287)
- Library Guide: Adding User Defined Functions: Scalar/Window/Aggregate/ #7307
- Adding custom
TableProviders(in Add library guide for table provider and catalog providers #7287) - Library Guide: Extending DataFusion's operators: custom LogicalPlan and
ExecutionPlans#7308
Describe alternatives you've considered
No response
Additional context
This idea was suggested by @MrPowers
Activity
alamb commentedon Jul 18, 2023
If someone wanted to help out the DataFusion project helping with this one would be awesome. A good first step would be to make the skeleton of the topics above in https://github.com/apache/arrow-datafusion/tree/main/docs and leave placeholder text (like "Coming Soon")
Then we can work together on writing the content in a few different PRs
MrPowers commentedon Jul 18, 2023
This sounds great, really excited!
We'll either want two user guides or one user guide that's half in Python / half in Rust.
I guess that 99% of the users that want to query data via an API will want to do so in SQL / Python. The Python DataFrame user guide is way more important than the Rust one.
Users leveraging DataFusion to build tools for other engines (e.g. delta-rs) are much more likely to be using Rust.
Perhaps we divide the documentation as follows:
I don't think we should invest in building out the DataFusion Rust DataFrame API docs yet because it's a lower ROI activity. We should build a URL structure that allows for this however.
alamb commentedon Jul 19, 2023
I agree this is more important for "end users" rather than developers who are building with Rust
That sounds great -- I filed apache/datafusion-python#432 to track the work for the python bindings
[-]Getting started guide for new users (who want to use DatatFusion in their proejct) [/-][+]Getting started guide for new users (who want to use DataFusion in their project) [/+]alamb commentedon Aug 16, 2023
I filed a bunch of tickets for follow on work and update the description of this ticket
#7302
#7304
#7305
#7306
#7307
#7308
aaryyya commentedon Apr 6, 2025
Hey is this still active?
alamb commentedon Apr 6, 2025
Hi @aaryyya -- I think thanks to @tshauck and others with the library user guide, this is mostly done now (so closing it)
Perhaps you could review the library user guide and help improve it anywhere it is not clear or is missing details