Repository navigation
Improve the docs for developers (Create a Developers / Hackers guide for DataFusion) #5501
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
DataFusion is primarily aimed at developers, as explained in https://github.com/apache/arrow-datafusion#use-cases
Thus, it will help if provide documentation that helps developers understand what DataFusion offers and if it is appropriate for their project or not.
Describe the solution you'd like
I would like a DataFusion Architecture guide – aimed at other developers, that contains high level information about how DataFusion is organized (for example, with the content described in #5499)
The trick will be to keep the guide helpful but general enough that it doesn't get out of date too quickly
Topics:
- Basic flow
- Important structures (LogicalPlan, Exprs, SessionContext, PhysicalExpr, ExecutionPlan)
- DataSources (TableProvider trait, etc)
Specific tasks
- Decide where to put the content (ANSWER: main crate page https://docs.rs/datafusion/latest/datafusion/index.html)
- Improve contributor guide and main API landing page #5921
- Improve / Add documentation to
TableProvider - Improve / Add documentation to
CatalogProvider - Improve / Add documentation to
SessionContext - Clarify the difference between
SessionContext,SessionStateandTaskContext - Update / Improve the planning flow / description in main API docs page
Describe alternatives you've considered
Additional context
Part of a larger effort to improve documentation #3058
Activity
alamb commentedon Apr 4, 2023
One thing I have been thinking about is where to put this information
So far I have found at least two possiblities:
https://arrow.apache.org/datafusion/contributor-guide/architecture.html
https://docs.rs/datafusion/latest/datafusion/index.html#parse-plan-optimize-execute
In general I think keeping the architecture / code documentation as close to the code as possible is a good idea, so I am leaning towards keeping in in doc comments
alamb commentedon Apr 5, 2023
I plan to work on this item over the next few weeks
alamb commentedon Apr 8, 2023
I have begun cleaning up. First PR: #5921
[-]Create a Developers / Hackers guide for DataFusion[/-][+]Improve the docs for developers (Create a Developers / Hackers guide for DataFusion)[/+]