Repository navigation
Add a "Gentle Introduction to Arrow / Record Batches" #11336
Description
Activity
take
Thank you @Adez017 -- I would personally suggest starting by porting some of the contents of https://jorgecarleitao.github.io/arrow2/main/guide/ into DataFUsion's docs
In terms of mechanism, I personally suggest making a PR into the arrow-rs repo: https://github.com/apache/arrow-rs and making the documentation doc comments in the arrow crate
This would mean they docs show up here:
And it will ensure the examples contine to run
I suggest starting with the sections
- Arrow format
- high level API
And see how that goes
Reacted by aditya singh rathoreThank you @Adez017 -- I would personally suggest starting by porting some of the contents of https://jorgecarleitao.github.io/arrow2/main/guide/ into DataFUsion's docs
In terms of mechanism, I personally suggest making a PR into the arrow-rs repo: https://github.com/apache/arrow-rs and making the documentation doc comments in the arrow crate
This would mean they docs show up here:
And it will ensure the examples contine to run
I suggest starting with the sections
- Arrow format
- high level API
And see how that goes
hi @alamb just a quick follow up , i got a little confused whether i need to PR the arrow-rs or datafusion and also in which specific folder .
Thank you @Adez017 -- I would personally suggest starting by porting some of the contents of https://jorgecarleitao.github.io/arrow2/main/guide/ into DataFUsion's docs
In terms of mechanism, I personally suggest making a PR into the arrow-rs repo: https://github.com/apache/arrow-rs and making the documentation doc comments in the arrow crate
This would mean they docs show up here:And it will ensure the examples contine to run
I suggest starting with the sections- Arrow format
- high level API
And see how that goes
hi @alamb just a quick follow up , i got a little confused whether i need to PR the arrow-rs or datafusion and also in which specific folder .
Thanks @Adez017 -- ideally we could put this in the arrow-rs repo -- perhaps we could make a new module called
intro.rssomewhere like https://github.com/apache/arrow-rs/blob/main/arrow/src/intro.rs that simply has a large doc comment with the guide, and then we can link that module from https://github.com/apache/arrow-rs/blob/main/arrow/src/lib.rs 🤔BTW thank you for all your help recently. I plan to work on at least one of the many docs you have created this upcomin week. Please don't take my lack of action as lack of enthusiam -- I just need to find significant contiguous time to sit down and review docs like that which has been hard for me to come by this past week
BTW thank you for all your help recently. I plan to work on at least one of the many docs you have created this upcomin week. Please don't take my lack of action as lack of enthusiam -- I just need to find significant contiguous time to sit down and review docs like that which has been hard for me to come by this past week
I can fully understand ,btw thanks for your efforts and time for reply as it helps me a lot
Hi all — quick check-in on #11336.
First-time contributor here. I’d love to help with the “Gentle Introduction to Arrow / RecordBatches.”. I think, it would fit perfectly
I see it’s assigned; I don’t want to step on toes.
I’m happy to co-author or take a clearly scoped subsection.
If helpful, I can open a small draft PR with a one-page outline for review.
Thanks!
Hi @sm4rtm4art , you can take this up .
Reacted by Martin and Andrew Lamb- added 2 commits that reference this issue
on Oct 13, 2025 - added a commit that references this issue
on Oct 27, 2025 - added 2 commits that reference this issue
on Dec 29, 2025
Part of #7013
Is your feature request related to a problem or challenge?
As @efredine notes on #11290 / #11290 (comment):
The in-memory examples are concise and its easy to get the gist of what's going on. But it also throws people in to the deep end of the Arrow format which lacks a gentle introduction IMO. The Arrow-rs documentation gets immediately into the weeds!
Describe the solution you'd like
It's likely that many users might never even need to know or access the arrow format directly. They will just read and write to csv or parquet.
I don't think this needs to change, but perhaps what's missing is a section on how and when to use the Arrow format? A gentler introduction to Record Batches
Describe alternatives you've considered
Add a section to the user guide on "a gentle introduction to arrow"
Additional context
here is a ticke tracking such a thing upstream: apache/arrow-rs#4071
I actually think the basic content / structure could be copied from https://jorgecarleitao.github.io/arrow2/main/guide/ with the examples being updated to reflect arrow-rs