Repository navigation
Document DataFusion Threading / tokio runtimes (how to separate IO and CPU bound work) #12393
Description
Is your feature request related to a problem or challenge?
DataFusion performs CPU bound work within async closures. This causes issues if running IO on the same async runtime, as the cooperative nature of such schedulers allows the CPU bound work to starve servicing of IO. This leads to errors such as apache/arrow-rs-object-store#272.
Describe the solution you'd like
I think at the very least this needs to be better documented, I couldn't find any mention of this in the DataFusion documentation following a cursory search.
I also think more holistic approach would be valuable to this, as it stands the use of async within DataFusion acts as a massive footgun that encourages users to intermix IO and CPU work in a way that is at best inefficient, but this can be tracked as a separate follow on task.
Describe alternatives you've considered
No response
Additional context
No response
Activity
alamb commentedon Sep 9, 2024
I recommend two things:
[-]Document DataFusion Threading[/-][+]Document DataFusion Threading (and how to separate IO and CPU bound work)[/+]ozankabak commentedon Oct 7, 2024
I think it'd be great to have a good documentation on this.
alamb commentedon Oct 25, 2024
100% agree -- @itsjunetime and @tustvold are working on a bit of it in apache/arrow-rs#6612. I'll try and help with the documentation as well
[-]Document DataFusion Threading (and how to separate IO and CPU bound work)[/-][+]Document DataFusion Threading / tokio runtimes (how to separate IO and CPU bound work)[/+]alamb commentedon Nov 16, 2024
Documentation
I hope to work on the example a bit more shortly
adriangb commentedon Dec 3, 2024
I was thinking, beyond examples, could we include a basic implementation in DataFusion? I'm not saying it's wired up by default, but put it behind a feature flag and make it provisional but ship it with the source code? Maybe that + docs on how to use it is enough for a lot of use cases?
I think we have a working version of this in 2 files and not that many LOC. We'd be happy to donate it.
alamb commentedon Dec 3, 2024
Yes I agree this would be ideal. Thank you @adriangb
That would be awesome -- thank you! I can potentially adapt my example to use what you have
Any chance you can make a PR?
adriangb commentedon Dec 3, 2024
@alamb #13634
alamb commentedon Dec 8, 2024
Update here:
alamb commentedon Jun 20, 2025
I am still looking for a committer to help review the next round of this example (which uses the object_store APIs)
alamb commentedon Jun 23, 2025
😅 "finally"