EPIC: Tuning guide for DataFusion #16178

Is your feature request related to a problem or challenge?

DataFusion currently has around 100 configuration settings: https://datafusion.apache.org/user-guide/configs.html — and the number is still growing.

It can be tricky to tune configurations based solely on the brief descriptions in the documentation. Some settings require more context to fully understand their semantics.

A better approach would be to group related configurations and explain them with examples — effectively creating a tuning guide.

We could place the main content for this guide into separate examples under the datafusion-examples repository. This way, we can provide executable code alongside the documentation, and link to these examples from a dedicated doc page.

Here are two configuration groups I think would particularly benefit from such a guide:

• Join-related configurations Tuning Guide for Joins in SQL Queries #16176
• Configurations for out-of-core (larger-than-memory) execution Tuning Guide for Larger-than-memory Queries #16177
• ... (suggestions for additional topics are welcome!)

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

DataFusion Comet has a great tuning guide at https://datafusion.apache.org/comet/user-guide/tuning.html