[EPIC] Complete SQL EXPLAIN Tree Rendering

Is your feature request related to a problem or challenge?

#14677 introduces basic tree rendering for SQL EXPLAIN output but requires follow-up work to improve completeness, test coverage, and code quality. Below are the key tasks and enhancements needed:

Here is an example of running the new explains:

set datafusion.explain.format = 'tree';
create table foo(x int, y int) as values (1,2), (3,4);
explain select * from foo where x = 4;

The output looks like

+---------------+------------------------------------+
| plan_type     | plan                               |
+---------------+------------------------------------+
| logical_plan  | Filter: foo.x = Int32(4)           |
|               |   TableScan: foo projection=[x, y] |
| physical_plan | ┌───────────────────────────┐      |
|               | │    CoalesceBatchesExec    │      |
|               | └─────────────┬─────────────┘      |
|               | ┌─────────────┴─────────────┐      |
|               | │         FilterExec        │      |
|               | └─────────────┬─────────────┘      |
|               | ┌─────────────┴─────────────┐      |
|               | │       DataSourceExec      │      |
|               | │    --------------------   │      |
|               | │    partition_sizes: [1]   │      |
|               | │       partitions: 1       │      |
|               | └───────────────────────────┘      |
|               |                                    |
+---------------+------------------------------------+

Items to complete

Information Completion for all `ExecutionPlans`

The current implementation lacks detailed information for many physical plans.

Add rendering logic for missing operators (In every physical plan's fmt_as function).

New Features:

Process / Documentation

Document the tree explain in the user documentation
Add upgrade guide in 47 for TreeRender in DisplayFormatType`
Use tree explain by default

Test Coverage Expansion

Add tests for plans with long metrics (e.g., partition_sizes: [20, 20, 20, 30, ...]) to validate line-wrapping and truncation logic.
Tests for TPCH queries (e.g., Q5, Q9) to test rendering of deeply nested plans.
Validate alignment and indentation for multi-level operators.

Code Cleanup & Improvements

TODOs

Make variables (e.g., indentation size, truncation thresholds) configurable:

// TODO: Make these variables configurable.

Dead Code Removal

Remove or utilize the unused Coordinate struct:

#[allow(dead_code)]  
pub struct Coordinate { ... }  
// TODO: It's never used.

I made a first PR to show how to add tree explain

Implement tree explain for FilterExec #15001

Once we get that one in I plan to file a bunch of other tickets for the remaining ExplainPlan nodes

This looks great! Would it be possible to extend this with svg output? It would be great for ballista ui

This looks great! Would it be possible to extend this with svg output? It would be great for ballista ui

There is already a graphviz version (that kind of looks crappy).

Since the output is text, perhaps there is some way to render text as a svg 🤔

For the setting side, how about using the existing EXPLAIN [ANALYZE] FORMAT <format> grammar? sqlparser can parse and generate a AnalyzeFormat, which the planner can use. But our current implementation just drops this format field:

datafusion/datafusion/sql/src/statement.rs

Lines 213 to 222 in ed517ef

    
           Statement::Explain { 
        
               verbose, 
        
               statement, 
        
               analyze, 
        
               format: _, 
        
               describe_alias: _, 
        
               .. 
        
           } => { 
        
               self.explain_to_plan(verbose, analyze, DFStatement::Statement(statement)) 
        
           }

We can also try to support GRAPHVIZ format in this way, it should be easier to be rendered to svg.

@waynexia

For the setting side, how about using the existing EXPLAIN [ANALYZE] FORMAT <format> grammar? sqlparser can parse and generate a AnalyzeFormat, which the planner can use. But our current implementation just drops this format field:

datafusion/datafusion/sql/src/statement.rs

Lines 213 to 222 in ed517ef

Statement::Explain {
verbose,
statement,
analyze,
format: _,
describe_alias: _,
..
} => {
self.explain_to_plan(verbose, analyze, DFStatement::Statement(statement))
}
We can also try to support GRAPHVIZ format in this way, it should be easier to be rendered to svg.

This is a great idea @waynexia

It seems like AnalyzeFormat currently only supports Text, JSON, and GRAPHVIZ 🤔

https://docs.rs/sqlparser/latest/sqlparser/ast/enum.AnalyzeFormat.html

It would be a good way to show graphviz plans

For the setting side, how about using the existing EXPLAIN [ANALYZE] FORMAT grammar? sqlparser can parse and generate a AnalyzeFormat, which the planner can use. But our current implementation just drops this format field:

I filed the following ticket for this

Support different EXPLAIN formats via SQL #15021

I also filed a ticket to avoid printing logical plans when in tree mode

Do not print logical plans in tree explain format #15020

Here is another PR to improve DataSource:

Implement tree explain for DataSourceExec #15029

This looks great! Would it be possible to extend this with svg output? It would be great for ballista ui

There is already a graphviz version (that kind of looks crappy).

Since the output is text, perhaps there is some way to render text as a svg 🤔

ballista uses graphviz-rust which generates those sub-optimal images (and needs local graph viz installation). spark uses

https://github.com/dagrejs/graphlib-dot

a javascript framework which works much better for them. Maybe we should keep dot representation if we already have it

This looks great! Would it be possible to extend this with svg output? It would be great for ballista ui

There is already a graphviz version (that kind of looks crappy).
Since the output is text, perhaps there is some way to render text as a svg 🤔

ballista uses graphviz-rust which generates those sub-optimal images (and needs local graph viz installation). spark uses

https://github.com/dagrejs/graphlib-dot

a javascript framework which works much better for them. Maybe we should keep dot representation if we already have it

I think that if we just made it easier to use the existing formats, that would be a big win.

Can make tree a subcommand of explain, like explain tree <your sql>.

Can make tree a subcommand of explain, like explain tree <your sql>.

It looks like this is something that #15021 is going to handle.👀

@waynexia

Can make tree a subcommand of explain, like explain tree <your sql>.

It looks like this is something that #15021 is going to handle.👀

Yes indeed -- maybe you could ask @waynexia if you could do that PR?

I also just made

Do not display logical_plan win explain tree mode 🧹 #15132

To clean up the display a bit

@irenjj

Thank you so much to @irenjj @zebsme @Standing-Man and others I think we are basically done with this epic...

It is even documented!

https://datafusion.apache.org/user-guide/sql/explain.html#tree-format

🎉

I think we should consider using tree mode by default. I have filed a ticket to discuss this:

[DISCUSS] Switch to tree explain by default #15343

Navigation Menu

Repository navigation

[EPIC] Complete `SQL EXPLAIN` Tree Rendering #14914

Is your feature request related to a problem or challenge?

Items to complete

Information Completion for all `ExecutionPlans`

Process / Documentation

Test Coverage Expansion

Code Cleanup & Improvements

TODOs

Dead Code Removal

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	Statement::Explain {
	verbose,
	statement,
	analyze,
	format: _,
	describe_alias: _,
	..
	} => {
	self.explain_to_plan(verbose, analyze, DFStatement::Statement(statement))
	}

[EPIC] Complete SQL EXPLAIN Tree Rendering #14914

Description

Is your feature request related to a problem or challenge?

Items to complete

Information Completion for all ExecutionPlans

Process / Documentation

Test Coverage Expansion

Code Cleanup & Improvements

TODOs

Dead Code Removal

Activity

alamb commented on Mar 4, 2025

milenkovicm commented on Mar 4, 2025

alamb commented on Mar 4, 2025

waynexia commented on Mar 4, 2025

alamb commented on Mar 5, 2025

alamb commented on Mar 5, 2025

alamb commented on Mar 5, 2025

milenkovicm commented on Mar 6, 2025

alamb commented on Mar 6, 2025

niebayes commented on Mar 10, 2025

irenjj commented on Mar 10, 2025

alamb commented on Mar 10, 2025

alamb commented on Mar 10, 2025

alamb commented on Mar 21, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

[EPIC] Complete `SQL EXPLAIN` Tree Rendering #14914

Information Completion for all `ExecutionPlans`