Repository navigation
[EPIC] Improve metrics tracking in EXPLAIN ANALYZE #18217
Description
Is your feature request related to a problem or challenge?
This issue is to track ideas for improving metrics tracking, such as implementing useful metrics that are currently missing.
- Add
output_bytesmetrics to Explain Analyze #16244 - Inconsistent
time_elapsed_scanning_totalresult across different queries scanning the same data #16619 - Add a fine grain memory usage tracking / break down of memory usage *within* each operator #16904
-
MetricValue::Customencoding\decoding support #16945 - Include metric
output_batchesintoBaselineMetrics#17027 - Epic: Decides default metrics to show in
EXPLAIN ANALYZEacross operators #18116 - Improve metrics in
DataSourceExecwithParquetsource #18195 -
RepartitionExecis missingBaselineMetrics#18218 - feat: Add
selectivitymetrics toFilterExec#18406 - Support
reduction_fatormetric (forEXPLAIN ANALYZE) inAggregateExec#18410 - Add selectivity metrics (for
Explain Analyze) in Hash Join #18409 - Support
selectivitymetrics (forExplain Analyze) in Nested Loop Join #18407 - Track individual expr's execution time in
ProjectExecmetrics (inEXPLAIN ANALYZE) #18456 - Minor enhancements to
EXPLAIN ANALYZEoutput formatting #18689 - Elapsed compute time not tracked for
csvfile scan #18795 - feat(parquet): Implement
scan_efficiency_ratiometric for parquet reading #18577
Describe the solution you'd like
No response
Describe alternatives you've considered
No response
Additional context
No response
Activity
simonvandel commentedon Oct 31, 2025
ProjectionExec currently shows the total elapsed time for all expressions. Would it make sense to show timings for each individual expression? Perhaps we could do it in VERBOSE mode
feat: Add `selectivity` metrics to `FilterExec` (#18406)
feat: Add `selectivity` metrics to `FilterExec` (apache#18406)
feat: Add `selectivity` metrics to `FilterExec` (apache#18406)
alamb commentedon Nov 19, 2025
Should we mention the improvements to
explain analyzein the blog post for 51?There are some neat improvements I think
I have a draf here
2010YOUY01 commentedon Nov 20, 2025
I have added more details to the draft blogpost.
There are several other useful metric additions that will be included in the upcoming 52.0.0 release. Also, the user guide (https://datafusion.apache.org/user-guide/metrics.html) hasn’t been updated yet, so let’s continue improving it and ship more updates in the next release 😄
alamb commentedon Nov 20, 2025
Sounds like a great plan!