Repository navigation
Add a fine grain memory usage tracking / break down of memory usage *within* each operator #16904
Description
Activity
Within the DataFusion CLI there is a flag called:
--top-memory-consumers <TOP_MEMORY_CONSUMERS> The number of top memory consumers to display when query fails due to memory exhaustion. To disable memory consumer tracking, set this value to 0 [default: 3]This helped me identify large consumers of memory in: #16776. Perhaps the same solution can be implemented for your request.
The problem with that is there is no breakdown on what the memory is actually spent on in each consumer
so if I have a aggregation that takes a lot of memory I won't know about it
I agree more details are good -- I think the trick will be figuring out how to do so in such as way that is reasonable to maintain as well as doesn't slow down the normal operation
Also possibly related:
With this feature, issues like this are much easier to debug:
Reacted by Andrew Lamb@kosiew has a PR that adds this capability -- please check it out if you have time
Basically it uses the existing
TrackConsumersPoolthat was contributed by @wiedld to get memory usage per operatorMaybe we need to add some more documentation / examples of how to use that API
@alamb -- as for more documentation, do you means adding code examples with the TrackConsumersPool docs? And/or in the
datafusion-examples?Reacted by Andrew LambSpecifically I think @rluvaton is requesting something that is already possible with TrackConsumersPool -- and if that is the case, then we could potentially resolve the ticket by adding some more docs / examples to the TrackConsumersPool.
We might add an example (ideally in the docstrings) of how to get the largest consumers of a plan / sample the current memory users while the plan is running
Reacted by wiedld@alamb -- here is an initial draft of some docs for the current functionality of
TrackConsumersPool:
#17213The new/incoming functionality (PR: #17021) will require an expansion of the docs to include it's usage in realtime monitoring (not just error reporting). That other PR adds other
TrackConsumersPoolAPIs and cumulative metrics for realtime reporting and monitoring.Please lmk @rluvaton if these docs, combined with the incoming CLI (same PR above) fulfills what you are seeking?
Specifically I think @rluvaton is requesting something that is already possible with TrackConsumersPool -- and if that is the case, then we could potentially resolve the ticket by adding some more docs / examples to the TrackConsumersPool.
nope, this is a new functionality.
Example:
in the case of:I want when memory failed and by some API that I can call it will print not only all the consumers but the consumers size breakdown.
for:
Resources exhausted: Failed to allocate additional 55.0 MB for GroupedHashAggregateStream[3]I don't really know what takes a lot of memory, is it group values? accumulators? which accumulator?. this makes it really hard to debug memory issues.
I want to get something like for all existing consumers (not only the one that failed):
GroupedHashAggregateStream[3]: accumulators: total size <> accumulator[1] <name>: <size> accumulator[2] <name>: <size> group_values: <size> group_ordering: <size> current_group_indices: <size>as this is the code that calculate the size in row_hash:
https://github.com/apache/datafusion/blob/8f15991f33bf6aca9d4da8958141b59d196b2ed6/datafusion/physical-plan/src/aggregates/row_hash.rs#L916C9-L922C1this will require almost everything that implement size to also implement some sort of explain memory
This PR by @kosiew was more what I meant:
the changes where deleted but this is gist:
pub trait MemoryUsage { /// Returns the total bytes used by `self`. fn size(&self) -> usize; /// Returns a breakdown of memory usage for `self`. fn explain_memory(&self, formatter: &mut std::fmt::Formatter,) -> std::fmt::Result; }
everything that have a
sizefunction will be replaced to force implement this trait for ensuring greater memory explain coverage.so
Accumulatortrait will no longer have the size function:
datafusion/datafusion/expr-common/src/accumulator.rs
Lines 74 to 82 in 8f15991
and instead it will be forced to implement the
MemoryUsagetrait.same for
GroupValues:
datafusion/datafusion/physical-plan/src/aggregates/group_values/mod.rs
Lines 99 to 100 in 8f15991
and for
MemoryReservation
datafusion/datafusion/execution/src/memory_pool/mod.rs
Lines 357 to 360 in 8f15991
- changed the title
[-]Add a way to get what takes memory[/-][+]Add a fine grain memory usage tracking / break down of memory usage *within* each operator[/+]on Aug 18, 2025 I've marked the docs PR as ready for review: #17213
And I'll separately start work on the proposed new API.
I've marked the docs PR as ready for review: #17213
And I'll separately start work on the proposed new API.
FYI about the API, I suggest you do it incrementally:
- Figure out how to add a new API that DOES NOT require changing all existing call sites (this will also be the upgrade path for existing code)
- Update one operator (e.g. GroupByHashStream) to use the new API
- Add an example (ideally in doc comments) that shows how to use the new API to produce the report @rluvaton desires
Using the example, we can then make sure the API looks good / does what we need, and then we can figure out how to roll it out to the rest of the code
@alamb -- I feel like we need a new epic to delineate the different needs, and then consider whether we have solutions to address these needs. (Please lmk, and I can create this Epic.)
I'm going to take a shot at consolidating the identified needs:
-
more fine grained memory tracking, per Add a fine grain memory usage tracking / break down of memory usage *within* each operator #16904 (comment) (THIS TICKET)
- This may be about tags or "sub-consumers" under a MemoryConsumer. Or maybe a graph/connections between existing consumers? (unclear here)
- as mentioned here, try not to have all callers need to change
-
provide APIs for memory tracking in non-error situations (a.k.a. Runtime Monitoring), per Add Memory Profiling Functionality #14510
- imo, this expands beyond the existing report_top, and is more aligned with the
ConsumerMemoryMetricsin this open PR: Add memory profiling support to DataFusion CLI and memory pool metrics #17021
- imo, this expands beyond the existing report_top, and is more aligned with the
-
provide CLI tooling for doing the memory profiling
- e.g. the
\memory_profilingin the same PR referenced above: Add memory profiling support to DataFusion CLI and memory pool metrics #17021
- e.g. the
-
consider how memory is accounted for in shared data buffers
- current datafusion ticket: [DISCUSSION] Memory accounting model discussion #16841
- perhaps building on work done for this epic in arrow-rs: [EPIC] Memory Pool for Memory Tracking arrow-rs#8137
Reacted by Yongting You-
@kosiew -- your linked PR #17021 (which is great btw, thank you) states that it closes this issue ticket. I haven't reviewed it yet, altho on quick skim I don't think it address the itemization (e.g. breakdown into sub consumers etc) as denoted here?
Please feel free to point out if I missed something. 🙏🏼
@wiedld
You're right.
The PR now summarizes, rather than displays the fine grain.
During development, I found that the fine grain reporting might be too detailed for everybody and so opted to report a summary.
I will amend the PR description accordingly.Reacted by wiedldThe open PR from @kosiew includes some new CLI tooling, which may take longer in the code review process.
To unblock progress on this ticket, here is a proposed solution which brings us closer to the example output mentioned by @rluvaton here. Specifically, it provides a bit more context and a lineage of how these memory consumers were created (to help debug). Please let me know if this solution is useful:
https://github.com/apache/datafusion/pull/17943/files#r2408421334
Is your feature request related to a problem or challenge?
Yes, debugging memory problems are hard, when running DF in production and the memory pool is unable to grow the memory it will return
ResourcesExhaustederror or panic depending on the memory pool implementation, however we don't know what takes all the memory.In my case I had to manually patch DataFusion
row_hashfile to print on error what every accumulator takes and the internalsDescribe the solution you'd like
I want to add new API - that has
sizefunction (not required just because I think it should be combined withexplain_memory) andexplain_memoryfunction or something (similar toDebugtrait) to get string with breakdown of the size with every thing that takes memory.So for
GroupedHashAggregateStreamit would say how much group values takes, and for each accumulators it would call the explain memory on eachDescribe alternatives you've considered
No response
Additional context
No response