Add Memory Profiling Functionality

Is your feature request related to a problem or challenge?

When executing long-running batch processes or streaming queries, it is difficult to diagnose and monitor memory usage. People need an easy way to inspect memory consumption to understand resource usage and troubleshoot potential OOM. issues.

Describe the solution you'd like

Implement a built-in memory profiling feature
(similar to DuckDB’s duckdb_memory() and duckdb_temporary_files() functions)

┌──────────────────┬────────────────────┬─────────────────────────┐
│       tag        │ memory_usage_bytes │ temporary_storage_bytes │
│     varchar      │       int64        │          int64          │
├──────────────────┼────────────────────┼─────────────────────────┤
│ BASE_TABLE       │          168558592 │                       0 │
│ HASH_TABLE       │                  0 │                       0 │
│ PARQUET_READER   │                  0 │                       0 │
│ CSV_READER       │                  0 │                       0 │
│ ORDER_BY         │                  0 │                       0 │
│ ART_INDEX        │                  0 │                       0 │
│ COLUMN_DATA      │                  0 │                       0 │
│ METADATA         │                  0 │                       0 │
│ OVERFLOW_STRINGS │                  0 │                       0 │
│ IN_MEMORY_TABLE  │                  0 │                       0 │
│ ALLOCATOR        │                  0 │                       0 │
│ EXTENSION        │                  0 │                       0 │
├──────────────────┴────────────────────┴─────────────────────────┤
│ 12 rows                                               3 columns │
└─────────────────────────────────────────────────────────────────┘

FROM duckdb_temporary_files();
┌────────────────────────────────┬───────────┐
│              path              │   size    │
│            varchar             │   int64   │
├────────────────────────────────┼───────────┤
│ .tmp/duckdb_temp_storage-0.tmp │ 967049216 │
└────────────────────────────────┴───────────┘

that allows to query:

The memory usage of different streams or even other objects (e.g., hash tables, intermediate results, buffers).
Current temporary file usage, if applicable.

Describe alternatives you've considered

No response

Additional context

No response

@berkaysynnada

That is nice @berkaysynnada I love it.

To implement it we have to use a memory reservation in most of intensive operations, like Sorting, Join, hash, scan, etc, now we have quite few operations supporting memory reservation, the spill support even fewer.

To break down this into manageable pieces we need to

figure out what parts of execution plan we wanna monitor
create a subissue to integrate memory reservation for each part of exec plan
Display it back to the user. We already have a metrics builder on top of the ExecutionPlan and we can access metrics through EXPLAIN ANALYZE so probably this can be a substitute for the builtin function? Should we output per partition or aggregated, max value or mean?

@comphead

Thank you @comphead for the great elaboration! I don't have a clear answer to your question yet, but since this is the final part of our work list, I believe someone can complete the first two items before deciding on how to collect and display.

I am interested in this function.

Tag memory that is allocated through the buffer manager, and add duckdb_memory() function by Mytherin · Pull Request #10496 · duckdb/duckdb
DuckDB adds memory tags when the buffer manager allocates memory, making it easier to trace memory usage.

Does DataFusion also need to implement it in a similar way?

I have just started to learn DataFusion’s memory management and I noticed:
https://docs.rs/datafusion/latest/datafusion/execution/memory_pool/trait.MemoryPool.html

Rather than tracking all allocations, DataFusion takes a pragmatic approach: Intermediate memory used as data streams through the system is not accounted (it assumed to be “small”) but the large consumers of memory must register and constrain their use. This design trades off the additional code complexity of memory tracking with limiting resource usage.

Do we need to trace all memory allocations or just focus on the part managed by MemoryPool?

@PokIsemaine

Hi @PokIsemaine I'm planning to experiment with this.
One thing as you mentioned absolutely correct is to use MemoryReservation which is the helper when memory allocated through the pool and can be read in metrics using EXPLAIN ANALYZE query statement. Challenge is our memory pool coverage needs to be improved and get a lot of transformation covered.

Another dirty trick which might work is to try to track the max process memory per the each transformation using https://crates.io/crates/sysinfo

This approach shows the process memory usage globally not just per specific node of physical plan, but using the
Having max memory fingerprint it would be easy to find trends and heavyweight operations

I am interested in this function.

Tag memory that is allocated through the buffer manager, and add duckdb_memory() function by Mytherin · Pull Request #10496 · duckdb/duckdb DuckDB adds memory tags when the buffer manager allocates memory, making it easier to trace memory usage.

Does DataFusion also need to implement it in a similar way?

No, they're different. I believe duckdb's memory pool is a textbook buffer pool, which manages memory spilling and reading back automatically for the operators.
For DataFusion, MemoryPool perhaps is a misleading naming, it's actually a 'memory tracker', operators have to use this tracker to estimate if it has exceeded the memory limit. If so they will explicitly do the spilling, or fail with a user friendly error message.

I have just started to learn DataFusion’s memory management and I noticed: https://docs.rs/datafusion/latest/datafusion/execution/memory_pool/trait.MemoryPool.html

Rather than tracking all allocations, DataFusion takes a pragmatic approach: Intermediate memory used as data streams through the system is not accounted (it assumed to be “small”) but the large consumers of memory must register and constrain their use. This design trades off the additional code complexity of memory tracking with limiting resource usage.

Do we need to trace all memory allocations or just focus on the part managed by MemoryPool?

Regarding tracking those small allocations by internal MemoryPool, maybe not. However, if we can use the system memory profiler, or internal metrics from a memory allocator (e.g. mimalloc), it would be great to also integrate them.

The PR in arrow rs to avoid mem overcount for shared buffers apache/arrow-rs#7303

The PR in arrow rs to avoid mem overcount for shared buffers apache/arrow-rs#7303

This was merged and is now available in DataFusion

@kosiew

Also it seems there is a related ticket

Add a fine grain memory usage tracking / break down of memory usage *within* each operator #16904

@kosiew has a PR / proposal

Add memory profiling support to DataFusion CLI and memory pool metrics #17021

Very exciting progress! One thing that would be incredibly helpful is to extend the configuration flag, --top-memory-consumers from the DataFusion CLI into the configs (https://datafusion.apache.org/user-guide/configs.html) to get informative error messages. This would display N number of top memory consumers when the query fails due to memory exhaustion. By default it can be set to 0 to disable this.

I think a big leap forward would be to implement continuous profiling of DataFusion, since it's been quite difficult to do so with https://datafusion.apache.org/library-user-guide/profiling.html, unless more capabilities have been added.

Thinking even farther out, some nice-to-haves would be equivalent tools such as https://github.com/DBatUTuebingen/duckdb-explain-visualizer.

This would display N number of top memory consumers when the query fails due to memory exhaustion. By default it can be set to 0 to disable this.

I think that is what already happens. If not, can you show an example of what message you are proposing to change?

I think a big leap forward would be to implement continuous profiling of DataFusion, since it's been quite difficult to do so with https://datafusion.apache.org/library-user-guide/profiling.html, unless more capabilities have been added.

Thinking even farther out, some nice-to-haves would be equivalent tools such as https://github.com/DBatUTuebingen/duckdb-explain-visualizer.

I agree this would be great.

DataFusion already has all the APIs needed to implement this type of tool (including getting the current values of all statisics). All that is needed now is for someone to actually implement the UI / modify an existing one

Navigation Menu

Repository navigation

Add Memory Profiling Functionality #14510

Is your feature request related to a problem or challenge?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Add Memory Profiling Functionality #14510

Description

Is your feature request related to a problem or challenge?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Activity

comphead commented on Feb 5, 2025

berkaysynnada commented on Feb 6, 2025

PokIsemaine commented on Feb 25, 2025

comphead commented on Feb 27, 2025

2010YOUY01 commented on Mar 6, 2025

comphead commented on Mar 18, 2025

alamb commented on Aug 12, 2025

alamb commented on Aug 12, 2025

zheniasigayev commented on Aug 12, 2025

alamb commented on Aug 18, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions