Repository navigation
feat: analyze memory usage for size functions #23393
Description
Is your feature request related to a problem or challenge?
A follow up on #23357
Asked Claude to create tests for size functions available in datafusion and compare values obtained from functions versus mimalloc to track where size return inaccurate values. Please see analysis below. So idea is to create small framework to fix and maintain sizes for DF objects
Summary
Several Accumulator::size(), GroupsAccumulator::size(), and related implementations report far less memory than they actually hold, which lets the MemoryPool limits be silently bypassed. An audit of
~157 fn size() sites plus a measurement test against a tracking allocator surface both worst-case defects and a systemic pattern.
Evidence
Measured with a tracking global allocator around each populated struct, comparing struct.size() to real heap-live bytes:
| struct | rows / inputs | size() reports |
actual heap | undercount |
|---|---|---|---|---|
SlidingDistinctCountAccumulator<Int64> |
1 024 distinct | 64 B | 165 960 B | −100.0% |
SlidingDistinctCountAccumulator<Int64> |
16 384 distinct | 64 B | 2 654 280 B | −100.0% |
SlidingDistinctCountAccumulator<Utf8> |
8 192 distinct | 64 B | 1 425 480 B | −100.0% |
ArrayAggAccumulator<Utf8> |
8 192 unique strings | 131 208 B | 164 288 B | −20.1% |
ArrowBytesSet<i32> |
32 768 unique utf8 | 2 419 608 B | 2 687 136 B | −10.0% |
ArrowBytesViewSet |
32 768 unique utf8-view | 2 592 992 B | 2 949 352 B | −12.1% |
ScalarValue::size() and TrivialFirst/LastValueAccumulator::size() were byte-exact — the pattern is fixable, not intrinsic.
End-to-end, on SELECT k, count(v), sum(v), avg(v) FROM t GROUP BY k over 200 k rows the aggregate operators reserve 55 MB against a real live peak of ~7 MB (720% ratio) — accounted allocations
also don't get released as batches emit downstream.
Root causes found in the audit (25 verified findings, high-severity subset)
SlidingDistinctCountAccumulator/SlidingDistinctSumAccumulator—size()returns onlysize_of_val(self), ignoring theHashMapbucket capacity and ScalarValue key heap.
functions-aggregate/src/count.rs:557,sum.rs:619.CountGroupsAccumulator— usessize_of::<usize>()for aVec<i64>. Wrong element type.count.rs:771.SymmetricHashJoinStream— omitsbatch_transformer, which can hold multi-MBRecordBatch.symmetric_hash_join.rs:1650.OneSideHashJoiner— double-counts fields already insize_of_val(self);HashSet<usize>counted at 8 B/slot instead of 16 B.symmetric_hash_join.rs:1170.ArrowBytesViewMap::size()— sumsbuffer.len()instead ofbuffer.capacity(); misses thecompleted: Vec<Buffer>container.binary_view_map.rs:476.GroupValuesPrimitive::size/GroupValuesRows::size— missingsize_of::<Self>().GroupValuesColumn::size— ignoresgroup_index_lists,emit_group_index_list_buffer, and the 5 Vecs invectorized_operation_buffers(can be 100s of KB).GroupOrdering::size— overcounts by adding bothsize_of::<Self>()and the inner variant'ssize()(which already includes it).aggregates/order/mod.rs:137.FirstLastGroupsAccumulator,ArrayAggGroupsAccumulator,VarianceGroupsAccumulator,AvgGroupsAccumulator,HllGroupsAccumulator,PercentileContGroupsAccumulator,MinMaxBytesState,ArrayMap,
TopK(missingcommon_sort_prefix_converter),TrivialNthValueAccumulator,StringAggAccumulator, andExprIntervalGraph— all confirmed under/over-count issues.
Full list with concrete failure scenarios and recommended fixes available on request.
Describe the solution you'd like
No response
Describe alternatives you've considered
No response
Additional context
No response
Activity
comphead commentedon Jul 8, 2026
Adding a small mimalloc comparison framework and modify the
SlidingAccumulatorsize to make sure numbers are approximately close to each othercomphead commentedon Jul 8, 2026
would be also useful to prevent regressions when the underlying object changed and
sizedoesn't reflect new dependenciesmbutrovich commentedon Jul 13, 2026
Fragmentation is inevitable. Allocations are pretty much always going to be page-sized so it's hard to take too much away from measuring small allocations against heap-reported memory since it gets amortized over time by a good allocator. Stuff like this:
That's valuable to surface since we should at least be counting non-ephemeral data structures.
More broadly, I think the biggest challenge with accurate memory accounting is when batches share underlying buffers, particularly the first stage out of a parquet read. For example in this PR, really anywhere we make decisions about data being accumulated and possibly spilling, I had to introduce logic to dedupe underlying buffers apache/datafusion-comet#4507. This commit was the subset apache/datafusion-comet@f909136 related to memory accounting.
chore: fix `SlidingDistinctCountAccumulator::size()` to include budge…
chore: adjust `size` accounting for `min_max` (apache#23899)
Fix aggregate accumulator capacity accounting (apache#24099)
chore: fix `SlidingDistinctCountAccumulator::size()` to include budge…
chore: adjust `size` accounting for `min_max` (apache#23899)
Fix aggregate accumulator capacity accounting (apache#24099)
Fix ArrowBytesViewMap retained capacity accounting (apache#24257)
Fix ArrowBytesViewMap retained capacity accounting (apache#24257)
Account for transformer-retained batches in symmetric hash join (apac…
Fix OneSideHashJoiner memory size accounting (apache#24744)
Fix AvgGroupsAccumulator::size() self-overcount (apache#25117)
Fix GroupValues retained memory accounting (apache#25188)
pmoust commentedon Oct 9, 2026
One more data point for this audit, from
ArrowBytesViewMap::size()indatafusion-physical-expr-common54.1.0 (binary_view_map.rs). We measured it with a heap profile during a string-keyedGROUP BY(ClickBench Q35, 20M rows).size()(around lines 473-486) counts the views asviews.len() * 16(around line 474). The vector grows by doubling (views.push, around line 451), so its capacity runs 1-2x its length. In our run the views held 89 MB, of which about 22 MB were uncounted on average.map_size.HashTableAllocExt::insert_accounted(datafusion-common/src/utils/proxy.rs, around lines 158-184) addscapacity * size_of::<Entry>()on each growth. hashbrown actually allocatesbuckets * (size_of::<Entry>() + 1), with buckets at capacity × 8/7, so about 18% more before allocator size-class rounding. In our run the table held 242 MB, about 36 MB of it uncounted.row_hash.rs, around lines 1067-1090). A table doubling is therefore allocated before it is charged, while the old table is still live during the rehash.Counting
views.capacity(), and sizing the table fromallocation_size()rather than from growth bumps, would close most of this.Fix GroupValues retained memory accounting (apache#25188)
Fix retained-capacity accounting for grouped byte MIN/MAX (apache#26154)