You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Extra memory allocated during partial aggregation early emit during OOM handling #22164
When the partial hash aggregation with GroupOrdering::None cannot reserve more memory, it switches to an early emit phase with EmitTo::First(n), where n is close to the length of group_values. take_n implementations use first_n_offsets = self.offsets.drain(0..n).collect::<Vec<_>>();, which copies the n elements into a new allocation. Furthermore, drain doesn't affect the Vec's capacity, so no memory freeing is done.
Activity
fix: reduce memory allocation overhead during partial aggregation ear… (
perf: hoist split_vec_min_alloc to datafusion-common and shrink the e…
perf: hoist split_vec_min_alloc to datafusion-common and shrink the e…