You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Partial aggregate over fully sorted input fails with ResourcesExhausted instead of emitting completed groups #26575
A Partial aggregate over input sorted on all grouping keys (ClusteredPartialAggregateStream, OrderedPartialAggregateStream in 55.x) reserves memory for the whole batch it just aggregated before emitting the completed groups. If that reservation is refused under GroupClustering::Full, Aggregating::handle_stage returns Err(oom). The memory is already allocated, and emitting the completed groups would shrink the table to the one active group. So in a shared pool the query fails although it could continue.
This is a regression from 54. GroupedHashAggregateStream, still used with enable_migration_aggregate = false, emits early for every Partial aggregate, and for fully ordered input GroupOrdering::oom_emit_to emits only the completed groups.
To Reproduce
In DataFusion Comet on 55.2 (87 sorted grouping keys, a spillable shuffle writer in the same pool), 39 of 40 tasks failed with:
The stream held 474 KB between batches and asked for 11,959,496 bytes after the next one.
Expected behavior
On a refused reservation with full clustering, emit take_completed_state_batch() and keep only the active group, as the legacy stream does. Return the error only when no group is complete. Outputting::handle_stage already reserves just the retained table when it cannot hold the output batch, and the emitted rows are the ones the success path emits, so results and order stay the same.
Activity
fix: emit completed groups when a fully sorted partial aggregate runs…
fix: emit completed groups when fully sorted final and single aggrega… (