Repository navigation
Remove the legacy GroupedHashAggregateStream and enable_migration_aggregate #25902
Description
Is your feature request related to a problem or challenge?
- Follow on to [EPIC] Split Aggregation Logic into Dedicated Streams #22710
- Earliest release this can land in: DataFusion
57.0.0(Release DataFusion57.0.0(Dec 2026 / Jan 2027) #25901)
@2010YOUY01 and others refactored GroupedHashAggregateStream into dedicated per-mode streams as part of
The old stream was kept behind datafusion.execution.enable_migration_aggregate = false as a fallback in case the new streams had bugs. The config option documentation says that the fallback will be removed after the 56.0.0 release.
Until it is removed, DataFusion carries two implementations, meaning:
- There are two code paths that must be kept correct, which makes the aggregate code larger and harder to understand and review.
- Bug fixes and features (spilling, metrics, memory accounting) have to be applied twice or explicitly excluded from the legacy path. (e.g. fix: duplicate groups after spilling in legacy hash aggregation with a fallback group key (type not having dedicated impl) #24889, Add aggregate-specific metrics to legacy grouped hash and TopK #24523, fix: leave memory for aggregate spill replay #25383)
- Users who set
enable_migration_aggregate = falseto work around a55.xbug silently keep running the old, unmaintained implementation.
Describe the solution you'd like
After 56.0.0 is released, delete the legacy stream and the config option, and note the removal in the upgrade guide.
The 56.0.0 release gives users one release with the fallback available, so any remaining regressions in the new streams can be reported and fixed before the fallback disappears.
Things to remove
Code:
datafusion/physical-plan/src/aggregates/grouped_hash_stream.rsStreamType::GroupedHashand the fallback branch inAggregateExec::execute_typedindatafusion/physical-plan/src/aggregates/mod.rsExecutionOptions::enable_migration_aggregateindatafusion/common/src/config.rsand its entry indocs/source/user-guide/configs.md
Docs:
- Section "6. Legacy grouped hash aggregation" of the
aggregatesmodule docs - Add an entry to
docs/source/library-user-guide/upgrading/for the removed config option
Describe alternatives you've considered
Keep the fallback for another release. This delays the cleanup and means every aggregate change during that time still has to consider the legacy path.
Additional context
- Unify the spilling aggregate streams behind one spill-replay driver #25537 asks that the spill driver unification land before the legacy stream is deleted, because the aggregation fuzzer uses the legacy stream as a differential oracle. Once the legacy stream is gone the fuzzer baseline needs another reference (for example single partition, no spilling, new streams).
Activity
fix(sql): keep the aggregate hold for DataFusion 55.2's legacy stream
fix(sql): keep the aggregate hold for DataFusion 55.2's legacy stream
fix(sql): keep the aggregate hold for DataFusion 55.2's legacy stream
fix(sql): keep the aggregate hold for DataFusion 55.2's legacy stream