Repository navigation
[EPIC] Extract remaining physical optimizer out of core #11502
Description
Is your feature request related to a problem or challenge?
Historically DataFusion was one (very) large crate datafusion, and as it grew bigger we extracted various functionality into separate crates. This leads to both faster compile times (as the crates can be compiled in parallel) as well easier to navigate code (as the crates force a cleaner dependency separation)
This project tracks moving the physical optimizers:
- The original physical optimizers are here https://github.com/apache/datafusion/tree/main/datafusion/core/src/physical_optimizer
- The new crate is here https://github.com/apache/datafusion/tree/main/datafusion/physical-optimizer
Describe the solution you'd like
Pull physical optimizer rules to datafusion-physical-optimizer.
There are some tests that have dependencies on datasource, which they are not possible to move out of core at this moment. We may need to leave those tests inside core for now.
- PhysicalOptimizerRule Create
datafusion-physical-optimizercrate #11507 - OutputRequirements
- AggregateStatistics
- Move
JoinSelectionintodatafusion-physical-optimizercrate #14073 - LimitedDistinctAggregation
- EnforceDistribution
- CombinePartialFinalAggregate
- EnforceSorting
- OptimizeAggregateOrder
- ProjectionPushdown
- CoalesceBatches
- OutputRequirements
- TopKAggregation
- Move
SanityCheckerintodatafusion-physical-optimizercrate #14072 - Move
ProjectionPushdownintodatafusion-physical-optimizercrate #14184 - Move
EnforceSortingintodatafusion-physical-optimizercrate #14185 - Move
EnforceDistributionintodatafusion-physical-optimizercrate #14186 - Move
PhysicalOptimizerand any remaining crates intodatafusion-physical-optimizercrate, remove datafusion/core/physical-optimizer
Describe alternatives you've considered
No response
Additional context
No response
Activity
[-][EPIC] Extract physical optimizer out of core[/-][+][EPIC] Extract remaining physical optimizer out of core[/+]jonathanc-n commentedon Nov 7, 2024
@jayzhan211 @lewiszlw How do we plan on moving the rest of the functions. If we move the tests separately many of the tests depend on the traits in the functions that are being moved. Due to this it would make sense to import together, but then problems with the datasource and other util dependencies will pop up again. Do you guys have any solutions in mind?
jayzhan211 commentedon Nov 8, 2024
We may need to move
datasourceout of core.We probably need to revisit this #10782
jayzhan211 commentedon Nov 8, 2024
Try to move optimization that doesn't have datasource dependency like
These are good first issues
Require datasource
If we only need datasource in test, leave the test in core
jonathanc-n commentedon Nov 8, 2024
@jayzhan211 Would you like to open a ticket for moving datasource out of core, so others are more aware of this transition?
jayzhan211 commentedon Nov 8, 2024
Not sure how do achieve this yet, the high level idea is tracked in #10782
josh0yeh commentedon Nov 9, 2024
take
alamb commentedon Jan 10, 2025
I filed some small tickets to get this project moving again:
SanityCheckerintodatafusion-physical-optimizercrate #14072JoinSelectionintodatafusion-physical-optimizercrate #14073alamb commentedon Jan 18, 2025
I filed a last few tickets
ProjectionPushdownintodatafusion-physical-optimizercrate #14184EnforceSortingintodatafusion-physical-optimizercrate #14185EnforceDistributionintodatafusion-physical-optimizercrate #14186Once those are moved I think we can move the last remnants of the physical optimizer and close this ticket
alamb commentedon Jan 26, 2025
We did it! Thank you @buraksenn @berkaysynnada and @logan-keede for pushing it the last little bit