Repository navigation
Logical optimizer rule "simplify expressions" should not depend on the core datafusion crate #2535
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
In the "simplify expressions" rule, we evaluate and fold expressions that result in scalar values and we depend on the physical expression crate to do this, which is fine, but we also depend on the core crate for create_physical_expr and we can't have the new datafusion-optimizer crate depend on core.
Describe the solution you'd like
It looks like we should be able to move create_physical_expr and associated functions to the datafusion-physical-expr crate.
Describe alternatives you've considered
None
Additional context
This is related to #2345
Activity
[-]Logical optimizer rule "simplify expressions" should not depend on physical expressions[/-][+]Logical optimizer rule "simplify expressions" should not depend on physical planner and expressions[/+]alamb commentedon May 24, 2022
I think it is important for expression simplification that expressions can be partially evaluated at optimization time.
In terms of evaluating on logical plan
Exprs directly, I agree there is no theoretical reason this can not be done, but if we did so it will result in dual expression evaluation systems that would have be kept in sync. This might be especially hard if someone wanted to use a different physical expression evaluation systemI understand the desire to remove the direct code dependence on the execution engine from the optimizer -- perhaps we can do so via a trait -- for example, we could pass some sort of
dyn ExprEvaluatorto the optimizer for use insimplify expressionsand elsewhere.By default, DataFusion could provide the evaluator based on
PhysicalExpr-- and for those users of the optimizer itself they could decide the most appropriate way to provide expression evaluation to the optimizerandygrove commentedon Jun 2, 2022
@alamb Now that I have started looking at this more closely it turns out to be less of an issue than I first thought. We just need to move
create_physical_exprfrom core to thephysical-exprcrate. I have updated the issue description.[-]Logical optimizer rule "simplify expressions" should not depend on physical planner and expressions[/-][+]Logical optimizer rule "simplify expressions" should not depend on the core datafusion crate[/+]