EPIC: use cp_solver framework to develop a more sophisticated predicate simplification #16511

Activity

1. xudong963

   added

   enhancementNew feature or requestNew feature or request

   on Jun 23, 2025
2. adriangb commented on Jun 24, 2025

   

   adriangb

   on Jun 24, 2025

   Contributor

   More actions

   Make it a new physical optimizer rule? Or add them to the current physical filter push down

   Do we need a phyiscal optimizer rule? My thought is that these optimizations can happen at the logical layer. At the phyiscal layer we do need some expression optimization, but primarily to deal with type casting to match the file schema, etc (i.e. #16530), expressions at this point should be already simplified from the logical layer and that doesn't really change except maybe in some edge cases which I'm not sure are worth dealing with.

   Some filters, such as partition columns related predicates, will be fully pushed down scan, that is, we don't have such predicates in some FilterExec. How to capture this?

   Not sure I understood this part
3. xudong963 commented on Jun 26, 2025

   

   xudong963

   on Jun 26, 2025

   Last edited by xudong963

   MemberAuthor

   More actions

   My thought is that these optimizations can happen at the logical layer

   Yes, almost all DBs do the optimization in logical layer, but as @alamb said, we want to use the cp solver framework to make it general(cp solver is in the physical layer).

   Not sure I understood this part

   In logical filter pushdown, the partition-related filters will be fully pushed down to scan(they are Exact), see code here:
   https://github.com/apache/datafusion/blob/main/datafusion/optimizer/src/push_down_filter.rs#L1128-L1153
4. alamb commented on Jun 26, 2025

   

   alamb

   on Jun 26, 2025

   Contributor

   More actions

   Even though the cp_solver is done at the physical layer, I think the overall simplification can still be done in an Optimizer that targets LogicalPlans -- much like the expression simplifier works, we would convert to a physical expr during planning:

   datafusion/datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs

   Lines 674 to 678 in 3649dc8

   	let phys_expr =
   	match create_physical_expr(&expr, &self.input_schema, self.execution_props) {
   	Ok(e) => e,
   	Err(err) => return ConstSimplifyResult::SimplifyRuntimeError(err, expr),
   	};
5. alamb

   added

   EPICA larger project, actively underway, with sub tasksA larger project, actively underway, with sub tasks

   PROPOSAL EPICA proposal being discussed that is not yet fully underwayA proposal being discussed that is not yet fully underway

   and removed

   EPICA larger project, actively underway, with sub tasksA larger project, actively underway, with sub tasks

   on Aug 12, 2025