Repository navigation
feature: adapt predicate pushdown for mismatched nested/struct schemas #16565
Description
Activity
incorporating the struct-aware casting logic from #16371 into
CastExprandTryCastExprYes. I think it is a necessary step to explore the feasibility of rewriting Expr for schema adaptation.
A few thoughts on this:
-
Reusability of struct-aware casting kernels: The core struct-aware casting kernel introduced in Add nested struct casting support and integrate into SchemaAdapter #16371 was designed to recursively handle casting between nested struct types, dealing with missing or reordered fields. Integrating this logic into
CastExprandTryCastExprwould unify casting semantics across physical expressions, ensuring consistent behavior during predicate rewrite and evaluation. -
Improved predicate pushdown: Predicate pushdown often requires projecting predicates down into struct subfields. If schemas differ between scans (e.g., fields missing or reordered), leveraging struct-aware casts would allow predicates to be adapted instead of dropped—thus generating more selective filters pushed down to data sources.
-
Implementation considerations: We would need to extend
CastExprandTryCastExprto detect when the input/output types are structs and invoke the struct-aware casting logic appropriately. This means:- Adding recursive traversal logic in these expressions.
- Handling missing subfields by either filling with NULLs or using default values as per the kernel's design.
- Handling field reordering and type promotion gracefully.
-
Challenge: Rewriting the struct-aware casting logic from Add nested struct casting support and integrate into SchemaAdapter #16371 into manipulating expressions. Your expression rewrite prowess bends my mind.
-
Question: will we be able to handle something like
Dict(UInt32, List(List(Struct(...))))?Implementation considerations: We would need to extend CastExpr and TryCastExpr to detect when the input/output types are structs and invoke the struct-aware casting logic appropriately. This means:
@kosiew I would actually suggest something different which is to add a function that "wraps" the arrow cast kernel
So it would look somethin glike
pub fn cast( array: &dyn Array, to_type: &DataType, ) -> Result<Arc<dyn Array>, ArrowError> { if matches!(to_type, DataType::Struct) { // call struct aware version in datafusion } else { // fallback to arrow implementatin arrow::compute::kernels::cast(array, to_type) } }
Then you need to update all the places that directly call arrow cast with this new cast version, but the upside is that now everything would have the same (consistent) semantics
Reacted by Adrian Garcia Badaracco and kosiewI agree with you @alamb.
The issue I see with both approaches is going to be nested types: once you call the arrow cast kernel you can't take back control, so this approach won't work for
List(Struct)and such. We'd essentially have to handle all non-leaf types in our wrapper and only delegate to arrow for the leaf types that can't possibly contain a struct / other type we want to handle.Reacted by kosiew and Andrew LambI agree with you @alamb.
The issue I see with both approaches is going to be nested types: once you call the arrow cast kernel you can't take back control, so this approach won't work for
List(Struct)and such. We'd essentially have to handle all non-leaf types in our wrapper and only delegate to arrow for the leaf types that can't possibly contain a struct / other type we want to handle.I think delegating to arrow cast for leaf types makes sense, given its limitations casting nested types
As per discussion in #16461 (comment) we should be able to have
datafusion/physical-expr/src/schema_rewriter.rshandle missing sub-fields in struct columns.