Simplified filter expression has a Null type instead of Int64 type across the FFI layer

Describe the bug

A custom table provider for a ParquetSource with a trivial catalog and an Int64 column yields some errors when a SQL query has a filter with a literal limit on that column of the form

assertion `left == right` failed: Simplified expression should have the same data type as the original
  left: Null
 right: Int64

The error does not occur when using datafusion-python 52; it also does not occur when running the query purely in a Rust SessionContext; the backtrace for the above error shows it coming from datafusion-ffi code as well.

This may of course not be a bug, but instead some bad practice that the version 52 set of crates tolerates but which is now invalid. A MRE of this custom table provider can be found in the public repo https://github.com/jwimberl/datafusion_python_53_int64filter_repro, which contains

a non-working datafusion53 version (in branch main)
a baseline working version (in branch datafusion52)

and a canned dummy dataset. The README.md of this repo has more details.

To Reproduce

In the main branch, build the py_repro_provider crate with maturin develop and run python repro.py. This loads the dummy dataset as a table dummy_table and runs two queries

SELECT * FROM dummy_table LIMIT 1, which is successful
SELECT * FROM dummy_table LIMIT 5, which panics

Itss output should be something like

Successful query:
   a   b
0  0  42
Unsuccesful query:

thread '<unnamed>' panicked at /home/jwimberley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/datafusion-physical-expr-53.1.0/src/simplifier/mod.rs:76:17:
assertion `left == right` failed: Simplified expression should have the same data type as the original
  left: Null
 right: Int64
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

followed by backtrace information.

Expected behavior

In the datafusion52 branch, build thepy_repro_provider with maturin develop and run python repro.py. It runs the same two queries, and its output should be

Successful query:
   a   b
0  0  42
Also successful query:
   a   b
0  0  42
1  1  42
2  2  42
3  3  42
4  4  42

Additional context

In either the main branch or datafusion52 branch, the Rust code for the table provider is in the directory repro_provider, and there is a corresponding cargo test that runs SELECT * FROM dummy_table WHERE a < 5. Without the FFI layer, this is successful with both datafusion 52 and 53.

@timsaucer

@timsaucer

Thanks! A quick look and I found that disabling the pushdown filters removes the problem, but I still haven't root caused why that's creating a problem with the optimizer.

I also found that there is no change (i.e. same error) from leaving the filter in place but replacing TableProviderFilterPushDown::Inexact with TableProviderFilterPushDown::Exact; not that I suspected this, just hunting for load bearing parts of the repro.

Ok, it looks like this is happening in the provider not in datafusion-python library as I had originally expected. What is happening is that we have a Column physical expression that is failing to simplify because simplify_const_expr_immediate cannot correctly downcast it to Column since it's foreign at that point.

This was an unintended side effect of apache/datafusion#18916

Here is what is happening:

The table provider is getting a push down filter expression (logical) and a '&dyn Session' that is the datafusion-python session context.
The ParquetSource it is using under the hood takes a PhysicalExpr as it's predicate. The PhysicalExpr is created by the datafusion-python session context during the call to create_physical_expr. This PhysicalExpr originates in the datafusion-python library so it is foreign in terms of the user library. That is, from the user library perspective it will be a ForeignPhysicalExpr
On the user library we are getting calls to simplify. It looks like this happens both in open and in row group filter (which is where we're hitting it here).
simplify checks to see if it is a Column during simplify_const_expr_immediate however it cannot downcast to Column because we are in the user library NOT the main datafusion-python library when this simplify gets called.

Here is a work around I have tested with this code

        let execution_props = ExecutionProps::new();
        let predicate = predicate
            .map(|predicate| {
                datafusion::physical_expr::create_physical_expr(
                    &predicate,
                    &df_schema,
                    &execution_props,
                )
            })
            .transpose()?
            // if there are no filters, use a literal true to have a predicate
            // that always evaluates to true we can pass to the index
            .unwrap_or_else(|| datafusion::physical_expr::expressions::lit(true));

I am working on an upstream issue that will resolve this in a more complete manner, but I hope that unblocks you for 53.

apache/datafusion#22367

I confirmed the workaround in the MRE and am now implementing and testing it in the more complex provider from which that was derived. What are the implications of creating the physical expression using the default execution properties instead of the session state? The docs say that the result of the SessionState::create_physical_expr call is

is not simplified or otherwise optimized

so I'd guess there's no custom optimizer rules that could come into play in it?

create_physical_expr does not do any simplification but the parquet provider is doing simplification.

Just confirming (unsurprisingly) that the workaround also works in the original context of the problem

Cool. I have some proper fixes going in upstream but probably won't land in time for 54. I'm sorry about the troubles.

fix: simplifier on leaf nodes returns null (apache#22368)

Not at all, thanks for the quick workaround and resolution!

Navigation Menu

Repository navigation

Simplified filter expression has a Null type instead of Int64 type across the FFI layer #1551

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Simplified filter expression has a Null type instead of Int64 type across the FFI layer #1551

Description

Activity

jwimberl commented on May 18, 2026

timsaucer commented on May 18, 2026

jwimberl commented on May 18, 2026

timsaucer commented on May 19, 2026

timsaucer commented on May 19, 2026

timsaucer commented on May 19, 2026

jwimberl commented on May 19, 2026

timsaucer commented on May 19, 2026

jwimberl commented on May 19, 2026

timsaucer commented on May 19, 2026

jwimberl commented on Jun 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions