Physical plan pushdown for volatile predicates

Describe the bug

This is a follow-up to a discussion in #16325 (comment), which is not directly related to table sampling but could affect it.

I'd like to double-check if a volatile filter pushdown to a Parquet executor is expected. I had implemented the disabling of volatile pushdown filters for a logical plan in #13268. But it seems like the physical optimiser still pushes this predicate to an executor. Should we implement a similar mechanism to make volatile predicates as unsupported filters? In a current physical plan implementation, there is a concept of "unsupported" filters, which can be easily reused for it.

Current behaviour:

Before:

[2025-06-18T18:20:07Z TRACE datafusion::physical_planner] Optimized physical plan by LimitedDistinctAggregation:
    OutputRequirementExec
      ProjectionExec: expr=[count(Int64(1))@0 as count(*)]
        AggregateExec: mode=Final, gby=[], aggr=[count(Int64(1))]
          AggregateExec: mode=Partial, gby=[], aggr=[count(Int64(1))]
            FilterExec: random() < 0.1
              DataSourceExec: file_groups={1 group: [[sample.parquet]]}, file_type=parquet

After:

[2025-06-18T18:20:07Z TRACE datafusion::physical_planner] Optimized physical plan by FilterPushdown:
    OutputRequirementExec
      ProjectionExec: expr=[count(Int64(1))@0 as count(*)]
        AggregateExec: mode=Final, gby=[], aggr=[count(Int64(1))]
          AggregateExec: mode=Partial, gby=[], aggr=[count(Int64(1))]
            DataSourceExec: file_groups={1 group: [[sample.parquet]]}, file_type=parquet, predicate=random() < 0.1

To Reproduce

set datafusion.execution.parquet.pushdown_filters=true;
create external table data stored as parquet location 'sample.parquet';
SELECT count(*) FROM data WHERE random() < 0.1;

Expected behavior

I expect the physical plan optimiser doesn't perform pushdown of volatile predicates.

Additional context

No response

@findepi

@findepi could you please tell if this behaviour makes sense? If not, I could try fixing the physical plan as in #13268, where you have a review

@alamb

@alamb @adriangb, given your most recent work on pushdowns, does it appear to be a bug?

I expect the physical plan optimiser doesn't perform pushdown of volatile predicates.

I am not sure -- does this result in wrong results?

It does make sense in general to have the physical pushdown optimizer follow the same behavior as the logical optimizer

I expect the physical plan optimiser doesn't perform pushdown of volatile predicates.

I am not sure -- does this result in wrong results?

We don't observe incorrect results (as was the case with the logical optimiser bug when evaluating a predicate twice).

It can produce skewed or non-reproducible results if a volatile predicate evaluates to different values for different row groups (I can create a test to reproduce it).

It could probably affect the table sampling implementation #16325 , which relies on inserting a volatile filter.

It does make sense in general to have the physical pushdown optimizer follow the same behavior as the logical optimizer

Yes, it is a reasonable assumption about an optimiser.

It seems reasonable to me to blacklist some filters. But given that users can create arbitrary trait implementations for PhysicalExpr we obviously can't black list everything. Unless there a volatility method on the trait?

@adriangb

Thank you for the improvement, @adriangb !

Navigation Menu

Repository navigation

Physical plan pushdown for volatile predicates #16545

Describe the bug

To Reproduce

Expected behavior

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Physical plan pushdown for volatile predicates #16545

Description

Describe the bug

To Reproduce

Expected behavior

Additional context

Activity

theirix commented on Jul 2, 2025

theirix commented on Jul 19, 2025

alamb commented on Jul 21, 2025

theirix commented on Jul 22, 2025

adriangb commented on Jul 22, 2025

theirix commented on Jul 30, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions