Skip to content

Only compute bounds/ dynamic filters if consumer asks for it #17527

Closed
#19546
Closed
Only compute bounds/ dynamic filters if consumer asks for it#17527
#19546
Assignees
LiaCastaneda
Labels
bugSomething isn't working

Description

@LiaCastaneda
LiaCastaneda
opened on Sep 11, 2025

Describe the bug

Just a follow up to this comment.

Currently, DataFusion computes bounds for all queries that contain a HashJoinExec node whenever the option enable_dynamic_filter_pushdown is set to true (default). It might make sense to compute these bounds only when we explicitly know there is a consumer that will use them.

One way to achieve this could be during physical planning: while traversing the plan, check whether there is any scan/leaf node that is “interested in” or supports dynamic filters (determined by gather_filters_for_pushdown). This might just require adding some logic to the filter pushdown optimization rule itself I think?

Then, only if there is at least one interested consumer that accepts the DynamicFilterPhysicalExpr, set a flag on HashJoinExec to build the bounds accumulator, otherwise, skip bounds computation entirely.

To Reproduce

No response

Expected behavior

No response

Additional context

No response

Activity

  1. LiaCastaneda
    added
    bugSomething isn't working
    on Sep 11, 2025

  2. rkrishn7 commented on Sep 11, 2025

    @rkrishn7
    rkrishn7
    on Sep 11, 2025
    Contributor

    I think this is intentionally not handled in handle_child_pushdown_result 🤔

    // Note that we don't check PushdDownPredicate::discrimnant because even if nothing said
// "yes, I can fully evaluate this filter" things might still use it for statistics -> it's worth updating

    cc @adriangb

  3. adriangb commented on Sep 12, 2025

    @adriangb
    adriangb
    on Sep 12, 2025
    Contributor

    I think this is intentionally not handled in handle_child_pushdown_result 🤔

    // Note that we don't check PushdDownPredicate::discrimnant because even if nothing said
// "yes, I can fully evaluate this filter" things might still use it for statistics -> it's worth updating

    cc @adriangb

    Yep that's the case. For example Parquet may not be able to fully evaluate the filter for each row but may still be able to use it for file level and row group stats pruning.

    I think we have two options:

    1. Expand the pushdown result enum from two yes/no cases to three cases: exact (parent can drop the filter), inexact (parent should still apply the filter but it may be referenced for stats/partial pruning) and unsupported (the filter is not going to be used at all).
    2. Build in some notion of "subscribers" to dynamic filters where the thing that's going to use them has to "subscribe".

  4. LiaCastaneda commented on Sep 12, 2025

    @LiaCastaneda
    LiaCastaneda
    on Sep 12, 2025
    ContributorAuthor

    Yep that's the case. For example Parquet may not be able to fully evaluate the filter for each row but may still be able to use it for file level and row group stats pruning.

    Ah, I didn't acocunt for that

    Expand the pushdown result enum from two yes/no cases to three cases: exact (parent can drop the filter), inexact (parent should still apply the filter but it may be referenced for stats/partial pruning) and unsupported (the filter is not going to be used at all).

    maybe we could start with this? although the subscriber option would still be interesting, especially now that we’re considering multiple filter types, a scan node could subscribe to just the type of filter it supports 🤔

  5. adriangb commented on Sep 12, 2025

    @adriangb
    adriangb
    on Sep 12, 2025
    Contributor

    I think the pro of the subscriber option is:

    1. Like you say it can be applied to only some filters and not others.
    2. It avoids making the general filter pushdown API more complex (although the complexity gets pushed somewhere else).

  6. LiaCastaneda commented on Sep 14, 2025

    @LiaCastaneda
    LiaCastaneda
    on Sep 14, 2025
    ContributorAuthor

    I imagine the "subscribe" action would still be done by the consumer during optimization, and "publish" by has join during execution. If no one else is interested, I can take a stab at this one

  7. adriangb commented on Sep 14, 2025

    @adriangb
    adriangb
    on Sep 14, 2025
    Contributor

    That could be a good opportunity to broadcast updates

  8. adriangb commented on Sep 14, 2025

    @adriangb
    adriangb
    on Sep 14, 2025
    Contributor

    That is something like #17370

  9. adriangb commented on Nov 25, 2025

    @adriangb
    adriangb
    on Nov 25, 2025
    Contributor

    In #18644 (comment) it was proposed to add an is_used function that uses the Arc reference count.

  10. LiaCastaneda commented on Nov 26, 2025

    @LiaCastaneda
    LiaCastaneda
    on Nov 26, 2025
    ContributorAuthor

    take

  11. github-actions
    assigned
    LiaCastaneda
    on Nov 26, 2025
  12. github-merge-queue
    added a commit that references this issue on Dec 31, 2025

    Compute Dynamic Filters only when a consumer supports them (#19546)

    f1e5c94
  13. adriangb
    closed this as completedin #19546on Dec 31, 2025
  14. nuno-faria
    added a commit that references this issue on Aug 24, 2026

    fix: decide hash join dynamic filter production at planning time (apa…

    a62e84a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions