You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Consider an enclosing-range fallback for IN lists above max_in_list_size #24708
Today an IN list longer than datafusion.execution.parquet.max_in_list_size falls through to the unhandled-predicate hook, so it contributes no container pruning at all. (Other predicates and literal/containment pruning, including Bloom filters, still apply.)
A much weaker form is available at close to no cost: collapse the list to its extrema and test interval overlap, roughly vmin <= col_max AND col_min <= vmax. Constant-size predicate, a couple of comparisons per container, and finding the extrema is one O(N) pass with no sort. It also wouldn't need a custom PhysicalExpr or the string-only restriction, so it could apply to any orderable type.
The tradeoff is that it can only rule out containers lying entirely outside the hull — it can never prune a gap, which is exactly the precision the compact form exists to preserve. How much it's worth probably depends on how clustered the requested values are relative to the data's ordering: for a set spread across the whole domain it prunes nothing, while for clustered or time-correlated identifiers (ULIDs came up in review) it may capture much of the available win.
Open questions:
Is this worth having as a tier below the cap's current give-up behaviour?
If so, what policy? It changes the current eligibility contract, and should presumably still respect max_in_list_size = 0 as a full opt-out.
Hello, I’m interested in investigating this. Would a small prototype with benchmarks for clustered and scattered value sets help decide whether to proceed?! I’d preserve max_in_list_size = 0 as disabled and initially focus on positive literal IN lists.
Activity
ShayanGho commentedon Sep 5, 2026
Hello, I’m interested in investigating this. Would a small prototype with benchmarks for clustered and scattered value sets help decide whether to proceed?! I’d preserve max_in_list_size = 0 as disabled and initially focus on positive literal IN lists.