Repository navigation
feat: Extend single-value NDV optimization to string types #21109
Description
Is your feature request related to a problem or challenge?
In #21077, we set distinct_count to Exact(1) when a filter predicate collapses a column interval to a single value. This works for numeric types (Int32, Float64, etc.) but not for strings (Utf8, Utf8View, LargeUtf8).
The root cause is that next_value_helper() in interval_arithmetic.rs has no implementation for string types - it returns the value unchanged, which prevents the constraint propagation solver from collapsing the interval.
For example, WHERE name = 'hello' should produce NDV=Exact(1) but currently stays at the original Inexact NDV.
Describe the solution you'd like
Detect equality predicates directly in collect_new_statistics (pattern-match BinaryExpr with Operator::Eq + Column + Literal), bypassing interval analysis for the NDV computation.
Describe alternatives you've considered
Implement next_value for strings in interval_arithmetic.rs. This is non-trivial since string ordering depends on locale/collation (see PostgreSQL's collation support for reference), making a universal "next string" ill-defined. That said, it would benefit other interval-based optimizations as well.
Additional context
- Part of EPIC: Making use of NDVs (number of distinct values) in DataFusion #20766
- Related PR: feat: Set NDV to Exact(1) for numeric equality filter predicates #21077 (NDV=Exact(1) for numeric types)
Activity
buraksenn commentedon Mar 23, 2026
I would like to work on this if you're not already working
asolimando commentedon Mar 23, 2026
Thanks @buraksenn for your interest, please go ahead!
FYI: I have filed #21111 too, similar in spirit
[-]feat: Extend single-value NDV optimization to string and non-numeric types[/-][+]feat: Extend single-value NDV optimization to string types[/+]buraksenn commentedon Mar 23, 2026
Then I'm also taking #21111 since they should follow similar approach
buraksenn commentedon Mar 23, 2026
take
buraksenn commentedon Apr 6, 2026
I'm waiting on #21077 to be merged btw @asolimando to not have some conflicts. I think I'm pretty close with both issues ( #21111 and this) will open PRs when your PR is merged. Just wanted to give update
asolimando commentedon Apr 7, 2026
Thanks for the update @buraksenn, makes sense, FYI #21077 has just been merged
buraksenn commentedon Apr 8, 2026
Thanks @asolimando for the heads up, I've opened the PR just now. Btw I want to mention that I can take a look to any follow-ups to these kind of issues I've enjoyed working on these and would love to help. Feel free to assign me or mention me for the ones you don't have bandwidth for. I would really appreciate it
asolimando commentedon Apr 8, 2026
Thanks, I will make sure to loop you in, happy to keep working on improving statistics too (code or reviews), feel free to ping me where you think I can help!
feat: extend single ndv optimization to non-arithmetic supporting typ…
feat: extend interval analysis support for temporal types (#21520)
feat: extend single ndv optimization to non-arithmetic supporting typ…
feat: extend interval analysis support for temporal types (apache#21520)