Repository navigation
Statistics: Migrate to Distribution from Precision #14896
Description
Is your feature request related to a problem or challenge?
DataFusion doesn't have a great statistics infrastructure, which will be long project to fix. Luckily, we have begun the process to implement an extensible statistics infrastructure, and we have already merged a new low-level framework to evaluate expressions in a probabilistic context (i.e. the evaluate_statistics and propagate_statistics API introduced in #14699). The next step in this project is to switch from the old Precision object to the new Distribution object across the DataFusion codebase. This way, we will be able to use the existing mechanism around the ColumnStatistics and Statistics (maybe we should rename this to TableStatistics?) objects to handle statistics.
Describe the solution you'd like
Deprecating/removing Precision, updating its usages with Distribution.
Describe alternatives you've considered
N/A.
Additional context
No response
Activity
[-]Statistics: Migrate to Distribution from Precision[/-][+]Statistics: Migrate to `Distribution` from `Precision`[/+]berkaysynnada commentedon Feb 27, 2025
Not just Precision, in some parts of the code,
Intervalobjects could(should) be also converted to Distribution objects.clflushopt commentedon Mar 2, 2025
Following our discussion from before and since I recently documented some parts of the old statistics API I would be happy to take this if it's not taken cc @alamb for vis.
alamb commentedon Mar 3, 2025
Thank you @clflushopt
Given what I have seen so far I think you would do a great job at it and I would be very happy to help find time to review your PRs (though I realize my history so far may not make that obvious). I think getting these APIs right would be super valuable
cj-zhukov commentedon Mar 3, 2025
I had a discussion with @ozankabak about it and I'm currently working on it now.
@clflushopt I'm afraid I should have mentioned this.
cj-zhukov commentedon Mar 3, 2025
take
clflushopt commentedon Mar 3, 2025
Sounds good @cj-zhukov thanks for taking this ! I haven't started working on it yet so that's perfect.
Statistics: Migrate to Distribution from Precision (apache#14896)
Statistics: Migrate to Distribution from Precision (apache#14896)
Statistics: Migrate to Distribution from Precision (apache#14896)
Statistics: Migrate to Distribution from Precision (apache#14896)
Statistics: Migrate to Distribution from Precision (apache#14896)
Statistics: Migrate to Distribution from Precision (apache#14896)
3 remaining items
Statistics: Migrate to Distribution from Precision (apache#14896)
Statistics: Migrate to Distribution from Precision (apache#14896)
Statistics: Migrate to Distribution from Precision (apache#14896)
Statistics: Migrate to Distribution from Precision (apache#14896)
Statistics: Migrate to Distribution from Precision (apache#14896)
Statistics: Migrate to Distribution from Precision (apache#14896)
Statistics: Migrate to Distribution from Precision (apache#14896)
Statistics: Migrate to Distribution from Precision (apache#14896)
Statistics: Migrate to Distribution from Precision (apache#14896)
cj-zhukov commentedon Jul 7, 2025
I’m unassigning myself from this issue for now as I haven’t been able to make progress.
cj-zhukov commentedon Jul 7, 2025
unassign me
alamb commentedon May 8, 2026
In case anyone else is following along, I am proposing to deprecate the V2 framework as it doesn't seem to have gained traction. Please comment if you have thoughts:
deprecate: mark Statistics V2 framework (PR apache#14699) as deprecat…
deprecate: mark Statistics V2 framework (PR apache#14699) as deprecat…