Repository navigation
Support parquet page filtering for more types: String, Binary(Decimal), Int96 #3833
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
(This section helps Arrow developers understand the context and why for this feature, in addition to the what)
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Activity
[-]Support more types for getting min max from page index[/-][+]Support parquet page filtering for more types (for getting min max from from strings)[/+]alamb commentedon Nov 7, 2022
Here is a PR for filtering on strings: #4132
[-]Support parquet page filtering for more types (for getting min max from from strings)[/-][+]Support parquet page filtering for more types: String, Binary(Decimal), Int96[/+]alamb commentedon Nov 26, 2022
Int96 appears to be the only unsupported type left
https://github.com/apache/arrow-datafusion/blob/ba73c8180ebd874614cabc33be8cbb0d1db52518/datafusion/core/src/physical_plan/file_format/parquet/page_filter.rs#L478-L481
I don't think Int96 is a widely used type nor supported for most other implementation -- thus I suggest we close this issue
Ted-Jiang commentedon Nov 26, 2022
@alamb Agree, Int96 also not supported in row_group prunning
https://github.com/alamb/arrow-datafusion/blob/d7a7fb61afe9ce2824aae737f65aec12d9513f7f/datafusion/core/src/physical_plan/file_format/parquet/row_groups.rs#L124