Support Boolean Parquet Data Page Statistics

@Weijun-H

Is your feature request related to a problem or challenge?

We are adding APIs to efficiently convert the data stored in Parquet's "PageIndex" into ArrayRefs -- which will make it significantly easier to use this information for pruning and other tasks.

Describe the solution you'd like

Add support to StatisticsConverter::min_page_statistics and StatisticsConverter::max_page_statistics for the types above

datafusion/datafusion/core/src/datasource/physical_plan/parquet/statistics.rs

Lines 637 to 656 in a923c65

    
           /// of parquet page [`Index`]'es to an [`ArrayRef`] 
        
           pub(crate) fn min_page_statistics<'a, I>( 
        
               data_type: Option<&DataType>, 
        
               iterator: I, 
        
           ) -> Result<ArrayRef> 
        
           where 
        
               I: Iterator<Item = (usize, &'a Index)>, 
        
           { 
        
               get_data_page_statistics!(Min, data_type, iterator) 
        
           } 
        
           /// Extracts the max statistics from an iterator 
        
           /// of parquet page [`Index`]'es to an [`ArrayRef`] 
        
           pub(crate) fn max_page_statistics<'a, I>( 
        
               data_type: Option<&DataType>, 
        
               iterator: I, 
        
           ) -> Result<ArrayRef> 
        
           where 
        
               I: Iterator<Item = (usize, &'a Index)>, 
        
           {

Describe alternatives you've considered

You can follow the model from @Weijun-H in #10931

Update the test for the listed data types following the model of test_int64

datafusion/datafusion/core/tests/parquet/arrow_statistics.rs

Lines 506 to 529 in a923c65

    
           async fn test_int_64() { 
        
               // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64" 
        
               let reader = TestReader { 
        
                   scenario: Scenario::Int, 
        
                   row_per_group: 5, 
        
               } 
        
               .build() 
        
               .await; 
        
               // since each row has only one data page, the statistics are the same 
        
               Test { 
        
                   reader: &reader, 
        
                   // mins are [-5, -4, 0, 5] 
        
                   expected_min: Arc::new(Int64Array::from(vec![-5, -4, 0, 5])), 
        
                   // maxes are [-1, 0, 4, 9] 
        
                   expected_max: Arc::new(Int64Array::from(vec![-1, 0, 4, 9])), 
        
                   // nulls are [0, 0, 0, 0] 
        
                   expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]), 
        
                   // row counts are [5, 5, 5, 5] 
        
                   expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]), 
        
                   column_name: "i64", 
        
                   check: Check::Both, 
        
               } 
        
               .run();

Add any required implementation in

datafusion/datafusion/core/src/datasource/physical_plan/parquet/statistics.rs

Lines 575 to 586 in 2f43476

    
           make_data_page_stats_iterator!(MinInt64DataPageStatsIterator, min, Index::INT64, i64); 
        
           make_data_page_stats_iterator!(MaxInt64DataPageStatsIterator, max, Index::INT64, i64); 
        
           macro_rules! get_data_page_statistics { 
        
               ($stat_type_prefix: ident, $data_type: ident, $iterator: ident) => { 
        
                   paste! { 
        
                       match $data_type { 
        
                           Some(DataType::Int64) => Ok(Arc::new(Int64Array::from_iter([<$stat_type_prefix Int64DataPageStatsIterator>]::new($iterator).flatten()))), 
        
                           _ => unimplemented!() 
        
                       } 
        
                   } 
        
               }

(follow the model of the row counts,

datafusion/datafusion/core/src/datasource/physical_plan/parquet/statistics.rs

Line 90 in 2f43476

macro_rules! make_stats_iterator {

)

Additional context

No response

Boolean type is supported, it was added in commit 9845e6e of PR #10711

Sorry for the noise

Actually I don't think this is actually done

This ticket covers extracting DataPage statistics (not row group statistics, which are annoyingly different in parquet 🤯 )

The data page statistics are extracted here

datafusion/datafusion/core/src/datasource/physical_plan/parquet/statistics.rs

Lines 612 to 627 in 18042fd

    
           macro_rules! get_data_page_statistics { 
        
               ($stat_type_prefix: ident, $data_type: ident, $iterator: ident) => { 
        
                   paste! { 
        
                       match $data_type { 
        
                           Some(DataType::UInt8) => Ok(Arc::new( 
        
                               UInt8Array::from_iter( 
        
                                   [<$stat_type_prefix Int32DataPageStatsIterator>]::new($iterator) 
        
                                       .map(|x| { 
        
                                           x.into_iter().filter_map(|x| { 
        
                                               x.and_then(|x| u8::try_from(x).ok()) 
        
                                           }) 
        
                                       }) 
        
                                       .flatten() 
        
                               ) 
        
                           )), 
        
                           Some(DataType::UInt16) => Ok(Arc::new(

In order to complete this issue, we need to change

datafusion/datafusion/core/tests/parquet/arrow_statistics.rs

Line 1956 in 58d23c5

check: Check::RowGroup,

to

 check: Check::Both,

And make the tests pass

Oh sorry, that was stupid of me.

Oh sorry, that was stupid of me.

No worries at all -- this stuff is tricky

Yeah, all those similar names do get to me sometimes...

On another note, I tried to implement this like all the others did, but the test fails with :

thread 'parquet::arrow_statistics::test_boolean' panicked at src/array/boolean_array.rs:407:33:
Iterator must be sized

The implementation is like this:

make_data_page_stats_iterator!(
    MinBooleanDataPageStatsIterator,
    |x: &PageIndex<bool>| { x.min },
    Index::BOOLEAN,
    bool
);
make_data_page_stats_iterator!(
    MaxBooleanDataPageStatsIterator,
    |x: &PageIndex<bool>| { x.max },
    Index::BOOLEAN,
    bool
);
...
macro_rules! get_data_page_statistics {
    ($stat_type_prefix: ident, $data_type: ident, $iterator: ident) => {
        paste! {
            match $data_type {
                Some(DataType::Boolean) => Ok(Arc::new(
                    BooleanArray::from_iter(
                        [<$stat_type_prefix BooleanDataPageStatsIterator>]::new($iterator).flatten()
                    )
                )),
       ...
}

These macros, functions, and tests jump around a lot before I get to the caller, which causes this panic. Do you or anyone else know why this happens?

The iterator must be sized thing comes from arrow -- one workaround is to collect the values into a Vec first and then create the array

I don't know why boolean is different than the other data page types 🤔

take

Navigation Menu

Repository navigation

Support `Boolean` Parquet Data Page Statistics #11027

Is your feature request related to a problem or challenge?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	/// of parquet page [`Index`]'es to an [`ArrayRef`]
	pub(crate) fn min_page_statistics<'a, I>(
	data_type: Option<&DataType>,
	iterator: I,
	) -> Result<ArrayRef>
	where
	I: Iterator<Item = (usize, &'a Index)>,
	{
	get_data_page_statistics!(Min, data_type, iterator)
	}

	/// Extracts the max statistics from an iterator
	/// of parquet page [`Index`]'es to an [`ArrayRef`]
	pub(crate) fn max_page_statistics<'a, I>(
	data_type: Option<&DataType>,
	iterator: I,
	) -> Result<ArrayRef>
	where
	I: Iterator<Item = (usize, &'a Index)>,
	{

	async fn test_int_64() {
	// This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
	let reader = TestReader {
	scenario: Scenario::Int,
	row_per_group: 5,
	}
	.build()
	.await;

	// since each row has only one data page, the statistics are the same
	Test {
	reader: &reader,
	// mins are [-5, -4, 0, 5]
	expected_min: Arc::new(Int64Array::from(vec![-5, -4, 0, 5])),
	// maxes are [-1, 0, 4, 9]
	expected_max: Arc::new(Int64Array::from(vec![-1, 0, 4, 9])),
	// nulls are [0, 0, 0, 0]
	expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
	// row counts are [5, 5, 5, 5]
	expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
	column_name: "i64",
	check: Check::Both,
	}
	.run();

	make_data_page_stats_iterator!(MinInt64DataPageStatsIterator, min, Index::INT64, i64);
	make_data_page_stats_iterator!(MaxInt64DataPageStatsIterator, max, Index::INT64, i64);

	macro_rules! get_data_page_statistics {
	($stat_type_prefix: ident, $data_type: ident, $iterator: ident) => {
	paste! {
	match $data_type {
	Some(DataType::Int64) => Ok(Arc::new(Int64Array::from_iter([<$stat_type_prefix Int64DataPageStatsIterator>]::new($iterator).flatten()))),
	_ => unimplemented!()
	}
	}
	}

Support Boolean Parquet Data Page Statistics #11027

Description

Is your feature request related to a problem or challenge?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Activity

LorrensP-2158466 commented on Jun 20, 2024

alamb commented on Jun 20, 2024

alamb commented on Jun 21, 2024

LorrensP-2158466 commented on Jun 21, 2024

alamb commented on Jun 21, 2024

LorrensP-2158466 commented on Jun 21, 2024

alamb commented on Jun 21, 2024

LorrensP-2158466 commented on Jun 21, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Support `Boolean` Parquet Data Page Statistics #11027