Skip to content

Support merge RowSelectors when creating RowSelection #2858

Closed

Description

@Ted-Jiang

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

When i implementing https://github.com/apache/arrow-datafusion/pull/3780/files#r992227669
find out if we create

let mut selection = RowSelection::from(vec![
            RowSelector::skip(10),
            RowSelector::select(10),
            RowSelector::skip(5),
            RowSelector::skip(5),
        ]);

we have tree pages each have 10 rows.
Before introduced #2473, when skipping two RowSelector::skip(5):
the first one is not enough to skip the whole page, so we need read the page bytes to skip 5 values decoding.
But after #2473, parquet reader not fetch the third page's bytes according to the selection

Describe the solution you'd like

Add annotation at

impl From<Vec<RowSelector>> for RowSelection {
fn from(selectors: Vec<RowSelector>) -> Self {
Self { selectors }
}

and add one API call from_and_combine merge it into

            RowSelector::skip(10),
            RowSelector::select(10),
            RowSelector::skip(10),

Describe alternatives you've considered

Additional context

Activity

  1. Ted-Jiang commented on Oct 11, 2022

    @Ted-Jiang
    MemberAuthor
  2. alamb commented on Oct 13, 2022

    @alamb
    Contributor

    @Ted-Jiang I remember some code that combines RowSelectors somewhere -- perhaps https://docs.rs/parquet/24.0.0/parquet/arrow/arrow_reader/struct.RowSelection.html has some inspiration

  3. alamb commented on Nov 11, 2022

    @alamb
    Contributor

    label_issue.py automatically added labels {'parquet'} from #2994

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature/additionparquetChanges to the parquet crate

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions