General virtual columns support + row numbers as a first use-case

[vustef]

Based on #7307.

Which issue does this PR close?

• Closes [Parquet] Support file row number in Parquet reader #7299

Rationale for this change

We need row numbers for many of the downstream features, e.g. computing unique row identifier in iceberg.

What changes are included in this PR?

New API to get row numbers as a virtual column:

let file = File::open(path).unwrap();
let row_number_field = Field::new("row_number", ArrowDataType::Int64, false).with_extension_type(RowNumber);
let options = ArrowReaderOptions::new().with_virtual_columns(vec![row_number_field]);
let reader = ParquetRecordBatchReaderBuilder::try_new_with_options(file, options)
    .unwrap()
    .build()
    .expect("Could not create reader");
reader
    .collect::<Result<Vec<_>, _>>()
    .expect("Could not read")
    ```

This column is defined as an extension type.
Parquet metadata is propagated to the array builder to compute first row indexes.
New Virtual column is included in addition to Primitive and Group.

Are these changes tested?

Yes

Are there any user-facing changes?

This is user facing feature, and has added docstrings.
No breaking changes, at least I tried not to, by creating a duplicate of public method to add more parameters.

[vustef]

TODO from original PR

[vustef]

this duplicates a lot, not sure if anything can be done here

[vustef]

the name used here is not aligned with what other convert_ functions do

[alamb]

Thanks again @jkylling @vustef and @etseidl

[vustef]

Thanks again @jkylling @vustef and @etseidl

It was my pleasure, thanks to you all from me as well.

[scovich]

Post-merge drive by review

[scovich]

I don't understand this part? The row groups were supplied in some particular order (by the row_groups iterator), and we're reordering by row group ordinal instead? Wouldn't that cause row number mismatches with other columns that continue reading in the original order? It seems like we actually need:

let selected_ordinals = HashMap<i16, usize> = row_groups
    .enumerate()
    .map(...)
    .collect::<Result<_>>()?;

and then ranges needs to use that enumeration ordinal (not the row group ordinal):

if let Some(i) = selected_ordinals.get(&ordinal) {
    ranges.push((i, ...);
}

... so that the sorted ranges match the original row_group iterator's order?

[vustef]

I messed this up when I started computing first row indexes...thank you for catching this. WIll follow up shortly with tests and the fix.

[scovich]

Suggested change

	}
	.map_or(false, |name| name.starts_with(VIRTUAL_PREFIX!()))

[scovich]

not a fan of panics, but if we're going to panic why not just

assert!(
    is_virtual_column(field), 
    "...", 
    field.name()
);

[vustef]

Me neither, but seemed like a unwritten rule that all these with_ methods in ArrowReaderOptions return Self rather than a Result. Please comment in the new PR if I should change that behaviour.

@alamb also checking for your opinion on this.

[etseidl]

Yes, the builder-like with_ functions don't give much opportunity for validity checking. We could probably use a proper ArrowReaderOptionsBuilder and do that kind of checking in build(). Or just change this one to a setter (so it's obvious it can't be chained) and return a Result<()>.

[alamb]

I think it is ok to return errors when trying to build options, like

let options = options.with_virtual_columns(cols)?;

If there is an error it is unlikely the code wants to continue configuring the options anyways