Repository navigation
Reading parquet with (pre-release) arrow fails with "out of order projection is not supported" #2543
Description
Describe the bug
When testing using a pre-release version of arrow-rs with apache/arrow-rs#1682 from @tustvold which added stricter error checking two datafusion tests fail with "out of order projection is not supported"
---- physical_plan::file_format::parquet::tests::evolved_schema_filter stdout ----
thread 'physical_plan::file_format::parquet::tests::evolved_schema_filter' panicked at 'called `Result::unwrap()` on an `Err` value: ArrowError(ExternalError(ParquetError(General("out of order projection is not supported"))))', datafusion/core/src/physical_plan/file_format/parquet.rs:968:14
---- physical_plan::file_format::parquet::tests::evolved_schema_inconsistent_order stdout ----
thread 'physical_plan::file_format::parquet::tests::evolved_schema_inconsistent_order' panicked at 'called `Result::unwrap()` on an `Err` value: ArrowError(ExternalError(ParquetError(General("out of order projection is not supported"))))', datafusion/core/src/physical_plan/file_format/parquet.rs:819:14
To Reproduce
See #2530
Expected behavior
Tests should succeed
Additional context
@tustvold suggests there is some interplay of the parquet schema adapter logic that was silently masking a bug in arrow-rs. Now that arrow-rs has stricter error checking, the query is erroring #2530 (comment)
Activity
alamb commentedon Jul 29, 2022
This has since been fixed