You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native Parquet scan fails on an empty file whose nested field type differs from the read schema #6506
Since #5681, the native Parquet scan checks the type conversion for every nested field when it opens a file, while Spark checks it only when it decodes a value. So a file with nothing to decode, either empty or with every row group pruned, whose nested field has a type that the read schema can't convert, now fails the query with SchemaColumnConvertNotSupportedException. Spark returns the rows from the other files, and so did 1.0.0.
When the file does have rows to decode, 1.1.0 now fails the way Spark does. That part is a fix: 1.0.0 returned null for the value it couldn't convert.
Steps to reproduce
withTempPath { dir =>valpath= dir.getCanonicalPath
spark.sql("select named_struct('x', 1) as s").write.parquet(path)
// An empty file whose nested field is a string
spark.sql("select named_struct('x', 'a') as s where false").write.mode("append").parquet(path)
checkSparkAnswer(spark.read.schema("s struct<x:int>").parquet(path))
}
Spark and 1.0.0 return [[1]]. 1.1.0-rc1 fails when it opens the empty file, for column [s, x], BINARY to int. A file whose only row group a filter prunes fails the same way, for example a nested decimal(10,4) read as decimal(10,2) with WHERE id = 100.
Expected behavior
The rows Spark returns.
Additional context
Found by the 1.1.0 regression audit (#6399) and tracked in #6402.
Activity
docs: list the _metadata block and empty-file regressions (apache#6505,
refactor: decide open-time versus decode-time Parquet rejection in on…
fix: defer Parquet conversion errors until a row group is decoded, as…
fix: defer Parquet conversion errors until a row group is decoded, as…
docs: drop the DSv2 decimal and empty-file regressions, which branch-…