Repository navigation
Native Iceberg scan fails when a data file omits an optional nested field #7033
Description
Describe the bug
The native Iceberg scan fails any query that reads a struct column, or a list of structs, when a data file omits one of its optional nested fields, even though the table schema never changed. Spark reads the missing field as NULL. Iceberg accepts such files: its write compatibility check allows a missing optional field at any depth, so a Spark write with write.spark.accept-any-schema=true produces one, and so can other engines.
The scan fails with Incorrect number of arrays for StructArray fields, expected 2 got 1, the error #6504 reported for files written before a nested field was added. #6543 added a CometScanRule fallback for that case, but IcebergReflection.nestedFieldsAddedOrRenamed compares the projected fields against the table's schema history. Here no schema in the history lacks the field, so the scan stays native. Reading the iceberg-rust code at the current pin, RecordBatchTransformer passes the file's narrower struct through, and arrow's struct cast then matches children by position and fails.
Steps to reproduce
In a CometTestBase suite with a Hadoop Iceberg catalog cat and spark.comet.scan.icebergNative.enabled=true:
spark.sql("""CREATE TABLE cat.db.t (id INT, s STRUCT<a: INT, b: INT>, l ARRAY<STRUCT<x: INT, y: INT>>)
USING iceberg TBLPROPERTIES ('format-version'='2', 'write.spark.accept-any-schema'='true')""")
val full = StructType(Seq(
StructField("id", IntegerType),
StructField("s", StructType(Seq(StructField("a", IntegerType), StructField("b", IntegerType)))),
StructField("l", ArrayType(StructType(Seq(StructField("x", IntegerType), StructField("y", IntegerType)))))))
val narrow = StructType(Seq(
StructField("id", IntegerType),
StructField("s", StructType(Seq(StructField("a", IntegerType)))),
StructField("l", ArrayType(StructType(Seq(StructField("x", IntegerType)))))))
withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
spark.createDataFrame(spark.sparkContext.parallelize(Seq(Row(1, Row(1, 2), Seq(Row(1, 2)))), 1), full)
.writeTo("cat.db.t").append()
// Iceberg accepts a file that omits the optional fields s.b and l.element.y.
spark.createDataFrame(spark.sparkContext.parallelize(Seq(Row(2, Row(20), Seq(Row(20)))), 1), narrow)
.writeTo("cat.db.t").append()
}
spark.sql("SELECT id, s FROM cat.db.t").collect()On main at 66c63bd (Spark 4.1.3, Iceberg 1.11):
| Query | Spark | Comet native scan |
|---|---|---|
SELECT id, s |
[1,[1,2]] [2,[20,null]] |
Iceberg scan error: Unexpected => Arrow Schema Error, source: Invalid argument error: Incorrect number of arrays for StructArray fields, expected 2 got 1 |
SELECT id, s.b |
[1,2] [2,null] |
the same error |
SELECT id, l |
[1,[[1,2]]] [2,[[20,null]]] |
the same error |
SELECT * |
the rows above | the same error |
Expected behavior
The native scan reads a missing optional nested field as NULL, as Spark does.
Additional context
Found in a nested-type audit of the native scans. Reading the code, the iceberg-rust bump in #6934 should fix this, because at b154603 RecordBatchTransformer reconciles struct children by field id and fills missing ones. That was not verified. Until then, the failure is loud rather than a wrong result.
Activity