Repository navigation
Int96Coercer drops the metadata of struct, list and map fields. #24786
Description
Reading a Parquet file with datafusion.execution.parquet.coerce_int96 set is supposed to only change the time unit that INT96 timestamp columns decode at. However it also empties the metadata of every struct, list and map field in the file.
Cause
In datafusion/datasource-parquet/src/schema_coercion.rs, leaf fields are cloned:
// :423
fn field_with_new_type(field: &FieldRef, new_type: DataType) -> FieldRef {
Arc::new(field.as_ref().clone().with_data_type(new_type))
}while container fields are constructed fresh, and Field::new* starts from empty metadata:
// :327
let processed_struct = Field::new_struct(
current_field.name(),
processed_children.as_slice(),
current_field.is_nullable(),
);
// :360
let processed_list = Field::new_list(
current_field.name(),
Arc::clone(&processed_children[0]),
current_field.is_nullable(),
);
// :392
DataType::Map(Arc::clone(&processed_children[0]), *sorted),Name, type and nullability are carried across; current_field.metadata() is never read.
A test that demonstrates the issue: https://github.com/ryzhyk/coerce_int96_bug/tree/main
Activity
Preserve field metadata when coercing INT96 timestamps
[adapters] Keep container field ids when coercing INT96
Preserve field metadata when coercing INT96 timestamps
[adapters] Pin a datafusion fork that preserves INT96 field metadata
iSparshP commentedon Sep 1, 2026
take
okay so i looked into this and i think i found what is going on. in the datasource parquet crate, inside the schema coercion file, the leaf fields get cloned so they keep there metadata, but the struct, list and map containers are all built fresh with Field new_struct, new_list and Map. those constructors start out with empty metadata, so the metadata on the current field just gets thrown away on the way through.
i think the fix is to carry it over at those three places by calling with_metadata using the current field metadata. i will also add a regression test that checks the container field metadata actualy survives when coerce_int96 is turned on. will put up a pull request soon.
iSparshP commentedon Sep 2, 2026
Update: @ryzhyk already has a fix open in #24790 that predates my attempt, so I've closed mine (#24856) in favor of it. Un-claiming this from my side. #24790 looks good and should close this.
Preserve field metadata when coercing INT96 timestamps (apache#24790)
[branch-55] Preserve field metadata when coercing INT96 timestamps (#…