You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Data files persisted without complex columns in the table schema hit "unexpected target column" on read #2618
If we write a data file and then evolve our table's schema to add a complex type field like list, map, struct, proceeding to read the data file back will error out.
To Reproduce
Create a table and write a data file (without column xs).
Evolve the schema to add xs list (or a map/struct column).
Read the table.
Reading the older file errors with unexpected target column type List(...).
Expected behavior
A column present in the table schema but absent from a data file is filled with a typed NULL array of the correct (possibly nested) Arrow type via new_null_array, for nested types just as for primitives. The read should succeed with the added column all-NULL for older files.
Activity
fix(arrow): fill absent nested (list/map/struct) columns with typed N…
test(arrow): restructure apache#2618 nested-null test; trim comments
fix: fill schema-added nested columns with typed NULL arrays on read (a…