[EPIC] Improved support for nested / structured types (Struct , List, ListArray, and other Composite types)

@lquerel

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
This ticket is designed to capture the work needed to properly support Arrow Struct types in DataFusion

https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html says that nested types are not supported; The are not fully supported, but there are parts of the support already present such as a way to serialize them via ArrowWriter and using field["nested_field"] syntax

Describe the solution you'd like
Research, and describe / implement what is else remains for proper support.

Array (`ListArray`) support:

Map (`MapArray`) support:

[EPIC] A collection of issues for supporting the MAP DataType #11429

Struct (`StructArray`) support:

Union (`UnionArray`) support

Support Union as a function #10206

Other

Known issues so far:

This https://github.com/apache/arrow-datafusion/blob/master/datafusion/core/src/physical_plan/file_format/mod.rs#L238 is one reason of errors related to column projection. It compares the complete enum, failing on different field order.

Arrow has a method to compare data types (https://github.com/apache/arrow-rs/blob/master/arrow/src/datatypes/datatype.rs#L674). I think this method should me made public, and used in above.

Currently datafusion uses match_field_names (default true), https://github.com/apache/arrow-rs/blob/master/arrow/src/record_batch.rs#L153 causing the error.

@nl5887

Thanks for the investigation @nl5887 -- that sounds definitely plausible. Feel free to file a PR with proposed changed -- we would love to review them

This one is also related: #2581

Reminder to write docs: #1222

Potential to add to list #7012

We are starting to make progress on struct support --

There is a PR up to support named_struct #9743 and work afoot to support nicer literal syntax: #9820 🚀

Hi, i think unnest support for struct can be an item in this epic right?

Hi, i think unnest support for struct can be an item in this epic right?

That would make sense to me -- is there a ticket that describes what this means?

i created a ticket: #10264

i created a ticket: #10264

Thank you. I added this to the list in the ticket description

I added an issue to support recursive unnest: #10660, i think it shoul belong to this epic

I added an issue to support recursive unnest: #10660, i think it shoul belong to this epic

Added

I added an issue to check the duplicate or null name for struct: #11438

I think #11445 is related to this epic

I think #11445 is related to this epic

Thank you -- added

@alamb

Right now datafusion doesn't support struct evolution very well. Imagine you have a struct named customData with field someOptionEnabled in one parquet file, later down the line you add a new field newAddedOption to the customData struct in another parquet file. Currently when you try and SELECT * FROM table you'll get this error:

{"message":"Failed to collect DataFrame batches: Plan(\"Cannot cast file schema field customData of type Struct([Field { name: \\\"someOptionEnabled\\\", data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]) to table schema field of type Struct([Field { name: \\\"someOptionEnabled\\\", data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \\\"newAddedOption\\\", data_type: Float64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }])\")","status":"error"}

Feels like we should handle this more gracefully. cc @alamb

I'm happy to make contributions if someone can point me to the right places to look.

@alamb

Right now datafusion doesn't support struct evolution very well. Imagine you have a struct named customData with field someOptionEnabled in one parquet file, later down the line you add a new field newAddedOption to the customData struct in another parquet file. Currently when you try and SELECT * FROM table you'll get this error:
{"message":"Failed to collect DataFrame batches: Plan(\"Cannot cast file schema field customData of type Struct([Field { name: \\\"someOptionEnabled\\\", data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]) to table schema field of type Struct([Field { name: \\\"someOptionEnabled\\\", data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \\\"newAddedOption\\\", data_type: Float64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }])\")","status":"error"}
Feels like we should handle this more gracefully. cc @alamb

I agree

I'm happy to make contributions if someone can point me to the right places to look.

My suggestion is to start with filing a ticket with a self contained reproducer (either rust code or SQL) that shows what you are trying to do.

This would likely become part of the test of any code improvement we make, as well as providing some more detail for other contributors to help point to the right place in the code

I think we will claim datafusion has pretty good structured type support now, so let's close this epic and treat additional work as new issues / items

@alamb

@alamb I just tried this on 45.0 and it still doesn't seem supported:

 Failed to collect data frame results: Plan("Cannot cast file schema field additionalInfo of type Struct([Field { name: \"location\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"timestamp_utc\", data_type: Timestamp(Millisecond, Some(\"UTC\")), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]) to table schema field of type Struct([Field { name: \"location\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"timestamp_utc\", data_type: Timestamp(Millisecond, Some(\"UTC\")), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"reason\", data_type: Struct([Field { name: \"_level\", data_type: Float64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"details\", data_type: Struct([Field { name: \"rurl\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"s\", data_type: Float64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"t\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }])

Do I need to flip a flag or something?

Filed an issue here:

Datafusion can't seem to cast evolving structs #14757

Navigation Menu

Repository navigation

[EPIC] Improved support for nested / structured types (`Struct` , `List`, `ListArray`, and other Composite types) #2326

Array (`ListArray`) support:

Map (`MapArray`) support:

Struct (`StructArray`) support:

Union (`UnionArray`) support

Other

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[EPIC] Improved support for nested / structured types (Struct , List, ListArray, and other Composite types) #2326

Description

Array (ListArray) support:

Map (MapArray) support:

Struct (StructArray) support:

Union (UnionArray) support

Other

Activity

nl5887 commented on Jun 9, 2022

alamb commented on Jun 10, 2022

nl5887 commented on Jun 26, 2022

tv42 commented on Feb 22, 2023

alexwilcoxson-rel commented on Aug 30, 2023

alamb commented on Mar 27, 2024

toaiduongdh commented on Apr 27, 2024

alamb commented on Apr 27, 2024

duongcongtoai commented on Apr 27, 2024

alamb commented on Apr 30, 2024

duongcongtoai commented on May 25, 2024

alamb commented on May 25, 2024

goldmedal commented on Jul 12, 2024

Throne3d commented on Jul 13, 2024

alamb commented on Jul 14, 2024

TheBuilderJR commented on Aug 13, 2024

alamb commented on Aug 15, 2024

alamb commented on Feb 13, 2025

TheBuilderJR commented on Feb 18, 2025

TheBuilderJR commented on Feb 19, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

[EPIC] Improved support for nested / structured types (`Struct` , `List`, `ListArray`, and other Composite types) #2326

Array (`ListArray`) support:

Map (`MapArray`) support:

Struct (`StructArray`) support:

Union (`UnionArray`) support