Repository navigation
Support Extension Types / User Defined Types in DataFusion #12644
Description
Activity
I'm not very knowledgeable about DataFusion internals or database theory, so it's hard for me to provide feedback on the proposal, but I'm very excited about the prospect of extension types to enable spatial types (#7859). I've been collaborating on the GeoArrow spec, which defines Arrow extension types for spatial data. It's important to have additional logical types because the same physical layout can be interpreted in multiple logical ways (e.g. an array of
LineStringandMultiPoint), and to store coordinate reference system information (what physical locations on earth these numbers represent) as part of the type. I'm happy to provide more motivating examples if that would help!FYI i touched upon the topic of types on DataFusion meetup in Belgrade yesterday.
The slides are here if anyone is interested: https://docs.google.com/presentation/d/1VW_JCGbN22lrGUOMRvUXGpAmlJopbG02hn_SDYJouiY . It was an attempt to summarize why we need both: simpler types (#11513), more types (#12644), and simple function "SDK" (#12635).
The document has comments disabled to avoid diverging the discussion from the github issue.I'm interested in this as well; however, I'm new to DataFusion development and I am not sure I have a handle on what the barriers are here (e.g., Is support for this blocked by lack of support in arrow-rs? Is there opposition to this on principle or is it just a matter of development time?).
A few more (albeit geospatial) reasons to add support for this are (not-quite-merged!) support for spatial data in Iceberg and Parquet. As currently written, those proposals include type-level metadata (notably: the CRS), which would mean that reading Parquet files with those types would loose information. There are also other Arrow canonical extension types that have type-level metadata (which to my current reading, means that canonical extension types are not usable in DataFusion?).
Reacted by Kyle BarronIs support for this blocked by lack of support in arrow-rs?
arrow-rs intentionally doesn't "directly support" extension types through
DataType. The alternative (now archived) arrow2 implementation had anExtensionvariant, but this has the downside of requiring all code that needs to match theDataTypeenum to be aware of logical extension types. See apache/arrow-rs#4472 for more discussion.Instead, arrow-rs supports extension types indirectly via the metadata hashmap on the
Fieldaccompanying someArray. So e.g. in geoarrow-rs, we have custom types likePointArraythat internally manage both the array data and the field metadata, but when exporting toarrowobjects, you'd need to carry theArrayand theFieldtogether.I haven't closely followed the logical type work in progress in Datafusion but I assume it would be associating some Field with the physical type, and extension types could inject metadata there.
Reacted by Dewey Dunnington, Courtney Robinson, Alex Kesling and Andrew DuffyI second @paleolimbot here, as I am also interested in this topic.
First of all, thank you for all your efforts in this area!To provide some context, we are currently creating a prototype that handles data that is encoded as a union with 10-ish different variants. Logically, this union represents the encoding of a single type. While we make ends meet by creating logical plans manually with specialized UDFs (e.g., for equality), we are in a very early phase. Only working with physical types will add lots of complexity to our code that can be eliminated with logical types (#12622) and its benefits (e.g., #12635) (I think™).
So what our (dream) scenario is that we could define this extension type (and its encoding) together with implementations for equality, orderings, etc, and DataFusion would automatically make use of these when joining or sorting (AFAIK, this should be supported according to #7923).
I'd like to support you in these efforts. However, while I read a few discussions and proposals, I am still a bit lost on where I can/should help out as this is (from what I see) a huge ongoing project across multiple issues and epics. Do you have pointers on where I could start helping out? For example, I found #13301, but I am unsure if the efforts on simple functions #12635 make these changes somewhat premature.
Thank you for helping me to navigate this project!
Reacted by Piotr Findeisen- Reacted by Dewey Dunnington
@mbrobbel has a nice proposal to add extension types in arrow-rs (which would potentially help ExtensionTypes in DataFusion). Would apprecaite any feedback:
Reacted by Dewey Dunnington and Nicholas RobertsI started thinking about how the ExtensionType trait would be used in DataFusion. I think we would need to improve the APIs a bit to be in terms of
Fieldrather thanDataType. See this ticket for more infoReacted by Kyle BarronComing soon (arrow 54.2 in a month), support for Extension Types:
Reacted by Kyle Barron- changed the title
[-]Extension Types[/-][+]Extension Types / User Defined Types[/+]on Feb 23, 2025 BTW DataFusion now has access to the arrow user defined types / is upgraded to 54.2 -- maybe now is a good time to start with this
@tobixdev has a usecase for specifying sort orders in #14828 -- maybe that would be a good thing to try to make work at first 🤔
- added a sub-issue
on Feb 23, 2025 14 remaining items
I agree that DataFusion would benefit from embracing its own definition of a (dynamic, session-registered) data type that allows a type author to specify behaviour in such a way that DataFusion internals (e.g., optimizer rules, casting) can benefit things that are not just built-in arrow types (and/or prevent DataFusion internals from introducing correctness issues that may arise from ignoring or dropping metadata). I'm also sensing resistance to take on responsibility for that, which is OK (engines that use DataFusion can implement all of this on top of metadata, although it is quite a bit of work that perhaps doesn't have to be duplicated if it's in DataFusion).
Is this the end state we want to achieve?
In my mind it is better than what is currently possible. I agree we can come up with other better designs too
I agree that DataFusion would benefit from embracing its own definition of a (dynamic, session-registered) data type that allows a type author to specify behaviour in such a way that DataFusion internals (e.g., optimizer rules, casting) can benefit things that are not just built-in arrow types
I mean I agree with this too -- in my mind the question is how actually to achieve this goal
I don't know how to achieve
DataType + metadata-based design without making things complicated and error-prone.
I think the bigger question is whether we as community agree regarding the end state we want to achieve. If we agree the end state is lean and pleasurable-to-work-with type system, I am sure we collectively figure it out how to get there.If we agree the end state is lean and pleasurable-to-work-with type system, I am sure we collectively figure it out how to get there.
I think it would he hard to find someone that doesn't want that end state 😆 . I think the differences might be in opinions on what that actually looks like and how to get there
For what it's worth, it looks like polars is implementing this as an
Extensionmember on theirDataTypeenum:pola-rs/geopolars#245 (comment)
Having something
Arc<dyn ExtensionTypeyThing>is nice because it limits the points at which a "registry" is necessary. The registry (ideally in the session) produces objects that know how to print, cast, coerce, and compare equality so that we don't have to pipe a reference to the registry into every single==or cast involving data types. In Arrow C++ the only points at which the static registry are used are when importing serialized IPC or importing a type via the C data interface.Having something Arc is nice because it limits the points at which a "registry" is necessary.
That's a valid point. If we further believe that all the metadata handling should be attached to
Field, maybe the field should not directly have a physicalDataType. More like something akin to:enum FieldDataType { Native(DataType), Extension(Arc<dyn ExtensionTypeyThing>) }
This could i) allow arrow kernels to keep on focusing on
DataTypeto interpret the underlying physical layout (which I think was the primary reason for rejecting theDataType::Extension(...)solution and ii) prevent accidentally using the wrong implementation of, for example, the+operator for an extension type as the type information stems from theField. Again, a huge breaking change but I think this cannot be prevented. We could offer a utility functions that allows ignoring any logical type to ease the migration.Any thoughts on that?
This would be a similar approach to
TypeSignature(https://github.com/apache/datafusion/blob/main/datafusion/common/src/types/logical.rs) .Reacted by Kyle Barron, Adrian Garcia Badaracco and Ning SunI started a new ticket to focus on the discussion about a registry of types / what exactly this API would look like in DataFusion.
I like the idea of this (maybe it is a new
DFType🤔 )enum DFType { Native(DataType), Extension(Arc<dyn ExtensionTypeyThing>) }
- changed the title
[-]Support Extension Types / User Defined Types in DataFusion[/-][+][EPIC] Support Extension Types / User Defined Types in DataFusion[/+]on Nov 5, 2025 - changed the title
[-][EPIC] Support Extension Types / User Defined Types in DataFusion[/-][+]Support Extension Types / User Defined Types in DataFusion[/+]on Nov 5, 2025 @paleolimbot and @tobixdev @timsaucer -- I feel like there is significant motion / momentum for better Field / metadata support in service of user defined types, but I couldn't find any high level discussion about it other than this ticket
Do you know of anything more specific? Basically I am looking for a "epic" style ticked with a high level description of what we are trying to do (aka ensure
Fieldis present everywhere instead of just DataType) and then a list of tickets for different areas that are neededIf no such thing exists, would it be ok if I created one?
I feel like there is significant motion / momentum for better Field / metadata support in service of user defined types
Glad to hear it!
Do you know of anything more specific?
I don't! There is some good discussion on #18223 with respect to some first steps.
what we are trying to do (aka ensure Field is present everywhere instead of just DataType)
I am not sure that is the consensus on this ticket or #18223 so far...I think we all want something better than that but aren't sure how to do it without changing arrow-rs or a lot of disruption of the DataFusion code base. Replacing the DataType with a Field in the logical plan and SQL parser is sort of the bare minimum for somebody else to invent their own type system on top of DataFusion with some effort.
If no such thing exists, would it be ok if I created one?
That would be great! I am definitely willing to do work, it's just hard to do so without consensus and I am not sure what that is. My personal first step might be to replace any
FieldRefthat is meant to communicate a data type withExprTypeorExprFieldorDFField(that is a thin wrapper around aFieldRef) so that we can get metadata everywhere in the short term with the flexibility to update the internal representation (e.g., with adyntype of some kind).Reacted by Tobias SchwarzingerThat would be great! I am definitely willing to do work, it's just hard to do so without consensus and I am not sure what that is.
Yeah I agree that is key missing part. Let me see if I can write up something that maybe will help us get there. Will try tomorrow
I basically agree with @paleolimbot here! I've also now opened a "discussion PR" for the extension type registry #18552 . Maybe this provides additional context.
Reacted by Dewey Dunnington- added a commit that references this issue
on Jul 31, 2026
Is your feature request related to a problem or challenge?
Currently DataFusion provides a lot of built-in types which are useful when building applications / query engines on top of DataFusion. However, even plethora of types is not enough. DataFusion doesn't have types existing in other systems, limiting DataFusion applicability as "LLVM for query engines"
For example, these types commonly found in other systems do not exist today
DataTypeand the closest Arrow has is "timestamp(zone)" where each value is in same zoneDataTypeand the closest Arrow has is "timestamp(zone)" with eg UTC zone. however cast to varchar for "timestamp(UTC)" and for "timestamp with local time zone" should behave differentlyDataTypeand the closest Arrow has Utf8 potentially with some metadata information. Utf8 might be a perfect carrier type for JSON data, but "cast(json AS T)" and "cast(utf8 AS T)" are usually pretty different operationsDescribe the solution you'd like
CAST(array<T> AS varchar)needs to know how to docast(T AS varchar). It cannot delegate this logic fully to Arrow, because Arrow won't have a notion of extension types.cast(... AS varchar). It cannot use the defaultcast(struct AS varchar).Describe alternatives you've considered
Everything is built-in
DataFusion could provide all types needed by applications building on top of DataFusion as built-in DataFusion types.
This would be easiest to implement, but could lead to scope-creep for the project. This could also lead to conflicts where types look the same but the desired behavior differs between applications building on top of DataFusion. For example Oracle's and Trino's "timestamp with time zone" can represent political zones while Snowflake's allows only fixed offsets.
No-op
Not providing extension types. This would limit DataFusion applicability.
DataFusion cannot be considered "LLVM for query engines" if it cannot serve as an engine, or potential engine, for existing popular query engines.
Additional context
The need to create extension types was raised in the [Proposal] Decouple logical from physical types
However introduction of DataFusion own types does not require introduction of extension types.
Extension types are complex enough (especially given their impact on functions) that they deserve their own roadmap issue.
The impact of extension types on functions, functions runtime and resolution is very clear, so this relates to Simple Functions initiative:
Having ExtensionType in arrow-rs would could the implementation simpler: