Skip to content

[EPIC] Support VARIANT type for unstructured data #16116

Open
Open
[EPIC] Support VARIANT type for unstructured data#16116
Labels
PROPOSAL EPICA proposal being discussed that is not yet fully underwayenhancementNew feature or request

Description

@alamb
alamb
opened on May 20, 2025

Is your feature request related to a problem or challenge?

Processing semi-structured data (basically think anything that can be represented in JSON) efficiently is becoming more and more important.

As @wjones127 says in https://github.com/apache/datafusion/issues/10987>

This would be a high-performance data type for semi-structured data, designed for better OLAP performance than JSON or BSON (discussed in #7845).

While it is certainly possible to implement semi-structured, JSON and even Variant support today using the DataFusion extension apis (e.g. https://github.com/datafusion-contrib/datafusion-functions-json) this ticket tracks adding such support to DataFusion itself

Parquet recently adopted the Variant type : https://github.com/apache/parquet-format/blob/master/VariantEncoding.md

We see adoption of this in other systems as well such as Iceberg and Spark.

I think DataBricks did a good job describing its rationale:

Without Variant, customers had to choose between flexibility and performance. To maintain flexibility, customers would store JSON in single columns as strings. To see better performance, customers would apply strict schematizing approaches with structs, which requires separate processes to maintain and update with schema changes. With Variant, customers can retain flexibility (there's no need to define an explicit schema) and receive vastly improved performance compared to querying the JSON as a string.

Describe the solution you'd like

No response

Describe alternatives you've considered

This will be a big project. Here are some of the related pre-requisites

It is not clear to me if variant should be "built in" or if it should be an add on (for example, add a variant feature and a datafusion-variant crate)

Additional context

Related tickets

Activity

  1. alamb
    added
    enhancementNew feature or request
    on May 20, 2025

  2. paleolimbot commented on Jul 15, 2025

    @paleolimbot
    paleolimbot
    on Jul 15, 2025
    Member

    Just listing a few specific places where I've had to integrate extension types outside of existing DataFusion mechanisms:

    • The Signature (i.e., how do you use the Signature mechanism to match a udf). One can also just use a kernel that matches anything and error in return_field() as well (and coerce the types yourself). If I'm remembering correctly the internal logic to coerce existing types isn't exposed in a way that makes it easy to that.
    • Casting: the Cast struct uses DataType (and anything that uses arrow-rs' casting will too). One can work around this by defining a UDF (e.g., custom_cast(x, to)) where to is a null scalar of the appropriate type.
    • Printing: the out-of-the box output you'll get is probably not what should be printed when you query a Parquet file with a variant column. (Could/should re-use the cast to string?)
    • CSV output (Could/should re-use the cast to string?)
    • The Parquet reader outputting the extension field (I assume this is in the works/is a parquet crate issue?)
    • Multiple Parquet files/unioning: out-of-the box a shredded and unshredded variant will probably fail to concatenate because of the differing layout (slash two shredded variants probably will too).
    • SQL parsing/unparsing (mostly expressing the type name)
    • Use of statistics to do pruning. I think DataFusion automatically disables Parquet pruning when it sees a column reference to a nested type like a struct (and its notion of a Column doesn't support nesting, and its notion of Statistics is somewhat limited), so there are possibly a few battles on this one.
    • Probably more!

    It is not clear to me if variant should be "built in" or if it should be an add on (for example, add a variant feature and a datafusion-variant crate)

    It's definitely easier to hard-code a type, although I think DataFusion will be better for allowing injected behaviour for more than just variant. Variant is definitely shinier, but UUIDs and geometry have many of the same problems. I'll put out a link to the vctrs R package ( https://vctrs.r-lib.org/articles/s3-vector.html ) which is a truly exceptional example of decentralized custom typing that supports custom printing, math, casting, and coercion for parameterized and unparameterized array types in R. Mostly this involves access to a registry of types in places that are currently stateless (can also be a static global variable like in Arrow C++ although I think the SessionContext is a better home).

  3. alamb commented on Sep 26, 2025

    @alamb
    alamb
    on Sep 26, 2025
    ContributorAuthor

    A brief update here -- as of Arrow 57.0.0, I think there will be enough Variant support in the parquet crate to actually use it in DataFusion and there is enough support, as described in Implementing User Defined Types and Custom Metadata in DataFusion for extension types

    I would suggest starting off with the basic input/output:

    Add some udfs for calling the arrow kernels

    1. variant_to_json to convert a variant column to String (and JSON)
    2. json_to_variant: the opposite of the above
    3. variant_get: Extract subpaths
    4. cast_to_variant: casting columns to variant

    I don't think any of this code should be in the "core" datafusion crate, and we could make a datafusion-variant crate either in the main repo, or maybe even in datafusion-contrib

    I am hoping that any features driven by variant (e.g. user defined casting for example) would result in APIs in the core that are implemented by the variant crate

  4. alamb
    changed the title [-]Support `VARIANT` type for unstructured data[/-] [+][EPIC] Support `VARIANT` type for unstructured data[/+] on Sep 29, 2025
  5. alamb
    added
    PROPOSAL EPICA proposal being discussed that is not yet fully underway
    on Sep 29, 2025

  6. alamb commented on Oct 4, 2025

    @alamb
    alamb
    on Oct 4, 2025
    ContributorAuthor

    Update here is that I have a branch / PR with an upgrade to a pre-release version of arrow 57 that includes variant support:

    The next steps I think would be to make a branch starting at #17888 and implement the functions above, and start writing some tests, etc

  7. friendlymatthew commented on Oct 7, 2025

    @friendlymatthew
    friendlymatthew
    on Oct 7, 2025
    Contributor

    Hi hi, I'm going to start working on the UDFs. I'm happy to upstream this as a separate repo

  8. alamb commented on Oct 7, 2025

    @alamb
    alamb
    on Oct 7, 2025
    ContributorAuthor

    Outstanding -- thanks @friendlymatthew that is great news!

    Would you like a separate repo in datafusion-contrib perhaps? We could potentially also reuse the https://github.com/datafusion-contrib/datafusion-functions-variant repo

    Also, FYI I think the WIP upgrade to arrow 57 in #17888 is now ready for use as a base (it passes all tests)

    While details of the variant implementation may change some more, the actual kernels (variant_get, cast_to_variant, etc) have all been pretty stable

  9. friendlymatthew commented on Oct 7, 2025

    @friendlymatthew
    friendlymatthew
    on Oct 7, 2025
    Contributor

    Outstanding -- thanks @friendlymatthew that is great news!

    Would you like a separate repo in datafusion-contrib perhaps? We could potentially also reuse the https://github.com/datafusion-contrib/datafusion-functions-variant repo

    Out of respect to the work done in functions-variant, could we make a new repo? I already started work in https://github.com/friendlymatthew/datafusion-variant, I'm happy to transfer this over

  10. alamb commented on Oct 7, 2025

    @alamb
    alamb
    on Oct 7, 2025
    ContributorAuthor

    Its transferred! 🛳️

    https://github.com/datafusion-contrib/datafusion-variant

  11. carpecodeum commented on Dec 15, 2025

    @carpecodeum
    carpecodeum
    on Dec 15, 2025

    Hi @alamb, I saw this epic, and I wanted to start contributing to Datafusion for some time. Can I help you or @friendlymatthew with anything in this epic?

  12. alamb commented on Dec 16, 2025

    @alamb
    alamb
    on Dec 16, 2025
    ContributorAuthor

    Hi @carpecodeum -- I suggest looking at the issues in https://github.com/datafusion-contrib/datafusion-variant -- we are working on the integration there

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    PROPOSAL EPICA proposal being discussed that is not yet fully underwayenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions