Repository navigation
Open Variant Type for semi-structured data #10987
Description
Activity
@samuelcolvin and @adriangb and I were speaking about datafusion-functions-json just today. They may be interested too
I have created https://github.com/datafusion-contrib/datafusion-functions-variant and added you as an admin 🚀
Reacted by Adrian Garcia Badaracco, Will Jones and Samuel ColvinInteresting, this is exactly what we need.
FWIW the variant type has been added to the parquet spec, and is now much closer to standardization and we are beginning work on the support at the parquet layer. See epic
There's a discussion happening on the Arrow dev mailing list about adding Variant as a canonical extension type in the Arrow spec. Input from the DataFusion developer community would be much appreciated.
- The discussion thread in the mailing list archive: https://lists.apache.org/thread/w06cxdojjcmry4m9vb0bo7owd1jsbtz5
- How to subscribe to the Arrow dev mailing list: https://arrow.apache.org/community/#mailing-lists
Reacted by Will Jones
Metadata
Metadata
Assignees
Labels
No labels
I've been starting to experiment with implementing the Open Variant Type 1 in Rust / DataFusion. There is a specification and Java library for this, and Spark will release this type in 4.0. There are also plans to integrate this into table formats such as Delta Lake 2 and Iceberg 3. This would be a high-performance data type for semi-structured data, designed for better OLAP performance than JSON or BSON (discussed in #7845). I've discussed a little bit in the Arrow repo about it's potential as an Arrow extension type 4.
I'm working on creating an extension similar to datafusion-functions-json. If we could create a new repo
datafusion-functions-variant, I'd be happy to develop that in the open.Footnotes
https://github.com/apache/spark/tree/master/common/variant ↩
https://www.databricks.com/blog/introducing-open-variant-data-type-delta-lake-and-apache-spark ↩
https://lists.apache.org/thread/xnyo1k66dxh0ffpg7j9f04xgos0kwc34 ↩
https://github.com/apache/arrow/issues/42069 ↩