Repository navigation
What's the Supported Input Data Type #64
Description
What is the problem the feature request solves?
Hi Team,
I am wondering will we have a documentation about what kinda of data types are supported by Comet/Datafusion (Like, Decimal(16, 6), interger .... ) and will not fallback to vanilla spark and cause the regression.
Or where I can lookup to find related information. Since this is very important while we are trying to evaluate whether to use Comet or not.
Thank you so much !
Describe the potential solution
No response
Additional context
No response
Activity
[-]Supported Input Data Type[/-][+]What's the Supported Input Data Type[/+]viirya commentedon Feb 20, 2024
We will add the document, maybe as part of user document in #57.
For a quick answer, primitive types should be supported now: ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType, StringType, BinaryType, TimestampType, DecimalType, DateType, BooleanType, NullType. The support I mean is, Comet query planner will accept query plan using these types. But for certain expression or operator, the supported types might be various, for example, if the corresponding DataFusion expression or operator may not support some data types.
I think one important goal of Comet is to fulfill the gap and we are working on it. So welcome to report any missing datatypes when you test or evaluate Comet.
jinwenjie123 commentedon Feb 20, 2024
Thank you !
One more quick question. Is Decimal(26, 6) currently supported ?
viirya commentedon Feb 20, 2024
DecimalType is supported.
sunchao commentedon Feb 21, 2024
We should support all primitive data types as of now (if not, we should definitely fill the gap).
Complex types (e.g., list, map, struct) are not supported in the Parquet scan. We do have plan to support it soon.
viirya commentedon May 6, 2024
We have better documentation for the issue: https://datafusion.apache.org/comet/user-guide/datatypes.html
viirya commentedon May 6, 2024
Close it now. Please let me know if you still have other questions. Thanks.