Repository navigation
Error writing to a partitioned table: : it is not yet supported to write to hive partitions with datatype Dictionary(UInt16, Utf8) #7891
Description
@suremarc got this error when writing to a partitioned table:
This feature is not implemented: it is not yet supported to write to hive partitions with datatype Dictionary(UInt16, Utf8)
Here is a repro using datafusion-cli:
CREATE EXTERNAL TABLE lz4_raw_compressed_larger
STORED AS PARQUET
PARTITIONED BY (partition)
LOCATION 'data/';
INSERT INTO lz4_raw_compressed_larger VALUES ('non-partition-value', 'partition');Here's a zip file with a single file in it, data/partition=A/lz4_raw_compressed_larger.parquet.
I noticed the unit tests specify the schema explicitly, but I am guessing if you have DataFusion infer the schema, the partition columns are encoded as dictionaries. I think this will limit the usefulness of this feature if partitioned writes don't work with tables whose schemas are inferred.
Originally posted by @suremarc in #7801 (comment)
Activity
devinjdangelo commentedon Oct 21, 2023
Hm, I am a little confused why Datafusion is inferring the schema of UTF8 data as Dictionary(some int type, UTF8).
🤔 will have to look into it. It does seem that #7891, #7892, and some of the inconveniences reported by @theelderbeever in #7860 are all related.
Perhaps the partitioning code could accept any arrow array type which can be explicitly cast to UTF8, rather than only strictly UTF8... I assume since these Dictionary columns are representing string data, they can be cast to a plain UTF8 array without panic/error.
devinjdangelo commentedon Oct 21, 2023
Ok, I read the
arrow-rsdocs on dictionary array types, so I understand what that means now... I took a stab at solving this in #7896