Repository navigation
Support more types for approx_distinct function #22989
Description
Is your feature request related to a problem or challenge?
#22921 has refactored approx_distinct into a more generic implementation over different input types, now it's easy to support additional arg types. (Thanks to @Jefffrey for pointing it out in the review)
Describe the solution you'd like
- Identify the unsupported types
- Plan them into the generic implementation of
AccumulatorandGroupsAccumulator(see refactor: Simplifyapprox_distinct(-200 LoC) #22921 for details) - Add
sqllogictests
Note we don't have to support all applicable types at once, some complex type might get tricky to test, so I prefer to do this incrementally
Describe alternatives you've considered
No response
Additional context
No response
Activity
mkleen commentedon Jun 17, 2026
take
alamb commentedon Jun 23, 2026
FWIW some types, such as floats, likely don't make sense to implement DISTINCT for (as we found in #23084).
It would probably be good to review what other database implementations do before implementing it for datafusion
mkleen commentedon Jun 24, 2026
I had a look at DuckDB. DuckDB doesn't declare the supported types in their docs for
approx_count_distincthttps://duckdb.org/docs/lts/sql/functions/aggregates.I did some manual testing and it seems it supports each of its data type:
TINYINT,SMALLINT,INTEGER,BIGINT,HUGEINTUTINYINT,USMALLINT,UINTEGER,UBIGINTFLOAT,DOUBLE,DECIMALVARCHAR,BLOB,BIT,UUIDDATE,TIME,TIMESTAMP,TIMESTAMPTZ,INTERVALBOOLEAN,JSONLIST,STRUCT,MAPmkleen commentedon Jun 24, 2026
The arrow-types which are currently unsupported in
approx_distinctin datafusion are:approx_distinctFloat16Float32Float64Decimal32Decimal64Decimal128Decimal256DurationIntervalBinaryViewFixedSizeBinaryListLargeListListViewLargeListViewFixedSizeListStructMapUnionDictionaryRunEndEncodedIt most likely doesn't make sense to support all types. But this is probably a good start.
feat: Support Decimal type in `approx_distinct` (apache#23190)
feat: Support interval type in approx_distinct (apache#23234)
feat: Support duration type in approx_distinct (apache#23291)
feat: Support BinaryView type in approx_distinct (apache#23333)
feat: Support FixedSizedBinary type for approx_distinct (apache#23417)
feat: Support Map type in approx_distinct (apache#23526)
feat: Support Struct type in approx_distinct (apache#23663)
mkleen commentedon Jul 18, 2026
We won't support
Floattypes. What aboutRunEndEncoded,UnionandDictionary? They are no SQL types mapped to them. Does it make sense to support them?Jefffrey commentedon Jul 19, 2026
the way i see it, if regular distinct supports the data type, approx distinct should also support it; given it seems regular distinct supports all types we should follow suit
mkleen commentedon Jul 19, 2026
Ok, thanks for the feedback!
feat: Support Union type in approx_distinct (apache#23714)
feat: Support List/ListView types in approx_distinct (apache#23443)
feat: Support Map type in approx_distinct (apache#23526)
feat: Support Struct type in approx_distinct (apache#23663)
feat: Support Union type in approx_distinct (apache#23714)
Add support for dictionary for approx_distinct (apache#24646)
feat: Support RunEndEncoded type in approx_distinct (apache#25951)
feat: Support RunEndEncoded type in approx_distinct (apache#25951)