You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
collect_set over floats beside count(DISTINCT) diverges from Spark on signed zeros and NaN #7203
#4727 made collect_list and collect_set native in aggregate chains with a PartialMerge. This is the plan Spark builds when a query mixes count(DISTINCT ...) with other aggregates. 1.1.0 kept any such chain in Spark (hasNativeArrayBufferAgg in CometExecRule).
On Spark versions before 4.2, with spark.comet.exec.strictFloatingPoint=false (the default), native collect_set keeps -0.0 and 0.0 as two entries and merges every NaN into one. Spark does the opposite: it collapses the zeros and keeps each NaN. The compatibility guide already documents this for plain collect_set, which was native in 1.1.0 too. What is new in 1.2.0 is that the distinct shape now reaches it. On 1.1.0 that shape matched Spark.
Steps to reproduce
Spark 4.1, default configs:
SELECT x, count(DISTINCT y), sort_array(collect_set(z))
FROMVALUES (1, 1, CAST('NaN'AS DOUBLE)), (1, 2, CAST('NaN'AS DOUBLE)),
(1, 3, -0.0D), (1, 4, 0.0D), (1, 5, 1.0D) AS t(x, y, z)
GROUP BY x
Spark: [0.0, 1.0, NaN, NaN] (or [-0.0, 1.0, NaN, NaN] when the rows come from Parquet)
1.2.0: [-0.0, 0.0, 1.0, NaN]
1.1.0: matches Spark, because the aggregate ran in Spark
FLOAT behaves the same way. The divergence shows up both when the partial aggregate runs in Spark and when it runs natively over a Parquet scan.
Expected behavior
The distinct shape matches Spark, as in 1.1.0. Two ways to get there:
Fall back for collect_set over FLOAT or DOUBLE elements in a PartialMerge chain on Spark before 4.2.
This is a regression since 1.1.0, found in the branch-1.2 audit. Related: #5312 and #6385. Every other element type checked matched Spark through both the Spark-partial decoder and the native path. That covered decimals up to (38,10), binary, temporal types, and nested arrays, structs and maps.
Activity
LinSimon-901101 commentedon Oct 25, 2026
take