Skip to content

collect_set over floats beside count(DISTINCT) diverges from Spark on signed zeros and NaN #7203

Open

Description

@andygrove

Describe the bug

#4727 made collect_list and collect_set native in aggregate chains with a PartialMerge. This is the plan Spark builds when a query mixes count(DISTINCT ...) with other aggregates. 1.1.0 kept any such chain in Spark (hasNativeArrayBufferAgg in CometExecRule).

On Spark versions before 4.2, with spark.comet.exec.strictFloatingPoint=false (the default), native collect_set keeps -0.0 and 0.0 as two entries and merges every NaN into one. Spark does the opposite: it collapses the zeros and keeps each NaN. The compatibility guide already documents this for plain collect_set, which was native in 1.1.0 too. What is new in 1.2.0 is that the distinct shape now reaches it. On 1.1.0 that shape matched Spark.

Steps to reproduce

Spark 4.1, default configs:

SELECT x, count(DISTINCT y), sort_array(collect_set(z))
FROM VALUES (1, 1, CAST('NaN' AS DOUBLE)), (1, 2, CAST('NaN' AS DOUBLE)),
            (1, 3, -0.0D), (1, 4, 0.0D), (1, 5, 1.0D) AS t(x, y, z)
GROUP BY x
  • Spark: [0.0, 1.0, NaN, NaN] (or [-0.0, 1.0, NaN, NaN] when the rows come from Parquet)
  • 1.2.0: [-0.0, 0.0, 1.0, NaN]
  • 1.1.0: matches Spark, because the aggregate ran in Spark

FLOAT behaves the same way. The divergence shows up both when the partial aggregate runs in Spark and when it runs natively over a Parquet scan.

Expected behavior

The distinct shape matches Spark, as in 1.1.0. Two ways to get there:

Additional context

This is a regression since 1.1.0, found in the branch-1.2 audit. Related: #5312 and #6385. Every other element type checked matched Spark through both the Spark-partial decoder and the native path. That covered decimals up to (38,10), binary, temporal types, and nested arrays, structs and maps.

Activity

  1. added this to the 1.2.0 milestone on Oct 25, 2026
  2. added
    bugSomething isn't working
    area:aggregationHash aggregates, aggregate expressions
    regressionA bug that did not affect the most recent Comet release
    on Oct 25, 2026
  3. LinSimon-901101 commented on Oct 25, 2026

    @LinSimon-901101
    Contributor

    take

  4. added
    priority:criticalData corruption, silent wrong results, security issues
    and removed on Oct 26, 2026
  5. modified the milestones: 1.2.0, 1.3.0 on Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:aggregationHash aggregates, aggregate expressionsbackport-1.2bugSomething isn't workingcorrectnesspriority:criticalData corruption, silent wrong results, security issuesregressionA bug that did not affect the most recent Comet release

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions