Skip to content

least orders a null inside a struct or array last, where Spark orders it first #6759

Closed

Description

@viirya

Describe the bug

least over structs or arrays orders a null inside the value after every other value. Spark orders it before every other value, in least as in greatest, so Comet can return a different value from Spark.

Comet uses its own SparkGreatestLeast only for floats and nested types with a float leaf. Other nested types go to DataFusion's least, whose comparator uses nulls_first: false.

Steps to reproduce

SELECT id, least(named_struct('a', CAST(id AS INT), 'b', 1),
                 named_struct('a', IF(id = 3, NULL, 0), 'b', 3))
FROM range(4);

With spark.comet.exec.range.enabled=true, spark.comet.sparkToColumnar.enabled=true, and spark.comet.sparkToColumnar.supportedOperatorList=Range, Comet runs this natively. For id = 3 Spark returns [null, 3] and Comet returns [3, 1]. least(array(CAST(id AS INT), 1), array(IF(id = 3, NULL, 0), 3)) behaves the same way.

Expected behavior

A null inside a struct or array ranks before every other value, so least returns [null, 3] for id = 3.

Metadata

Metadata

Assignees

Labels

area:expressionsExpression evaluationbugSomething isn't workingcorrectnesspriority:criticalData corruption, silent wrong results, security issues

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions