You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native sort orders null elements of array and struct keys by the key's null order, unlike Spark #6476
Spark orders a null element of an array key, and a null field of a struct key, below every other value, whatever the key's NULLS FIRST or NULLS LAST, which applies only to a null key itself. Comet's native sort takes the position of nested nulls from the key's null order instead. With the default null orders, ASC NULLS FIRST and DESC NULLS LAST, the two agree, but ASC NULLS LAST and DESC NULLS FIRST put a key that holds a null element at the other end from Spark.
Steps to reproduce
CREATETABLEt(id INT, i INT) USING parquet;
INSERT INTO t VALUES (1, 1), (2, NULL), (3, 3);
SELECT id FROM t ORDER BY array(i) DESC NULLS FIRST, id;
-- Spark: 3, 1, 2 Comet: 2, 3, 1SELECT id FROM t ORDER BY named_struct('x', i) NULLS LAST, id;
-- Spark: 2, 1, 3 Comet: 1, 3, 2
Expected behavior
A null element or field sorts below every other value, as in Spark, whatever the key's null order. The same applies to TopK and to window order keys.
Additional context
Found while fixing #5507 in #6475, whose fixture keeps the default null orders to stay clear of this.
Activity
fix: decline nullable nested float keys in strict mode and drop the n…
fix: match Spark's float ordering for sort and window keys nested in …
fix: fall back for nested sort keys whose null elements Spark orders …
feat: run nullable nested float sort keys natively in strict floating…