arrays_except type mismatch #3646

Activity

1. comphead

   added

   bugSomething isn't workingSomething isn't working

   good first issueGood for newcomersGood for newcomers

   on Mar 7, 2026
2. n0r0shi commented on Mar 7, 2026

   

   n0r0shi

   on Mar 7, 2026

   Last edited by n0r0shi

   Contributor

   More actions

   For this, I think there are 2 approaches. One is to fix upstream: make check_datatypes() ignore inner field nullability. Another is to fix in Comet: normalize inner nullability before calling array_except. Avoids upstream change but adds overhead.

   I'd lean toward (1), but I'd like to confirm how others think @comphead
3. comphead commented on Mar 7, 2026

   

   comphead

   on Mar 7, 2026

   Last edited by comphead

   ContributorAuthor

   More actions

   I tried to replace .equals_datatype to .contains which is more nullability friendly, however it would still doesn't work this scenario.

   .contains ignores nullability if the primary datatype is nullable, but let data_type = args[0].data_type(); doesn't guarantee exactly this datatype would be nullable.

   I feel we can tweak check_datatype and if incoming datatypes are lists then take first NULLABLE datatype instead of just first
4. andygrove

   added

   priority:mediumFunctional bugs, performance regressions, broken featuresFunctional bugs, performance regressions, broken features

   area:expressionsExpression evaluationExpression evaluation

   on Mar 27, 2026
5. yuboxx commented on May 7, 2026

   

   yuboxx

   on May 7, 2026

   Contributor

   More actions

   @comphead could you share the Spark and Comet version you used to produce this error? I can't seem to reproduce it with a local script running Spark 3.5.8 and the Comet local build based on latest of comet.
6. comphead commented on May 7, 2026

   

   comphead

   on May 7, 2026

   ContributorAuthor

   More actions

   Thanks @yuboxx
   I think you can reproduce by running src/test/resources/sql-tests/expressions/array/array_except.sql

   ./mvnw test -Dsuites="org.apache.comet.CometSqlFileTestSuite array_except" -Dtest=none
   
7. stantheman0128 commented on Sep 25, 2026

   

   stantheman0128

   on Sep 25, 2026

   Contributor

   More actions

   I'd like to take this. It still reproduces on a recent main (1c25b492) with allowIncompatible on: removing the ignore on the literal query in array_except.sql fails with array_except received incompatible types: List(Int32), List(non-null Int32).

   The plan is to make both sides agree on a nullable element in the serde, reusing normalizeContainerNullability as suggested on #4237, and then re-enable that query. #5526 unifies nested element types for the set ops and leaves primitive elements alone, so I'll keep this to the list-level nullability and rebase onto it if it lands first.

   ❤️React with ❤️1Reacted by Oleks V
8. contentforge-press commented on Oct 2, 2026

   

   contentforge-press

   on Oct 2, 2026

   More actions

   Working on a fix for this! 🔧

   I've analyzed the issue and identified the root cause:

   Problem

   The error occurs because DataFusion's array_except function receives arrays with different element nullability:

   • One array: List(Int32) (nullable elements)
   • Other array: List(non-null Int32) (non-nullable elements)

   Root Cause

   In CometArrayExcept.convert, we're passing the raw Spark expressions without normalizing the element type nullability. DataFusion is strict about type compatibility and rejects the mismatch.

   Proposed Fix

   Normalize element type nullability before calling the native function:

   // In CometArrayExcept.convert
   val leftType = expr.left.dataType.asInstanceOf[ArrayType]
   val rightType = expr.right.dataType.asInstanceOf[ArrayType]
   
   // Cast to nullable if needed to ensure compatibility
   val normalizedLeft = if (!leftType.containsNull) {
     Cast(expr.left, ArrayType(leftType.elementType, containsNull = true))
   } else {
     expr.left
   }
   
   val normalizedRight = if (!rightType.containsNull) {
     Cast(expr.right, ArrayType(rightType.elementType, containsNull = true))
   } else {
     expr.right
   }

   This ensures both arrays have the same element nullability before passing to DataFusion.

   Testing

   The fix should enable the ignored test case in array_except.sql:

   query ignore(https://github.com/apache/datafusion-comet/issues/3646)
   SELECT array_except(array(1, 2, 3), array(2, 3, 4)), array_except(array(1, 2), array()), array_except(array(), array(1)), array_except(cast(NULL as array<int>), array(1))

   Next Steps

   1. Implement the fix in spark/src/main/scala/org/apache/comet/serde/arrays.scala
   2. Add unit tests
   3. Enable the ignored test case
   4. Verify with the full test suite

   I'll submit a PR shortly. Working on it now!

   cc: @andygrove @parthchandra @kazuyukitanimura
9. contentforge-press

   added a commit that references this issue on Oct 2, 2026

   Fix nested array regression in array_except type normalization

   ...

   1dbc948