Skip to content

bug: null IN () returns false instead of null under legacy null-in-empty behavior #4786

Closed

Description

@mbutrovich

Describe the bug

For a null operand against an empty IN list, Comet always returns false. Spark returns null when spark.sql.legacy.nullInEmptyListBehavior=true. Comet ignores the config and the null operand.

Steps to reproduce

Empty IN lists are not expressible in SQL, so build via the DataFrame API; ConvertToLocalRelation and OptimizeIn must be disabled or the IN is folded away. Add to a suite extending CometTestBase (config exists in Spark 4+):

test("null IN empty list under legacy behavior") {
  assume(org.apache.comet.CometSparkSessionExtensions.isSpark40Plus)
  withSQLConf(
    "spark.sql.optimizer.excludedRules" ->
      ("org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation," +
        "org.apache.spark.sql.catalyst.optimizer.OptimizeIn"),
    "spark.sql.legacy.nullInEmptyListBehavior" -> "true") {
    val data: Seq[(Integer, Integer)] =
      Seq((Integer.valueOf(1), Integer.valueOf(1)), (null, Integer.valueOf(2)))
    withParquetTable(data, "t") {
      val df = sql("SELECT _1 AS a FROM t").select(col("a"), col("a").isin())
      checkSparkAnswer(df)
    }
  }
}
== Results ==
!== Spark Answer - 2 ==            == Comet Answer - 2 ==
 struct<a:int,(a IN ()):boolean>   struct<a:int,(a IN ()):boolean>
 [1,false]                         [1,false]
![null,null]                       [null,false]

Expected behavior

null IN () returns null under legacy behavior, matching Spark. (Under ANSI-on, the Spark 4 default, false is expected and Comet already matches, so only the legacy path is affected.)

Additional context

Found while enabling CometLocalTableScanExec by default (#4393), but reproduces over a Parquet scan. Upstream test: EmptyInSuite "IN with empty list".

Activity

  1. added theissue type on Jul 1, 2026
  2. comphead commented on Jul 1, 2026

    @comphead
    Contributor

    for legacy params we decided to fallback to spark

  3. self-assigned this
    on Jul 1, 2026
  4. added this to the 1.0.0 milestone on Jul 27, 2026
  5. added a commit that references this issue on Jul 29, 2026
    038fd6e
  6. added a commit that references this issue on Jul 29, 2026
    01313d7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:expressionsExpression evaluationbugSomething isn't workingcorrectnesspriority:criticalData corruption, silent wrong results, security issues

Type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions