You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
bug: null IN () returns false instead of null under legacy null-in-empty behavior #4786
For a null operand against an empty IN list, Comet always returns false. Spark returns null when spark.sql.legacy.nullInEmptyListBehavior=true. Comet ignores the config and the null operand.
Steps to reproduce
Empty IN lists are not expressible in SQL, so build via the DataFrame API; ConvertToLocalRelation and OptimizeIn must be disabled or the IN is folded away. Add to a suite extending CometTestBase (config exists in Spark 4+):
test("null IN empty list under legacy behavior") {
assume(org.apache.comet.CometSparkSessionExtensions.isSpark40Plus)
withSQLConf(
"spark.sql.optimizer.excludedRules"->
("org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation,"+"org.apache.spark.sql.catalyst.optimizer.OptimizeIn"),
"spark.sql.legacy.nullInEmptyListBehavior"->"true") {
valdata:Seq[(Integer, Integer)] =Seq((Integer.valueOf(1), Integer.valueOf(1)), (null, Integer.valueOf(2)))
withParquetTable(data, "t") {
valdf= sql("SELECT _1 AS a FROM t").select(col("a"), col("a").isin())
checkSparkAnswer(df)
}
}
}
null IN () returns null under legacy behavior, matching Spark. (Under ANSI-on, the Spark 4 default, false is expected and Comet already matches, so only the legacy path is affected.)
Additional context
Found while enabling CometLocalTableScanExec by default (#4393), but reproduces over a Parquet scan. Upstream test: EmptyInSuite "IN with empty list".
Activity
comphead commentedon Jul 1, 2026
for legacy params we decided to fallback to spark
docs: document known correctness issues in the compatibility guide (#…
fix: honour Spark's legacy `null IN ()` behavior (apache#5127)
fix: honour Spark's legacy `null IN ()` behavior (#5127) (#5134)
docs: document known correctness issues in the compatibility guide (a…
fix: honour Spark's legacy `null IN ()` behavior (apache#5127)