You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native lpad and rpad fail when the string argument evaluates to a scalar, such as a scalar subquery inside IF #7245
lpad and rpad fail natively when their string argument references a column but evaluates to a scalar for the whole batch. CometStringLPad and CometStringRPad only keep a Literal string on Spark, so IF(id > 0, (SELECT max(s) FROM strs), s) is serialized. When every row in the batch takes the subquery branch, the conditional returns a scalar, and the native read_side_padding kernel only accepts an array as its first argument:
org.apache.comet.CometNativeException: Unsupported arguments [Scalar(Utf8("cd")), Scalar(Int32(5)), Scalar(Utf8("0"))] for function rpad/lpad/read_side_padding.
Spark returns the padded value. Default configs are enough to hit this.
Steps to reproduce
Seq((1, "ab"), (2, "cd")).toDF("id", "s").repartition(1).write.parquet("/tmp/strs")
spark.read.parquet("/tmp/strs").createOrReplaceTempView("strs")
sql("SELECT id, lpad(IF(id > 0, (SELECT max(s) FROM strs), s), 5, '0') FROM strs").collect()
rpad, and CASE WHEN id > 0 THEN (SELECT max(s) FROM strs) ELSE s END as the string argument, fail the same way.
Reproduced on main at c4a0b1f with the default Spark 4.1 profile.
Expected behavior
Spark's rows: 000cd for both rows with lpad, and cd000 with rpad.
Additional context
This is the padding half of the scalar-argument problem in #3336. #3336 says hour, minute, second and unix_timestamp need ConstantFolding disabled to fail, but a plain scalar subquery reaches them with default configs too. SELECT hour((SELECT max(ts) FROM t)) FROM t fails with hour(scalar) should be fold in Spark JVM side., and unix_timestamp fails the same way. #5889 carries fixes for both halves that don't depend on its struct support.
Activity