Skip to content

Native lpad and rpad fail when the string argument evaluates to a scalar, such as a scalar subquery inside IF #7245

Open

Description

@andygrove

Describe the bug

lpad and rpad fail natively when their string argument references a column but evaluates to a scalar for the whole batch. CometStringLPad and CometStringRPad only keep a Literal string on Spark, so IF(id > 0, (SELECT max(s) FROM strs), s) is serialized. When every row in the batch takes the subquery branch, the conditional returns a scalar, and the native read_side_padding kernel only accepts an array as its first argument:

org.apache.comet.CometNativeException: Unsupported arguments [Scalar(Utf8("cd")), Scalar(Int32(5)), Scalar(Utf8("0"))] for function rpad/lpad/read_side_padding.

Spark returns the padded value. Default configs are enough to hit this.

Steps to reproduce

Seq((1, "ab"), (2, "cd")).toDF("id", "s").repartition(1).write.parquet("/tmp/strs")
spark.read.parquet("/tmp/strs").createOrReplaceTempView("strs")
sql("SELECT id, lpad(IF(id > 0, (SELECT max(s) FROM strs), s), 5, '0') FROM strs").collect()

rpad, and CASE WHEN id > 0 THEN (SELECT max(s) FROM strs) ELSE s END as the string argument, fail the same way.

Reproduced on main at c4a0b1f with the default Spark 4.1 profile.

Expected behavior

Spark's rows: 000cd for both rows with lpad, and cd000 with rpad.

Additional context

This is the padding half of the scalar-argument problem in #3336. #3336 says hour, minute, second and unix_timestamp need ConstantFolding disabled to fail, but a plain scalar subquery reaches them with default configs too. SELECT hour((SELECT max(ts) FROM t)) FROM t fails with hour(scalar) should be fold in Spark JVM side., and unix_timestamp fails the same way. #5889 carries fixes for both halves that don't depend on its struct support.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:expressionsExpression evaluationbugSomething isn't workingpriority:mediumFunctional bugs, performance regressions, broken featuresrequires-triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions