You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native execution fails with offset overflow when a large scalar string is expanded to the batch length #7247
When a large string scalar, from a scalar subquery or a literal, is expanded to the batch length, the result can exceed Arrow's 32-bit offsets. With a 256 KiB string and the default 8,192-row batch, the expanded column needs 2 GiB of value bytes. Native execution then panics with offset overflow, or fails with byte array offset overflow. Spark evaluates the same queries row by row and succeeds.
Steps to reproduce
spark.conf.set("spark.sql.adaptive.enabled", "false")
sql("SELECT repeat('x', 262144) AS s").repartition(1).write.parquet("/tmp/wide")
sql("SELECT CAST(id % 2 AS INT) AS k FROM range(8192)")
.repartition(1).write.parquet("/tmp/outer_ids")
Seq("wide", "outer_ids").foreach(t =>
spark.read.parquet(s"/tmp/$t").createOrReplaceTempView(t))
Query
Spark
Comet
SELECT k, count((SELECT max(s) FROM wide)) FROM outer_ids GROUP BY k
(0, 4096), (1, 4096)
native panic: offset overflow
SELECT count(*) FROM (SELECT k FROM outer_ids DISTRIBUTE BY k, (SELECT max(s) FROM wide))
8192
native panic: offset overflow
SELECT sum(length(c)) FROM (SELECT concat((SELECT max(s) FROM wide), CAST(k AS STRING)) AS c FROM outer_ids)
2147491840
byte array offset overflow
SELECT sum(length(c)) FROM (SELECT concat(repeat('x', 262144), CAST(k AS STRING)) AS c FROM outer_ids)
2147491840
byte array offset overflow
The last query has no subquery. The literal is enough. All four plans run natively, through CometHashAggregate, CometExchange and CometProject.
Reproduced on main at c4a0b1f with the default Spark 4.1 profile.
Expected behavior
The same results as Spark.
Additional context
count of a scalar doesn't need the expanded column, and a hash partitioner could hash a scalar once. For kernels whose output really exceeds 2 GiB, the batch would have to be split, use 64-bit offsets, or fall back. #4718 and #7192 are other offset-size failures. #5889 adds bounded evaluation for merged struct subqueries only, so these plain-subquery and literal shapes still fail on that branch.
Activity
zhangfengcdt commentedon Oct 27, 2026
take