You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Array functions fall back to Spark for binary and struct element types (ArraysBase type gate) #5582
ArraysBase.isTypeSupported (spark/src/main/scala/org/apache/comet/serde/arrays.scala:888) rejects BinaryType outright and StructType pending #1307, and childTypesSupportLevel turns that into Unsupported:
.map(dt =>Unsupported(Some(s"data type not supported: $dt")))
arrays.scala:911
Seven serdes mix in ArraysBase: CometArrayRemove (:36), CometFlatten (:700), CometArrayPosition (:791), CometShuffle (serde/collectionOperations.scala:64), CometArrayInsert (:409), CometCreateArray (:503) and CometArrayReverse (:572). None routes the declined types through the dispatcher, so array<struct<...>> and array<binary> — both very ordinary shapes in nested Parquet data — fail the whole projection back to Spark.
docs/source/user-guide/latest/expressions.md already records this for three of them: array_position, flatten and shuffle all carry "Binary/struct/map elements fall back".
Both element types are admitted by CometBatchKernelCodegen.isSupportedDataType, which handles ArrayType, StructType and MapType recursively and accepts BinaryType as a leaf.
One important carve-out: the "null elements" half of those doc notes is not dispatchable. NullType is absent from isSupportedDataType, so array<null> will still fall back and the doc notes should be updated to say so rather than being deleted wholesale.
CometShuffle is nondeterministic, which is fine — canHandle explicitly admits nondeterministic and stateful expressions, and each cache entry gets a single init(partitionIndex) call. Worth a test that confirms the shuffled permutation still matches Spark through the dispatcher.
CometArrayInsert, CometCreateArray and CometArrayReverse also decline inside convert, which the dispatcher cannot see; see the prerequisite issue on convert-side declines.
Activity
chetan26 commentedon Sep 11, 2026
take