[Spark 4.0] Add string collation support #2190
Description
What is the problem the feature request solves?
A major new feature of Spark 4.0.0 is string collation support. Unfortunately, DataFusion does not support string collation (apache/datafusion#9192); therefore, Comet does not support it either.
Describe the potential solution
In the short term, we likely need to fall back to Spark for any queries involving string collation.
Additional context
https://www.databricks.com/blog/introducing-collations-databricks
https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-collation
Activity
[-][native_iceberg_compat] String collation support[/-][+][Spark 4.0] Add string collation support[/+]parthchandra commentedon Aug 22, 2025
I don't understand what this needs from
native_iceberg_compat. Collation sequences are needed for sort. Is the requirement to ensure that the Spark type does not drop the collation information during the scan?andygrove commentedon Aug 22, 2025
I really don't know yet. For now, we just want to fall back to Spark for any queries where the schema has string collation types and the scan is the first place where we see this.
It has been a while since I started looking into this, but IIRC, I was getting errors in the scan when converting the Spark schema to the Arrow schema because we did not recognize the string collation types.
parthchandra commentedon Aug 22, 2025
Hmm. Spark Schema -> Arrow Schema -> Spark Schema is going to drop the collation type even if we address this in the conversion method.
Spark's conversion methods don't worry about it - https://github.com/apache/spark/blob/c863717831029c44069ec17dcf8091147e14a7b3/sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala#L41
parthchandra commentedon May 13, 2026
This is non-trivial to implement. Here's the summary of my research - https://github.com/parthchandra/datafusion-comet/wiki/Spark-4-Collation-Support-in-Comet
docs(audit): cross-reference filed follow-up issues from string audit…
fix: keep collated array_join and array_intersect native via codegen …
fix: array expression audit follow-ups (#4503) (#4713)
fix: array expression audit follow-ups (apache#4503) (apache#4713)
fix: array expression audit follow-ups (apache#4503) (apache#4713)
dwsmith1983 commentedon Sep 11, 2026
The short-term goal stated here is in place:
CometScanRulefalls back for any scan whose schema contains a collated string type (spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala),QueryPlanSerde.supportedScalarSortElementTyperejects collated sort keys, and #4948 added the same for collated predicate operands, so Spark 4 collation queries run correctly on Spark rather than producing wrong byte-order results. What remains is actual collation-aware native execution, which DataFusion still lacks and which parthchandra's research note scopes out as non-trivial; that is a different piece of work from the fallback this issue asked for and deserves its own issue with that note as the starting point, so the tracking here can close.