You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug] str_to_map does not honour Spark 4.1.1 legacy.truncateForEmptyRegexSplit #4477
Spark 4.1.1 added a legacySplitTruncate flag to StringToMap (and several other split-based expressions) that is driven by spark.sql.legacy.truncateForEmptyRegexSplit. When the flag is enabled (legacy mode), CollationAwareUTF8String.splitSQL truncates trailing empty matches from the split result. Comet's native str_to_map does not honour this flag and always behaves as if it were false (the non-legacy default).
Surfaced by the map-expressions audit (collection PR queue).
Spark 4.1.1 with the legacy flag enabled would truncate the trailing empty ("","null") entry; the non-legacy default and Comet keep it.
Expected behavior
Either propagate the flag through the proto so the native impl can branch on it, or downgrade CometStrToMap to Incompatible when spark.sql.legacy.truncateForEmptyRegexSplit=true on Spark 4.1.1+.
Additional context
Comet wiring: QueryPlanSerde.scala -> classOf[StringToMap] -> CometStrToMap (a one-liner over CometScalarFunction("str_to_map")).
Activity