You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
translate falls back to Spark by default instead of using the codegen dispatcher like the other string functions #5585
The reason is real — DataFusion's translate iterates Unicode graphemes where Spark uses code points, and substitutes U+0000 rather than treating it as a deletion sentinel. But with no CodegenDispatchFallback mixin, the consequence is that translate falls the whole projection back to Spark unless the user sets spark.comet.expression.StringTranslate.allowIncompatible=true and accepts wrong answers.
Every comparable string function in the same file already has the better shape: compatible by default through the dispatcher, native as an explicit opt-in. See CometInitCap (:146), CometStringReplace (:177), CometCaseConversionBase for upper / lower (:50), and the regex family at :368, :470, :508, :545, :598.
Fix: mix CodegenDispatchFallback into CometStringTranslate. StringTranslate is a plain codegen expression over StringType, so canHandle admits it.
Once that lands, docs/source/user-guide/latest/expressions.md should report translate as Hybrid, and the existing note about the grapheme/code-point difference becomes an opt-in caveat rather than a description of default behavior.
Fixed by #5032, which mixes CodegenDispatchFallback into CometStringTranslate so translate runs through the codegen dispatcher by default, and string_translate.sql covers the grapheme and U+0000 cases that separate it from the native path.
Activity
adibmbrk commentedon Aug 31, 2026
take
andygrove commentedon Sep 28, 2026
Fixed by #5032, which mixes CodegenDispatchFallback into CometStringTranslate so translate runs through the codegen dispatcher by default, and string_translate.sql covers the grapheme and U+0000 cases that separate it from the native path.