Skip to content

translate falls back to Spark by default instead of using the codegen dispatcher like the other string functions #5585

Closed

Description

@andygrove

Part of #5572.

CometStringTranslate reports Incompatible unconditionally:

override def getSupportLevel(expr: StringTranslate): SupportLevel = Incompatible(
  Some(incompatReason))

spark/src/main/scala/org/apache/comet/serde/strings.scala:119-120

The reason is real — DataFusion's translate iterates Unicode graphemes where Spark uses code points, and substitutes U+0000 rather than treating it as a deletion sentinel. But with no CodegenDispatchFallback mixin, the consequence is that translate falls the whole projection back to Spark unless the user sets spark.comet.expression.StringTranslate.allowIncompatible=true and accepts wrong answers.

Every comparable string function in the same file already has the better shape: compatible by default through the dispatcher, native as an explicit opt-in. See CometInitCap (:146), CometStringReplace (:177), CometCaseConversionBase for upper / lower (:50), and the regex family at :368, :470, :508, :545, :598.

Fix: mix CodegenDispatchFallback into CometStringTranslate. StringTranslate is a plain codegen expression over StringType, so canHandle admits it.

Once that lands, docs/source/user-guide/latest/expressions.md should report translate as Hybrid, and the existing note about the grapheme/code-point difference becomes an opt-in caveat rather than a description of default behavior.

Activity

  1. adibmbrk commented on Aug 31, 2026

    @adibmbrk

    take

  2. andygrove commented on Sep 28, 2026

    @andygrove
    MemberAuthor

    Fixed by #5032, which mixes CodegenDispatchFallback into CometStringTranslate so translate runs through the codegen dispatcher by default, and string_translate.sql covers the grapheme and U+0000 cases that separate it from the native path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions