Skip to content

Initcap behaves differently in Spark and in DataFusion (also Comet) #1052

Closed
#2387
Closed
Initcap behaves differently in Spark and in DataFusion (also Comet)#1052
#2387
Labels
bugSomething isn't working

Description

@Blizzara
Blizzara
opened on Nov 4, 2024

Describe the bug

DataFusion's initcap behaves differently than Spark's. While both do "upper-case the first letter of each word and lowercase others", Spark considers as words anything separated by whitespace (' '), while DataFusion considers anything separated by non-ascii-alphanumeric as words. (DF's code would also fail to uppercase or lowercase non-ascii chars, but that doesn't materialize as a separate issue as it considers them separators already in the first place.)

#1051 shows the problem by adding two cases to the test, one using a dash and one using non-ascii letters (from Finnish).

== Results ==
!== Correct Answer - 7 ==       == Spark Answer - 7 ==
 struct<initcap(name):string>   struct<initcap(name):string>
 [James Smith]                  [James Smith]
 [James Smith]                  [James Smith]
![James Ähtäri]                 [James äHtäRi]
 [Michael Rose]                 [Michael Rose]
 [Rames Rose]                   [Rames Rose]
![Robert Rose-smith]            [Robert Rose-Smith]
 [Robert Williams]              [Robert Williams]

Steps to reproduce

Call initcap with an input containing non-ascii-alphanumeric non-whitespace characters

Expected behavior

Match Spark

Additional context

No response

Activity

  1. Blizzara
    added
    bugSomething isn't working
    on Nov 4, 2024
  2. andygrove
    added this to the 0.4.0 milestone on Nov 4, 2024

  3. viirya commented on Nov 4, 2024

    @viirya
    viirya
    on Nov 4, 2024
    Member

    Thanks for reporting this bug.

  4. comphead commented on Dec 13, 2024

    @comphead
    comphead
    on Dec 13, 2024
    Contributor

    DataFusion InitCap doesn't support nonASCII
    Tracked in apache/datafusion#13711

  5. comphead commented on Dec 13, 2024

    @comphead
    comphead
    on Dec 13, 2024
    Contributor
    ![Robert Rose-smith]            [Robert Rose-Smith]

    already fixed in latest DF

    > select initcap('robert rose-smith');
+------------------------------------+
| initcap(Utf8("robert rose-smith")) |
+------------------------------------+
| Robert Rose-Smith                  |
+------------------------------------+
1 row(s) fetched. 
Elapsed 0.005 seconds.

    UPD: Spark returns

    scala> spark.sql("select initcap('robert rose-smith')").show(false)
+--------------------------+
|initcap(robert rose-smith)|
+--------------------------+
|Robert Rose-smith         |
+--------------------------+

    But reg to DF SQL policy apache/datafusion#13706 DF supports a builtin functions to be in sync with PG which returns Robert Rose-Smith and Comet likely need to implement own UDF function to cover this case

  6. Blizzara commented on Dec 13, 2024

    @Blizzara
    Blizzara
    on Dec 13, 2024
    ContributorAuthor

    already fixed in latest DF

    How is that fixed? :) The diff I showed is exactly the same as the diff you meantion below, no?

  7. comphead commented on Dec 13, 2024

    @comphead
    comphead
    on Dec 13, 2024
    Contributor

    already fixed in latest DF

    How is that fixed? :) The diff I showed is exactly the same as the diff you meantion below, no?

    Please refer to message update above

  8. kazuyukitanimura commented on Dec 16, 2024

    @kazuyukitanimura
    kazuyukitanimura
    on Dec 16, 2024
    Contributor

    Looks like unicode init cap is getting fixed in DF apache/datafusion#13752

  9. kazuyukitanimura commented on Jan 9, 2025

    @kazuyukitanimura
    kazuyukitanimura
    on Jan 9, 2025
    Contributor

    The fix is already included in https://github.com/apache/datafusion/commits/branch-44/

  10. kazuyukitanimura
    modified the milestones: 0.4.0, 0.5.0 on Jan 11, 2025
  11. andygrove
    modified the milestones: 0.5.0, 0.6.0 on Jan 24, 2025
  12. andygrove
    removed this from the 0.6.0 milestone on Feb 4, 2025

  13. andygrove commented on Feb 4, 2025

    @andygrove
    andygrove
    on Feb 4, 2025
    Member

    Removing from 0.6.0 since this expression is disabled by default

  14. andygrove commented on Sep 12, 2025

    @andygrove
    andygrove
    on Sep 12, 2025
    Member

    This expression is now documented as incompatible and disabled by default, but users can opt it. I will close this issue.

  15. andygrove
    closed this as completedon Sep 12, 2025

  16. kazuyukitanimura commented on Sep 12, 2025

    @kazuyukitanimura
    kazuyukitanimura
    on Sep 12, 2025
    Contributor

    This expression is now documented as incompatible and disabled by default, but users can opt it. I will close this issue.

    @andygrove I thought this is fixed in DF44 with this commit apache/datafusion@a267784 but did you mean that the fix is not sufficient?

  17. andygrove commented on Sep 12, 2025

    @andygrove
    andygrove
    on Sep 12, 2025
    Member

    @kazuyukitanimura The unicode issue maybe is fixed, but there are other incompatibilities still.

    !== Correct Answer - 7 ==       == Spark Answer - 7 ==
 struct<initcap(name):string>   struct<initcap(name):string>
 [James Smith]                  [James Smith]
 [James Smith]                  [James Smith]
 [James Ähtäri]                 [James Ähtäri]
 [Michael Rose]                 [Michael Rose]
 [Rames Rose]                   [Rames Rose]
![Robert Rose-smith]            [Robert Rose-Smith]
 [Robert Williams]              [Robert Williams]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions