Skip to content

CJK characters break some features like @ mentions #3734

Closed
Closed
CJK characters break some features like @ mentions#3734
Assignees
kommander
Labels
bugSomething isn't workingopentuiThis relates to changes in v1.0, now that opencode uses opentui

Description

@yukukotani
yukukotani
opened on Nov 2, 2025

Description

Expects that inputting 世界 @ shows auto-completion of @ mentions, but actually shows nothing.

The problem is here.

https://github.com/sst/opencode/blob/89492b30026e9b635b4b7e57877cbe7c8821b8b7/packages/opencode/src/cli/cmd/tui/component/prompt/autocomplete.tsx#L374-L376

props.input().visualCursor.offset is based on Unicode codepoints and counts 世界 @ as 5, while props.value.at counts it as 3. Also props.input().logicalCursor.offset returns 5.

Looking more closely at input, it seems expected behaviour in the test.

https://github.com/sst/opentui/blob/54365ebca04b8e556338d417d33a8d042dcc3169/packages/core/src/editor-view.test.ts#L326-L355

I'm not sure what logical means here, but maybe it should returns 3? Or to align JavaScript side to zig implementation, Bun.stringWidth looks similar but has some difference like \n proceed the cursor by 1 but Bun.stringWidth returns 0.

OpenCode version

v1.0.10

Steps to reproduce

input @ after CJK characters, like 世界 @

Screenshot and/or share link

No response

Operating System

No response

Terminal

No response

Activity

  1. yukukotani
    added
    bugSomething isn't working
    on Nov 2, 2025
  2. github-actions
    added
    opentuiThis relates to changes in v1.0, now that opencode uses opentui
    on Nov 2, 2025
  3. yukukotani
    changed the title [-]CJK characters break features like `@` mentions[/-] [+]CJK characters break some features like `@` mentions[/+] on Nov 2, 2025

  4. github-actions commented on Nov 2, 2025

    @github-actions
    github-actionsBot
    on Nov 2, 2025 – with GitHub Actions
    Contributor

    This issue might be a duplicate of existing issues. Please check:

    All of these issues seem to stem from similar underlying problems with Unicode/CJK character handling in the TUI components. The root cause appears to be inconsistent character width calculation between Unicode codepoints and display width, which is exactly what you've identified in the autocomplete.tsx file.

    Feel free to ignore if none of these address your specific case with @ mention autocomplete functionality.

  5. yukukotani commented on Nov 2, 2025

    @yukukotani
    yukukotani
    on Nov 2, 2025
    ContributorAuthor

    @kommander FYI since I assume you are working on this

  6. yukukotani commented on Nov 2, 2025

    @yukukotani
    yukukotani
    on Nov 2, 2025
    ContributorAuthor

    I can take this on if I get a bit of guidance on the right approach.

  7. kommander commented on Nov 2, 2025

    @kommander
    kommander
    on Nov 2, 2025
    Collaborator

    That is a tricky one. The 3 is in "number of utf16 characters", while the 5 is in "in number of terminal cells", so visualCursor.offset is actually number of cells. Currently the underlying EditBuffer in zig uses "number of cells occupied" as main coordinate system for the cursor. Which is bad.

    However, there are methods to convert to a byte offset. So I think opentui needs to expose some methods to work with the Editbuffer properly.

    String.at is not grapheme aware, it coincidentally works in this specific case, but for combining characters like acutes or devanagari viramas it would treat the combining chars as separate chars.

    For example 世 is U+4E16 in three utf8 bytes.

    Edit: I'll follow up on this.

  8. kommander commented on Nov 3, 2025

    @kommander
    kommander
    on Nov 3, 2025
    Collaborator

    I think the simplest solution to make this work properly, without going into the huge refactor that is needed for the text handling in opentui, is exposing a getTextRange method for the TextBuffer in opentui, so opencode can use that instead of String.at. Then charBeforeCursor would be something like input.getTextRange(cursor.offset - 1, cursor.offset). I'll check how hard it would be to implement that. Would at least make it reliable for now, no matter how opentui treats text internally.

  9. kommander
    added a commit that references this issue on Nov 3, 2025

    use new opentui getTextRange method and Bun.stringWidth instead of va…

    6deaf54

  10. kommander commented on Nov 3, 2025

    @kommander
    kommander
    on Nov 3, 2025
    Collaborator

    This should be mitigated in 1.0.16

  11. yukukotani commented on Nov 3, 2025

    @yukukotani
    yukukotani
    on Nov 3, 2025
    ContributorAuthor

    @kommander Thanks, the problem of auto-completion is solved!

    Syntax highlighting is still broken. It seems that extmarks from opentui is based on terminal cells while PromptInfo.parts is based on UTF16 characters.

    https://github.com/sst/opencode/blob/d06afd87e5a0325730b11478eb49af4ced12608a/packages/opencode/src/cli/cmd/tui/component/prompt/autocomplete.tsx#L87-L108

    Image
  12. thdxr
    assigned
    kommander
    on Nov 4, 2025
  13. kommander
    added a commit that references this issue on Nov 12, 2025

    fix CJK extmarks sim offsets (#308)

    5c5acef

  14. kommander commented on Nov 13, 2025

    @kommander
    kommander
    on Nov 13, 2025 – with Linear
    Collaborator

    Should be better in the latest version?

  15. zenyr commented on Nov 27, 2025

    @zenyr
    zenyr
    on Nov 27, 2025
    Contributor

    Update as of v1.0.117

    1. The original issue seems mostly fixed ~2 weeks ago
      👇 Working fine now 👍
      Image

    2. However [Pasted] is a bit broken AFTER submitting
      👇 CJK/Emojis breaking [Pasted] marker 🥲 (Before & after)
      Image

      안녕 {cmd+v} 하세요

  16. msmps
    added a commit that references this issue on Jan 8, 2026

    fix CJK extmarks sim offsets (#308)

    4232f31
  17. thdxr
    closed this as completedon Mar 25, 2026
  18. Rwanbt
    added a commit that references this issue on May 5, 2026

    use new opentui getTextRange method and Bun.stringWidth instead of va…

    e13b82f
  19. Rwanbt
    added a commit that references this issue on Jun 14, 2026

    use new opentui getTextRange method and Bun.stringWidth instead of va…

    80ff64a
  20. meSingh
    added a commit that references this issue on Sep 6, 2026

    use new opentui getTextRange method and Bun.stringWidth instead of va…

    0356aa2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingopentuiThis relates to changes in v1.0, now that opencode uses opentui

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions