[jiongjiongli]

Fix Bug

#35447 : Tokenizer does not split text according to newly added input tokens

Resolution

Method Trie.split: Add steps to ignore partial match that should be removed

@ArthurZucker and @itazap

[ArthurZucker]

Hey ! Sorry but I cannot seem to reproduce!

[jiongjiongli]

Hey ! Sorry but I cannot seem to reproduce!

Hello @ArthurZucker , this issue repos using tokenizer.add_tokens(["red", "e"]) rather than tokenizer.add_tokens(["e"]) .

Below is repro code and actual result:

# Code:
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-cased", use_fast=False)
tokenizer.add_tokens(["red", "e"])
output_tokens = tokenizer.tokenize("read")
print(f"Output tokens: {output_tokens}")

# Actual Output:
Output tokens: ['read']

[ArthurZucker]

Right, sorry!
Could you add another test with another model for example? 🤗
Seems to not work for ? for example or for emojis

[ArthurZucker]

Thanks for digging into the Trie.split behaviour! Since then src/transformers/tokenization_utils.py was removed in #40936 (the Trie now lives in src/transformers/tokenization_python.py), and #35447 has been closed, so this branch can't be merged anymore. If you can still reproduce the issue on main, please open a fresh PR there with the extra test cases (e.g. ? / emojis) mentioned in the review. Closing for now.