Repository navigation
GPT2Tokenizer strips spaces surrounding special tokens #7901
Description
Environment info
transformersversion: 3.3.1- Platform: Darwin-19.6.0-x86_64-i386-64bit
- Python version: 3.7.9
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
Who can help
@mfuntowicz, this seems to be an issue related to tokenization. So I hope you are the right person to ping here.
Information
Model I am using (Bert, XLNet ...):
The problem arises when using:
- the official example scripts: (give details below)
- my own modified scripts: (give details below)
The tasks I am working on is:
- an official GLUE/SQUaD task: (give the name)
- my own task or dataset: (give details below)
To reproduce
Minimal working example:
import os
os.makedirs('models/', exist_ok=True)
from transformers import GPT2Tokenizer
from tokenizers import ByteLevelBPETokenizer
open('train.txt', 'w').write('Training data including a <special> token.')
special_tokens = ['<special>']
bpe_tokenizer = ByteLevelBPETokenizer()
bpe_tokenizer.train(
files=['train.txt'],
special_tokens=special_tokens
)
bpe_tokenizer.save_model('models/')
gpt2_tokenizer = GPT2Tokenizer.from_pretrained(
'models/',
additional_special_tokens=special_tokens,
)When encoding below text, the two tokenizers yield different outputs:
>>> text = 'A <special> token.'
>>> bpe_tokenizer.encode(text).tokens
['A', 'Ġ', '<special>', 'Ġ', 't', 'o', 'k', 'e', 'n', '.']
>>> gpt2_tokenizer.tokenize(text)
['A', '<special>', 't', 'o', 'k', 'e', 'n', '.'] # <----- Note the missing space (`Ġ`) around `<special>` Expected behavior
I would expect that both tokenizers give the same output when encoding the sentence. Furthermore, because GPT2Tokenizer seems to remove the spaces surrounding the special token, the decode(encode()) does not return the original string.
assert bpe_tokenizer.encode(text).tokens == ['A', 'Ġ', '<special>', 'Ġ', 't', 'o', 'k', 'e', 'n', '.']
assert gpt2_tokenizer.tokenize(text) == ['A', 'Ġ', '<special>', 'Ġ', 't', 'o', 'k', 'e', 'n', '.']It is possible that I misunderstand the GPT2Tokenizer API. Please advise if I should pass special_tokens in a different way. Thank you in advance.
Activity
stale commentedon Dec 24, 2020
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
jantrienes commentedon Dec 24, 2020
The issue is still present.
github-actions commentedon Apr 16, 2021
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
jantrienes commentedon Apr 17, 2021
This issue is still present in
transformers==4.5.1.LysandreJik commentedon Apr 19, 2021
Hey @jantrienes, sorry for getting back so late to you. The issue here is on our side - the added tokens work differently between the slow and fast tokenizers and we'll have to patch that. In the meantime, you can use
AddedTokens instead of the strings to define the strategy w.r.t the stripping of whitespace. This should behave normally:Note how the special tokens are defined using
AddedTokeninstead of a string. Unfortunately, these cannot be passed during initialization as you've done, but I'm fixing this in #11325.You can control the
AddedToken's behavior relative to whitespace using therstripandlstripkeyword arguments.github-actions commentedon May 15, 2021
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
bjascob commentedon Nov 29, 2021
This issue is present for "non-special" tokens in transformer 4.12.3. The code explicitly strips spaces from the surrounding tokens. It does NOT look for the
AddedTokenclass to get the behavior so there is no way to change this.See
tokenization_utils.py::tokenizer()line 517:This leads to incorrect behavior unless the added token is in the middle of a longer word.
I should point out that the situation is more complicated than just changing the behavior above. Even if you comment out those lines (at least with the T5Tokenizer) there are still no spaces around the added tokens because, just below those lines, the logic to concatenate all the tokens is also stripping the spaces as well (see
tokenized_text.extend(self._tokenize(token))).I'm not sure what the right solution is but this is in the base class so it's happening for a number of tokenizers including T5 and bart. It is also happening for T5Fast.
LysandreJik commentedon Nov 30, 2021
@SaulLu, would you like to take a look at this?
SaulLu commentedon Dec 1, 2021
Glad to look at this issue in more detail! I'll dive in tomorrow 🧐
jbmaxwell commentedon Jun 2, 2022
Yeah, I'm noticing spaces around my added tokens when creating a custom Bart tokenizer. Did this get resolved for you? Is there a way to work around it?