mT5 additional_special_tokens seems not work #9747

Activity

1. LysandreJik commented on Jan 25, 2021

   

   LysandreJik

   on Jan 25, 2021

   Member

   More actions

   Hi! Could you either post a link to your notebook or your code as actual code? Images makes it impossible to copy/paste or for others to search with similar issues. Thanks.
2. PiggyFan commented on Jan 28, 2021

   

   PiggyFan

   on Jan 28, 2021

   Author

   More actions

   Hi! Could you either post a link to your notebook or your code as actual code? Images makes it impossible to copy/paste or for others to search with similar issues. Thanks.

   Here is colab notebook link. Thanks for your reply.

   https://colab.research.google.com/drive/1fbp7VvnUvbf5r8CSitOg2pDZyM47y9xj?usp=sharing
3. LysandreJik commented on Jan 28, 2021

   

   LysandreJik

   on Jan 28, 2021

   Member

   More actions

   Hi! Indeed, this is a bit misleading. Special tokens are considered as those that were in the pre-training, that is: unknown tokens, bos tokens, eos tokens, etc.

   If you want to use special tokens that you use as special tokens, I would argue it is better to define them as simple tokens. Therefore doing the following:

   >>> from transformers import MT5Tokenizer, MT5ForConditionalGeneration
   ... import torch
   ... special_tokens = ['<POS>', '<NEG>','<CON_START>','<START>','<END>'] # Set the special tokens
   ... mt5_add_tokenizer = MT5Tokenizer.from_pretrained("google/mt5-small")
   ... mt5_add_tokenizer.add_tokens(special_tokens)
   ... print(mt5_add_tokenizer.tokenize("<POS> <CON_START> the biscuits and gravy were . <START>"))

   You'll get the following output:

   ['<POS>', '<CON_START>', '▁the', '▁b', 'iscuit', 's', '▁and', '▁grav', 'y', '▁were', '▁', '.', '<START>']
   

   Let me know if that makes sense.

   👍React with 👍2Reacted by PiggyFan and TIGEERR
4. PiggyFan commented on Jan 29, 2021

   

   PiggyFan

   on Jan 29, 2021

   Author

   More actions

   Thank you very much, it works well.
   I was confused by issue5940 which mentioned "special tokens are carefully handled by the tokenizer (they are never split)". If use add_token() method, the additional simple tokens may still split?
   In addition, the code in Colab notebook as above link, OpenaiTokenizer should use add_tokens method rather than add_special_tokens (define them as a simple tokens) ?
5. LysandreJik commented on Jan 29, 2021

   

   LysandreJik

   on Jan 29, 2021

   Member

   More actions

   For tokens that cannot be identified as being either:

   • eos_token
   • bos_token
   • cls_token
   • unk_token
   • pad_token
   • sep_token
   • mask_token

   Then I would recommend using the add_token method. These tokens shouldn't split either:

   >>> from transformers import BertTokenizer
   >>> tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
   >>> tokenizer.add_tokens(["<CON_START>", "<CON_ST"])
   2
   >>> tokenizer.tokenize("<CON_START>")
   ['<CON_START>']
   >>> tokenizer.tokenize("<CON_STAR>")
   ['<CON_ST', 'AR', '>']
6. PiggyFan commented on Feb 1, 2021

   

   PiggyFan

   on Feb 1, 2021

   Author

   More actions

   Thank for your help and recommend.
7. PiggyFan

   closed this as completedon Feb 1, 2021
8. Victordongy commented on Apr 6, 2022

   

   Victordongy

   on Apr 6, 2022

   More actions

   For tokens that cannot be identified as being either:

   • eos_token
   • bos_token
   • cls_token
   • unk_token
   • pad_token
   • sep_token
   • mask_token

   Then I would recommend using the add_token method. These tokens shouldn't split either:

   >>> from transformers import BertTokenizer
   >>> tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
   >>> tokenizer.add_tokens(["<CON_START>", "<CON_ST"])
   2
   >>> tokenizer.tokenize("<CON_START>")
   ['<CON_START>']
   >>> tokenizer.tokenize("<CON_STAR>")
   ['<CON_ST', 'AR', '>']

   Hi it seems add_tokens not working with AutoTokenizer , but work with specific-defined tokenizer like BertTokenizer,

   special_tokens = ["-Title-"]
   #tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
   tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
   tokenizer.add_tokens(special_tokens)
   tokenizer.tokenize("-Title-")
   # ['-', 'title', '-']
9. ArthurZucker commented on Jun 1, 2023

   

   ArthurZucker

   on Jun 1, 2023

   Collaborator

   More actions

   He! Thanks for reporting. This should be fixed by #23909