Activity

1. sgugger commented on Jul 21, 2020

   

   sgugger

   on Jul 21, 2020

   Collaborator

   More actions

   For some reasons those functions do not appear in the documentation, will have a look at why. The docstrings state of add_special_tokens states:

   Add a dictionary of special tokens (eos, pad, cls...) to the encoder and link them to class attributes. If special tokens are NOT in the vocabulary, they are added to it (indexed starting from the last index of the current vocabulary).
   Using add_special_tokens will ensure your special tokens can be used in several ways:

   • special tokens are carefully handled by the tokenizer (they are never split)
   • you can easily refer to special tokens using tokenizer class attributes like tokenizer.cls_token. This makes it easy to develop model-agnostic training and fine-tuning scripts.

   Though the second point would not apply in your case. The docstring of add_tokens states:

   Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to it with indices starting from length of the current vocabulary.
   When possible, special tokens are already registered for provided pretrained models (ex: BertTokenizer cls_token is already registered to be '[CLS]' and XLM's one is also registered to be '')

   Hope that helps!

   👍React with 👍4Reacted by Minh Quang Nguyen, gugaio, NivinaNull and LukeLin

   🚀React with 🚀1Reacted by Minh Quang Nguyen
2. kugwzk commented on Jul 22, 2020

   

   kugwzk

   on Jul 22, 2020

   Author

   More actions

   Thank a lot for your help. Therefore, if I use the add_tokens(), the tokens may still split?

   👍React with 👍4Reacted by Bilal Ghanem, NivinaNull, LukeLin and Mengqi Li
3. kugwzk

   closed this as completedon Jul 31, 2020
4. TobiasJacob commented on Jul 22, 2021

   

   TobiasJacob

   on Jul 22, 2021

   More actions

   I would also appreciate some clarification on the difference between the function and when to use which one.

   👍React with 👍5Reacted by Maja Stahl, Terry, NivinaNull, mdabbah-deci and LukeLin
5. LukeLIN-web commented on Mar 5, 2025

   

   LukeLIN-web

   on Mar 5, 2025

   More actions

   Thank a lot for your help. Therefore, if I use the add_tokens(), the tokens may still split?

   In my practice yes,

   👀React with 👀1Reacted by Mengqi Li
6. LukeLIN-web commented on Mar 5, 2025

   

   LukeLIN-web

   on Mar 5, 2025

   Last edited by LukeLIN-web

   More actions

   I would also appreciate some clarification on the difference between the function and when to use which one.

   For example if you want your <SEG> as whole, don't split to < , SEG, > three token, you need add_special_tokens()

   👀React with 👀1Reacted by Mengqi Li