[remi-or]

This PR addresses an issue when loading a tiktoken tokenizer that has holes in its vocabulary.
Currently, when loading a tiktoken tokenizer, if there are tokens in added_tokens_decoder that are not in the base vocabulary, they will be added contiguously at the end of the vocab. For instance, Kimi 2.7 has a base vocab size of 163584 and 23 tokens in added_tokens_decoder: these tokens are added contiguously at the end of the vocab, increasing its final size to 163607. But if we look at the ids of those tokens, they don't match with where they end up: for instance, [PAD] ends up in 163606 instead of 163839. That seems impossible, because 163839 is strictly greater than 163607, the vocab size.
This is because the tokenizer was constructed differently: instead of adding extra tokens contiguously, moonshot added a block of 256 token ids at the end of their vocab, making its length 163840, and hen picked some ids in that block for special tokens. So we need to do the same: we look at the maximum id in added_tokens_decoder, and create a block of placeholder so the vocabulary extends to this id, and we populate that block with the real tokens. This means there will be placeholders left, as is the case with llama

Also, this PR bypasses the try/except block where we try a SentencePiece conversion when the name of the vocab file is tiktoken.model and directly runs a tiktoken conversion in that case. This avoids a nasty warning. For this reasons, I refactored the code for the conversion in their dedicated class methods, which makes the code a bit more readable IMO. This is the major part of the delta but it does not change the underlying behavior.

Example script: https://gist.github.com/remi-or/7a0447afdaf432df8e8bdfc399e6eddc

On main:

OFFICIAL IMPLEMENTATION:
Special tokens: 23 / 23 are correct.
Length of the tokenizer: 163840
Input ids of a user saying 'hi' in the chat template: [163587, 2482, 163601, 9663, 163586, 163588, 69702, 163601, 163606]
[transformers] Could not extract SentencePiece model from /home/remi_ouazan/.cache/huggingface/hub/models--moonshotai--Kimi-K2.7-Code/snapshots/74797c9c62378b951a1f6fcf5c4631024e9b8bef/tiktoken.model using sentencepiece library due to Error parsing message with type 'sentencepiece.ModelProto': Wire format was corrupt. Falling back to TikToken extractor.

RUNTIME IMPLEMENTATION:
Special tokens: 0 / 23 are correct.
Length of the tokenizer: 163607
Input ids of a user saying 'hi' in the chat template: [163585, 2482, 163591, 9663, 163584, 163586, 69702, 163591, 163603]

Input ids are different

On this branch:

OFFICIAL IMPLEMENTATION:
Special tokens: 23 / 23 are correct.
Length of the tokenizer: 163840
Input ids of a user saying 'hi' in the chat template: [163587, 2482, 163601, 9663, 163586, 163588, 69702, 163601, 163606]

RUNTIME IMPLEMENTATION:
Special tokens: 23 / 23 are correct.
Length of the tokenizer: 163840
Input ids of a user saying 'hi' in the chat template: [163587, 2482, 163601, 9663, 163586, 163588, 69702, 163601, 163606]

Input ids are the same

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

ty

[github-actions]

---

CI recap

Dashboard: View test results in Grafana
Latest run: 33491367943:1
Result: success | Jobs: 16 | Tests: 185,106 | Failures: 1 | Duration: 11h 57m