[hmellor]

Context

Some Mistral models are released with both a tokenizers format tokenizer (tokenizer.json) and a mistral-common format tokenizer (tekken.json).

If we examine TOKENIZER_MAPPING_NAMES in src/transformers/models/auto/tokenization_auto.py we see the following pattern repeated for all Mistral models:

transformers/src/transformers/models/auto/tokenization_auto.py

Lines 204 to 209 in 08a7ef0

	(
	"mistral",
	"MistralCommonBackend"
	if is_mistral_common_available()
	else ("TokenizersBackend" if is_tokenizers_available() else None),
	),

From these we can reasonably assume that if mistral-common is:

• not installed then AutoTokenizer should use TokenizersBackend (i.e. tokenizer.json)
• is installed then AutoTokenizer should use MistralCommonBackend (i.e. tekken.json)

Therefore, for checkpoints which have both tokenizer.json and tekken.json, the decision of which to use comes down to whether mistral-common is installed.

Next, if we look at:

transformers/src/transformers/models/auto/tokenization_auto.py

Lines 718 to 746 in 08a7ef0

	# if there is a config, we can check that the tokenizer class != than model class.
	# Use the config class if it's a specialized tokenizer, otherwise fall back to TokenizersBackend.
	if (
	tokenizer_auto_map is None
	and tokenizer_config_class is not None
	and config_model_type is not None
	and config_model_type != ""
	and TOKENIZER_MAPPING_NAMES.get(config_model_type) is not None
	and (TOKENIZER_MAPPING_NAMES.get(config_model_type).removesuffix("Fast"))
	!= (tokenizer_config_class.removesuffix("Fast"))
	):
	registered_class_name = TOKENIZER_MAPPING_NAMES.get(config_model_type).removesuffix("Fast")
	if registered_class_name not in ("TokenizersBackend", "PythonBackend", "PreTrainedTokenizerFast"):
	# If the hub class is known incorrect for this model type, use the registered class; otherwise trust the hub.
	class_name = (
	registered_class_name
	if config_model_type in MODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS
	else tokenizer_config_class
	)
	tokenizer_class = tokenizer_class_from_name(class_name)
	if tokenizer_class is not None and tokenizer_class.__name__ not in (
	"TokenizersBackend",
	"PythonBackend",
	"PreTrainedTokenizerFast",
	):
	return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
	if TokenizersBackend is not None:
	return TokenizersBackend.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)

We can see that if:

• We enter the first if block because:
  • tokenizer_config.json exists and contains a tokenizer_class field
  • TOKENIZER_MAPPING_NAMES contains an entry for this model type
• When mistral-common is:
  • not installed, we never enter the second if block because registered_class_name is always TokenizersBackend for Mistral models
  • installed, we do enter the second if block and always load from tokenizer_config_class

This last line is a bug and leads to incorrect tokenization.

Reproducer

Using the following file as a reproducer:

# repro.py
from transformers import AutoTokenizer
from transformers.utils.import_utils import is_mistral_common_available

print(f"mistral-common is installed: {is_mistral_common_available()}")
tokenizer = AutoTokenizer.from_pretrained("mistralai/Ministral-8B-Instruct-2410")
tokenizer_cls = type(tokenizer).__name__
print(f"AutoTokenizer resolved to {tokenizer_cls=}")
# Tokens for "Hello, how are you?"
token_ids = [22177, 1044, 2606, 1584, 1636, 1063]
print(tokenizer.decode(token_ids))

We see the following two outputs with and without mistral-common:

$ python repro.py 
mistral-common is installed: True
AutoTokenizer resolved to tokenizer_cls='LlamaTokenizer'
Hello,ĠhowĠareĠyou?

$ python repro.py 
mistral-common is installed: False
AutoTokenizer resolved to tokenizer_cls='TokenizersBackend'
Hello, how are you?

The case where mistral-common is installed resolves the incorrect class and produces incorrect detokenization.

Solution

The solution is to treat MistralCommonBackend more similarly to TokenizersBackend. If MistralCommonBackend is the registered class, try to use it and fall back to TokenizersBackend if there wasn't actually a tekken.json file.

This behaviour:

• Matches the intent indicated by the snippets in TOKENIZER_MAPPING_NAMES
• Never allows the incorrect classes such as LlamaTokenizer to load; which is the same behaviour as when mistral-common is not installed

Running the reproducer again with and without mistral-common we see:

$ python repro.py 
mistral-common is installed: True
AutoTokenizer resolved to tokenizer_cls='MistralCommonBackend'
Hello, how are you?

$ python repro.py 
mistral-common is installed: False
AutoTokenizer resolved to tokenizer_cls='TokenizersBackend'
Hello, how are you?

which is the expected behaviour.

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto

[github-actions]

CI Dashboard: View test results in Grafana

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[github-actions]

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46622&sha=b11320

[itazap]

Agreed with this change here: #46091

good to merge this as an isolated PR, thanks!

[ArthurZucker]

Thanks for the great description !
Let's just fix the CIs which are related pytest tests/models/lighton_ocr/test_processor_lighton_ocr.py::LightOnOcrProcessorTest::test_save_load_pretrained_additional_features seems to go through

src/transformers/tokenization_mistral_common.py:1513: in from_pretrained
    tokenizer_path = download_tokenizer_from_hf_hub(

is weird as it has qwen2 tokenizer....

[itazap]

the reproducer works with / without mistral-common in #46091

[itazap]

should be good to close as we merged this instead #46667
Thanks harry!