[subin9]

Model type "llama" spans both SentencePiece (Llama-1/2) and byte-level (Llama-3 / tiktoken) tokenizers under one Hub tokenizer_class (LlamaTokenizerFast). In v5, LlamaTokenizer.init unconditionally installs a Metaspace pre-tokenizer/decoder, which silently drops spaces for byte-level repos (see #45488), e.g. deepseek-ai/DeepSeek-R1-Distill-Llama-*.

The existing MODEL_IDS_TO_TOKENIZERS_BACKEND allowlist only covers specific checkpoints (e.g. the 8B) and misses others (the 70B is still broken). Instead, for the small set of dual-scheme model types, inspect the serialized tokenizer.json: if it declares a ByteLevel pre_tokenizer/decoder, route to TokenizersBackend (which respects tokenizer.json). SentencePiece Llama-1/2 stays on LlamaTokenizer unchanged.

What does this PR do?

model_type == "llama" covers two incompatible tokenizer schemes under a single Hub tokenizer_class
(LlamaTokenizerFast):

• Llama-1/2 — SentencePiece (Metaspace, ▁)
• Llama-3 and its derivatives (e.g. deepseek-ai/DeepSeek-R1-Distill-Llama-*) — byte-level BPE (GPT-2 / tiktoken, Ġ)

In v5, LlamaTokenizer.__init__ unconditionally installs a Metaspace pre-tokenizer/decoder, overwriting whatever
tokenizer.json declares. For byte-level repos this silently drops spaces on both encode and decode
("Hello world" → "Helloworld"), which is a silent accuracy regression — see #45488.

The current mitigation (MODEL_IDS_TO_TOKENIZERS_BACKEND, from #46091) is a per-checkpoint allowlist. It fixes
deepseek-ai/deepseek-r1-distill-llama-8b but misses others — for example DeepSeek-R1-Distill-Llama-70B is
still broken on main (it is not in the list).

Instead of enumerating checkpoints, this PR adds a small content-based rule: for the (rare) set of dual-scheme
model types ({"llama"}), inspect the serialized tokenizer.json; if it declares a ByteLevel
pre-tokenizer/decoder, route to TokenizersBackend (which respects tokenizer.json). SentencePiece Llama-1/2
(no ByteLevel in tokenizer.json) keeps using LlamaTokenizer unchanged. LlamaTokenizer.__init__ is not
modified, so there is no risk to the SentencePiece path.

This covers any byte-level Llama repo (8B, 70B, future distills) without maintaining a checkpoint allowlist.

Fixes #45488 for Llama-3-derived checkpoints.

Verification

Round-trip on "Hello world.\nI'm an AI, so I don't have consciousness." (transformers main):

repo	tokenizer	before (main)	after
DeepSeek-R1-Distill-Llama-70B (not in allowlist)	byte-level	LlamaTokenizer, roundtrip False ("Helloworld.I'manAI.")	TokenizersBackend, roundtrip True
DeepSeek-R1-Distill-Llama-8B	byte-level	TokenizersBackend, True	TokenizersBackend, True
NousResearch/Llama-2-7b-hf	SentencePiece	LlamaTokenizer, True	LlamaTokenizer, True (unchanged)

A no-network unit test for the detection helper is added in
tests/models/auto/test_tokenization_auto.py::AutoTokenizerTest::test_tokenizer_json_is_byte_level.

[Rocketknight1]

cc @ArthurZucker @itazap

[tboerstad]

This has been an issue since 5.3.0 I believe, would be great to get this fixed.
Thank you @subin9

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[subin9]

Thanks for the review @itazap! (and thanks @qflen for the redirect 🙏)

Thanks for flagging the redundant work — you're right that _tokenizer_json_is_byte_level fetches + fully parses tokenizer.json, and then convert_to_native_format parses it again right after. I dug into moving this to an existence check like _has_tekken_tokenizer_file, and here's where I landed:

On the pure tokenizer.model existence check: it's cheaper, but it's a heuristic proxy rather than an authoritative signal, so a straight swap has a couple of correctness edges:

• A byte-level Llama-3 repo that happens to ship a root tokenizer.model (community re-uploads, save_pretrained artifacts) would be misread as SentencePiece → routed to LlamaTokenizer → the exact space-dropping bug this PR fixes comes back.
• A SentencePiece Llama-1/2 fine-tune that only kept tokenizer.json (no tokenizer.model) would be misread as byte-level → behavior change for models that currently work.
• The current check is conservative (only diverts on positive byte-level detection); an absence-based check diverts much more broadly.

Note the official meta-llama/Meta-Llama-3 repos keep .model under original/, so root existence works for them — but "Llama-3 never ships tokenizer.model" isn't reliably true across the ecosystem.

What I'd propose instead — a hybrid that keeps the authoritative decision but kills the "parse on every llama load" cost:

if (
    tokenizer_auto_map is None
    and TokenizersBackend is not None
    and config_model_type in _DUAL_SCHEME_MODEL_TYPES
    and not has_file(pretrained_model_name_or_path, "tokenizer.model", **kwargs)  # SP Llama-1/2 short-circuits, no parse
    and _tokenizer_json_is_byte_level(pretrained_model_name_or_path, **kwargs)     # confirm byte-level by content
):
    return TokenizersBackend.from_pretrained(...)

So Llama-1/2 loads only do a cheap has_file HEAD and never parse, while byte-level routing stays content-authoritative. I'll match the subfolder handling from _has_tekken_tokenizer_file.

Separately, I can drop the double-parse by threading the already-parsed json (or resolved path) into convert_to_native_format if you think that's worth doing here.

Does the hybrid sound reasonable to you, or would you prefer the simpler existence check and accept the edge cases?

---

@itazap the 7 red checks look unrelated to this PR (which only touches tokenization_auto.py):

• tests/cli/* → 'HFCliTyperGroup' object has no attribute '_add_completion', the known huggingface_hub 1.22.0 breakage tracked in tests/cli fail with huggingface_hub 1.22.0: 'HFCliTyperGroup' object has no attribute '_add_completion': #47059.
• seed_oss::test_tp_forward → TP vs non-TP numeric flakiness (max diff 0.23).

Both came in with the main merge, not the diff — could you re-run once main is green? Thanks!

[subin9]

Following up on my own proposal above, @itazap — I tested the hybrid before pushing it, and it doesn't hold. The has_file short-circuit is an AND-gate, so it skips the content check in exactly the case the content check exists for.

The edge I described above as hypothetical turns out to be real: benyamini/DeepSeek-R1-Distill-Llama-8B-AWQ-w4g128 is byte-level, ships a root tokenizer.model, and declares LlamaTokenizer — quantization pipelines copy that file over from the SentencePiece checkpoint they were derived from. Round-trip on "Hello world. I'm an AI.":

	resolved class	decode
main	LlamaTokenizer	"Helloworld.I'manAI." ❌
existence check only	LlamaTokenizer	"Helloworld.I'manAI." ❌
hybrid (my proposal above)	LlamaTokenizer	"Helloworld.I'manAI." ❌
this PR as-is	TokenizersBackend	✅

I also mis-diagnosed the cost above: it's one Hub round-trip, not the parse. Warm cache on DeepSeek-R1-Distill-Llama-8B — has_file 197ms, cached_file 209ms, json.load 12–61ms. So the existence check saves ~5% and can't see the case above, while the hybrid is the slowest of the three (two round-trips). The one genuinely redundant piece is the second parse in convert_to_native_format (~60ms); happy to thread the resolved json through if you'd like that gone.

So I'd keep the routing logic as it stands. I've pushed a no-network regression test pinning the case above — it fails on main ('Helloworld' != 'Hello world') and passes here. CI is green.

[subin9]

Rebased onto main — this was 131 commits behind, so the previous green run predated both #47296 and #45936. CI is green on the rebase.

Re-measured on b6d5084fb4, round-trip of "Hello world. I'm an AI.":

checkpoint	main	this PR
deepseek-ai/DeepSeek-R1-Distill-Llama-8B	TokenizersBackend ✅	✅
deepseek-ai/DeepSeek-R1-Distill-Llama-70B	TokenizersBackend ✅	✅
neuralmagic/DeepSeek-R1-Distill-Llama-70B-FP8-dynamic	LlamaTokenizer → "Helloworld.I'manAI." ❌	✅
benyamini/DeepSeek-R1-Distill-Llama-8B-AWQ-w4g128	LlamaTokenizer ❌	✅
the 8B above, loaded from a local directory	LlamaTokenizer ❌	✅

SentencePiece llama checkpoints are unaffected: NousResearch/Llama-2-7b-hf and lmsys/vicuna-7b-v1.5 still resolve to LlamaTokenizer, codellama/CodeLlama-7b-hf to CodeLlamaTokenizer, all round-tripping correctly.

The id allowlist covers the deepseek-ai/ distills since #47296, but quantized re-uploads live under other orgs and don't match a pattern. The last row is the part I don't think an allowlist can reach: loading from a local directory makes config._name_or_path the path, so no repo pattern can ever match it.

I also reordered — the byte-level check now runs after MODEL_IDS_TO_TOKENIZERS_BACKEND, so listed checkpoints resolve on the in-memory match and never read tokenizer.json; only repos no pattern covers pay for the lookup. That should address the redundant-work concern from the earlier review.

Happy either way on the shape, @itazap — keep this as the fallback, or if you'd rather stay with the allowlist I'll add these ids there instead and close this.

[subin9]

Rebased onto main. The one red check is examples_torch, SIGKILLed on the runner mid-run
and unrelated to this; tests_tokenization is green on all 8 shards.

@ArthurZucker #47862 tightens the SentencePiece decoder on LlamaTokenizer, which is why
I am flagging this one: it is about byte-level checkpoints not landing on that class in
the first place. It touches only tokenization_auto.py, so the two do not overlap.

The row I would most value your read on is in the table above:
deepseek-ai/DeepSeek-R1-Distill-Llama-8B is in MODEL_IDS_TO_TOKENIZERS_BACKEND, but
loading those same files from a local directory resolves to LlamaTokenizer and drops the
spaces, since the list is matched against config._name_or_path.

[forest-david]

I verified this fix in a real vLLM deployment, and it resolves the garbled
tokenizer output for DeepSeek-R1-Distill-Llama-70B.

Environment

• Model: deepseek-ai/DeepSeek-R1-Distill-Llama-70B
• Model source: local filesystem checkpoint
• Transformers: 5.14.1
• Tokenizers: 0.22.2
• vLLM: 0.25.1
• Tensor parallel size: 4

Before the fix

Loading the tokenizer from the local model directory resolved to
LlamaTokenizer instead of TokenizersBackend.

A tokenizer round-trip already demonstrated the problem:

Tokenizer class:
transformers.models.llama.tokenization_llama.LlamaTokenizer

Input:
Hello world.\nHow are you?

Decoded:
Helloworld.\nHowareyou?

When serving the original local checkpoint with vLLM, the chat completion
response exposed raw ByteLevel markers such as Ġ for spaces and Ċ for
newlines:

{
  "message": {
    "role": "assistant",
    "content": "<think>ĊĊ</think>ĊĊHello!ĠHowĠcanĠIĠassistĠyouĠtoday?"
  }
}

After applying this PR

Loading the same local model directory resolved to:

transformers.tokenization_utils_tokenizers.TokenizersBackend

The tokenizer round-trip was correct:

Input:
Hello world.\nHow are you?

Decoded:
Hello world.\nHow are you?

PASS: true

I then served the original model directory with vLLM, without a separate
--tokenizer override.

The OpenAI-compatible API returned correctly decoded output:

{
  "message": {
    "role": "assistant",
    "content": "Hello! How can I assist you today?"
  }
}

Spaces and newlines were preserved, and the output no longer contained the raw
ByteLevel markers Ġ or Ċ.

This confirms that the content-based tokenizer routing in this PR fixes local
filesystem checkpoints and works end-to-end with vLLM.

Thanks for working on this fix. I would be happy to test additional local or
quantized checkpoints if useful.

[ArthurZucker]

@ArthurZucker #47862 tightens the SentencePiece decoder on LlamaTokenizer, which is why
I am flagging this one: it is about byte-level checkpoints not landing on that class in
the first place.

This is wrong. There seems to be a lack of understanding of why one checkpoints is a LlamaTokenizer (llama original sentecepiece) vs tokenizerrs Backend (llama3-so on) which could have explicit Llama3Tokenizer, but they don't

[subin9]

That taxonomy is the premise of this PR — first two paragraphs of the description.

There is no explicit Llama3Tokenizer, and TOKENIZER_MAPPING_NAMES has no
llama entry either, so for model_type: llama the class is whatever
tokenizer_config.json declares. The only thing that overrides a wrong
declaration today is MODEL_IDS_TO_TOKENIZERS_BACKEND, matched against
config._name_or_path. That is the part I am asking about. On 1162241:

>>> from transformers import AutoTokenizer
>>> def rt(p):
...     t = AutoTokenizer.from_pretrained(p)
...     return type(t).__name__, t.decode(t.encode("Hello world. I'm an AI.", add_special_tokens=False))
...
>>> rt("deepseek-ai/DeepSeek-R1-Distill-Llama-8B")
('TokenizersBackend', "Hello world. I'm an AI.")
>>> rt("./DeepSeek-R1-Distill-Llama-8B")   # the same files, downloaded
('LlamaTokenizer', "Helloworld.I'manAI.")
>>> rt("neuralmagic/DeepSeek-R1-Distill-Llama-70B-FP8-dynamic")
('LlamaTokenizer', "Helloworld.I'manAI.")

The second one is a checkpoint already on the allowlist. _name_or_path becomes
the filesystem path on load, and save_pretrained deletes the key anyway
(to_diff_dict, configuration_utils.py:1058), so no id pattern can reach it.

Routing rather than a fix inside the class, because convert_to_native_format
takes tokenizer.json verbatim only for TokenizersBackend itself or a subclass
that defines no __init__ (tokenization_utils_tokenizers.py:112-118). Anything
byte-level that lands on LlamaTokenizer gets Metaspace rebuilt over it by
design. SentencePiece is unaffected: same class, same output on every
model_type: llama SP checkpoint I checked (Llama-2 7B/13B, OpenLLaMA, TinyLlama,
Sheared-LLaMA, Orca-2, Vicuna, CodeLlama).

Can an id list cover a local directory? If it can, I'll close this.

[ArthurZucker]

I am happy to fix MODEL_IDS_TO_TOKENIZERS_BACKEND to cover this case, if that's what you are asking.

[subin9]

Thanks, that covers the rows above.

Sweeping the most-downloaded llama/deepseek repos for model_type: llama with a
byte-level tokenizer.json declaring Llama*Tokenizer: 18 found, 12 broken on
e4075b9. They fall into three unrelated name families across six orgs. Nine are
r1-distill re-uploads (unsloth, mlx-community, RedHatAI, casperhansen, huihui-ai).
The other three are deepseek-ai's own:

deepseek-ai/deepseek-llm-7b-base     -> "Helloworld.I'manAI."
deepseek-ai/deepseek-llm-7b-chat     -> "Helloworld.I'manAI."
deepseek-ai/DeepSeek-Prover-V2-7B    -> "Helloworld.I'manAI."

I'll send the list change if you want it.

[subin9]

I am happy to fix MODEL_IDS_TO_TOKENIZERS_BACKEND to cover this case

Here's the list. I swept the 50,000 most-downloaded model_type: llama repos that ship both
tokenizer_config.json and tokenizer.json, applying main's own resolution path: no auto_map, declared
LlamaTokenizer/LlamaTokenizerFast, not matched by MODEL_IDS_TO_TOKENIZERS_BACKEND, byte-level
tokenizer.json.

1,866 repos, 929 orgs, 670k combined downloads.

None of the 1,866 are matched by the list today, which is by definition since that's one of the
filters. A pattern in the same style, deepseek-ai/deepseek-*, would reach 12 of them, worth 42% of
the downloads because the deepseek-ai originals account for most of that traffic. But 954 of the
1,866 have no deepseek or r1-distill anywhere in
the name, and 897 are missed even by *deepseek*, *r1-distill*, *-ds-* and *opencodeinterpreter*
put together.

I ran the top 30 of that 897 by hand on main @ 94f09cf. All 30 give back "Helloworld.I'manAI.".
No exception either, and nothing in the logs: I re-ran two of them with
warnings.simplefilter("always") and the transformers logger at DEBUG and got zero lines.
type(tokenizer).__name__ is LlamaTokenizer for all 30, and for the SentencePiece repos I used as
controls, so you can't tell from the class name either. The 30 include
microsoft/wavecoder-ultra-6.7b, Salesforce/xLAM-1b-fc-r, aisingapore/Llama-SEA-LION-v3.5-70B-R,
AI-MO/NuminaMath-7B-TIR and stockmark/stockmark-13b. The biggest one is
LSX-UniWue/LLaMmlein_1B_prerelease at 185k downloads, a German llama trained from scratch with a
32000 vocab.

CSV and the sweep script are in the gist.
https://gist.github.com/subin9/05e9a1e8886c4a0e19525f38c8be4ea8

Tell me which ones you want in MODEL_IDS_TO_TOKENIZERS_BACKEND and I'll send that patch.

[kangzhengyang]

Up to now, the problem still hasn’t been solved.

[ArthurZucker]

Was asked again mostly for llama cpp compat, having another look

[ArthurZucker]

For transformers

TL;DR: we don't want to parse / read the content of the tokenizer.json.

In v5, AutoTokenizer / XTokenizer.from_pretrained gives you that class's tokenizer.
The class defines the normalizer, pre-tokenizer and decoder, and tokenizer.json only supplies the vocab, merges and added tokens. That holds for every model, not just Llama.
If you want exactly what's serialized, use TokenizersBackend.from_pretrained(path), that's the big refacto.

Parsing the tokenizer.json file for llama only:

• Breaks that contract for one model type. AutoTokenizer and LlamaTokenizer.from_pretrained would return different tokenizers for the same repo, and no other model behaves like that.
• Hides the real bug: wrong metadata. These checkpoints declare tokenizer_class: LlamaTokenizerFast but have byte-level BPE. Meta's own Llama-3 declares PreTrainedTokenizerFast and loads correctly.
• Is a heuristic we'd maintain forever. Which components count? Sequence pre-tokenizers? Split + ByteLevel?

THe only "fix" is to add all models that are wrong on the hub path.

For llamacpp

The tokenizer issue: get_vocab_base uses AutoTokenizer to read back what's serialized.
In v5, AutoTokenizer returns the class's pipeline; for example, LlamaTokenizer always installs Metaspace, whatever tokenizer.json says. To read the file as-is, use TokenizersBackend:

Since transformers is pinned to v4, this should help:

from transformers import AutoTokenizer, TokenizersBackend
if (self.dir_model / "tokenizer.json").is_file():
    tokenizer = PreTrainedTokenizerFast.from_pretrained(self.dir_model)
else:
    tokenizer = AutoTokenizer.from_pretrained(self.dir_model)

This fixes most issues.
Some hashes can also be re-computed.
I also advice bumping transformers there. I'll open a PR and check in details but TLDR this PR is not a fix!

[subin9]

Your call, and if the answer is still allowlist only, fine. But the bug doesn't go away with it, and two of the reasons above don't match the code.

I applied your fix at its fullest: all 1,866 affected repos from the August sweep added to MODEL_IDS_TO_TOKENIZERS_BACKEND. Main as of today, round trip of "Hello world. I'm an AI.":

	main	main + all 1,866 ids	this PR
40 random affected repos (seed 47017), by Hub id	0/40	40/40	40/40
the same 40, from a local download	0/40	0/40	40/40
deepseek-ai/DeepSeek-R1-Distill-Llama-8B / -70B, already listed, from a local download	0/2	0/2	2/2
SentencePiece controls (Llama-2, Vicuna, TinyLlama) and Llama-3	4/4	4/4	4/4

So the "only fix" leaves every local load broken, including the models already on the list, silently: LlamaTokenizer, "Helloworld.I'manAI.", no exception, no warning. _name_or_path is a path there, so no pattern will ever match it. That is how vLLM serves a checkpoint and how llama.cpp converts one.

It also doesn't keep up. Since August, 125 more affected repos (41 orgs), zero matched by the list. Ten random ones all drop spaces, including maksimbbbi/DeepSeek-R1-Distill-Llama-8B, a re-upload of a listed model.

On "we don't want to read tokenizer.json": convert_to_native_format already parses it on every load and reads the normalizer out of it. On "AutoTokenizer and LlamaTokenizer would disagree, no other model does that": the allowlist already does exactly that for every id it lists. This PR does the same thing based on the file instead of the name.

Fair enough if you want the class to own the pipeline. That's the part I'd push back on: wrong output with no error is still a bug, and this list can't reach it. If there's a mechanism you'd accept that also covers a local directory, I'll write it.

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto

[CISC]

Since transformers is pinned to v4, this should help:

It's only pinned because v5 breaks.

from transformers import AutoTokenizer, TokenizersBackend
if (self.dir_model / "tokenizer.json").is_file():
    tokenizer = PreTrainedTokenizerFast.from_pretrained(self.dir_model)
else:
    tokenizer = AutoTokenizer.from_pretrained(self.dir_model)

I don't understand how this would fix anything (mismatching import/use, so probably not what you meant)?

[subin9]

@ArthurZucker Good news: the "heuristic we'd maintain forever" made it into main via #48317, at the same spot in from_pretrained, so the llama.cpp folks should be happy.

Since #48317 matches '"ByteLevel"' anywhere in the file rather than in pre_tokenizer/decoder, the "which components count?" question from the review above is still open. Happy to rebase this if a stricter check is wanted. Leaving it open in the meantime.

[ArthurZucker]

@subin9 did you read the code changes?
If you can make your agent read it it would be nice 😉

_hub_declared_class = tokenizer_config_class or getattr(config, "tokenizer_class", None)
        if (
            tokenizer_auto_map is None
            and TokenizersBackend is not None
            and _hub_declared_class is not None
            and _hub_declared_class.removesuffix("Fast") == "LlamaTokenizer"
            and (TOKENIZER_MAPPING_NAMES.get(config_model_type) or "").removesuffix("Fast") != "MistralCommonBackend"
            and TokenizersBackend._tokenizer_json_has_byte_level(pretrained_model_name_or_path, **kwargs)

this means:

1. there is not custom code
2. there is a hub class
3. the hub class is Llama
4. the mapping in transformers does not include Mistral

So no #48317 does NOT matche '"ByteLevel"' anywhere in the file.
@itazap found a good compromise, and I was wrong.

Now, if you don't want to get banned, please refrain from spamming threads with AI, its tedious and really not fun for us to read 50 lines of bloat.

Please make the effort of understanding what you are working on, and minimizing your answers. If there is indeed an issue, and it matters to you, we are more than obliged to help. But it needs to go both ways.

[subin9]

@ArthurZucker If you can make your agent read the function, it would be nice 😉The four points you listed only decide which repos reach the check. The check itself:

transformers/src/transformers/tokenization_utils_tokenizers.py

Line 1425 in 4936867

return '"ByteLevel"' in reader.read()

return '"ByteLevel"' in reader.read()

A substring match on the whole file, not pre_tokenizer/decoder as its docstring says. A Llama-2 tokenizer.json with ByteLevel only in post_processor passes the gate and loads as TokenizersBackend.

[ArthurZucker]

😃