[OrangerXu]

What does this PR do?

Fixes #49578.

Since 5.18, a checkpoint whose config.json has a Qwen3.5-family model_type loads as Qwen3_5Tokenizer, a class with a custom __init__ that rebuilds the backend tokenizer from vocab/merges. convert_to_native_format forwards vocab, merges, post_processor, padding and truncation from tokenizer.json to that rebuild — but not the normalizer, pre-tokenizer or decoder. The class's own defaults then silently replace whatever the checkpoint serialized.

For tencent/WeMM-Embedding-2B (model_type: qwen3_5), the file's normalizer is Sequence[Replace(Regex(\ \$\), ""), NFC] — it strips one trailing newline. The class default is plain NFC, so the Replace is lost and every input keeps an extra newline token:

tok = AutoTokenizer.from_pretrained("tencent/WeMM-Embedding-2B")
tok("<|im_start|>user
hello<|im_end|>
", add_special_tokens=False).input_ids
# 5.17: [248045, 846, 198, 14556, 248046]
# main: [248045, 846, 198, 14556, 248046, 198]  <- extra newline token

This is an embedding model that reads the last position, so the extra token changes every embedding. The same loss hits any other custom-__init__ class whose checkpoint customized one of the three stages.

The fix mirrors the existing post_processor mechanism exactly: convert_to_native_format extracts the three stages as Rust objects from the file, and TokenizersBackend.__init__ applies them after the subclass has built the backend, so a stage serialized in tokenizer.json wins over the class default. A null stage in the file leaves the class default untouched, keeping canonical checkpoints (whose files match the class defaults) byte-identical.

Reproduction (verified locally)

Downloaded WeMM's tokenizer.json + tokenizer_config.json, added a minimal config.json with model_type: qwen3_5, then AutoTokenizer.from_pretrained(local_dir):

	class	backend normalizer	ids
main	Qwen3_5Tokenizer	NFC()	trailing 198 (bug)
this PR	Qwen3_5Tokenizer	Sequence[Replace, NFC]	matches 5.17

Middle newlines are unaffected (tok("a b") keeps the newline token); loading the same files via the generic TokenizersBackend path was and stays correct.

Tests

• New test_tokenizer_json_normalizer_survives_the_class_rebuild in tests/models/qwen3_5/test_tokenization_qwen3_5.py: builds a tiny synthetic tokenizer.json (offline, byte-level BPE) whose normalizer strips one trailing newline, loads it through Qwen3_5Tokenizer.from_pretrained, and asserts both the structural preservation (file normalizer wins over the class default) and the behavior (trailing newline gone, middle newline kept). Red without the fix, green with it.
• Full tests/models/qwen3_5 suite: 48 passed.
• Regression sweep with the fix: bert / gpt2 / t5 / qwen2 tokenization suites 118 passed with no new failures; llama has 8 pre-existing environmental failures (missing tiktoken, gated hub repos) — identical failing test IDs on pristine main.

[github-actions]

Thank you for your contribution 🤗!

CI Security Gate — waiting for maintainer approval

CI is approved automatically only for authors who already have a contribution merged in this repository. A maintainer will review this PR and approve CI to run on it.

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: qwen3_5

[tomaarsen]

Hello!

Have you figured out which commit caused this change in behaviour? I'm quite curious if we can just fix at the root instead of patching the symptoms. It could've also been the upgrade to a newer tokenizers that did it. Either way, it should be simple to narrow down by continuously checking out different commits locally until the behaviour switches.

cc @AdnanElAssadi (thanks for finding this regression, I wonder if it goes beyond just the WeMM model)

• Tom Aarsen

[OrangerXu]

Hi Tom — I bisected it before writing the PR; the culprit is in our conversion layer, not tokenizers.

Timeline (from the repo history):

• 73a13f86f6 (2025-12-09, Refactor-tokenization-more #42563) introduced the elif branch in convert_to_native_format: for any class with a custom __init__, tokenizer.json is reduced to vocab/merges and the class rebuilds the backend from its own defaults. At that point every pipeline stage in the file was silently dropped.
• fc91372258 (2026-02-09, Adding Support for Qwen3.5 #43830) added Qwen3_5Tokenizer, so model_type: qwen3_5 checkpoints began routing through a custom __init__. Canonical Qwen3.5 checkpoints were unaffected — their files match the class defaults exactly — so the gap stayed latent.
• fd6bc380c8 (2026-03-04, [vllm + v5 fix] handle TokenizersBackend fallback properly for v5 #44255) noticed one of the lost stages and fixed exactly it: it started forwarding post_processor (plus padding/truncation) from the file. That is the precedent this PR follows for the remaining three stages (normalizer, pre_tokenizer, decoder), which [vllm + v5 fix] handle TokenizersBackend fallback properly for v5 #44255 left dropped.
• WeMM is a derivative checkpoint whose normalizer diverges from the class default, which is what turned the latent gap into a behavior change between 5.17 and 5.18 for that model family.

Why I don't think there is a deeper root to fix at: the elif branch is deliberate — a custom __init__ defines the class's canonical pipeline, and the file is treated as the vocab source (the wholesale from_file path is reserved for classes without a custom __init__, or trust_remote_code). Given that design, "the class defaults silently replace the serialized stages" is the bug, and #44255 already established the fix pattern at this exact layer. The alternative — always loading the full tokenizer.json even for custom classes — would bypass the __init__ logic of ~53 subclasses and change behavior far beyond this regression.

And to rule out the tokenizers upgrade: the normalizer is dropped in convert_to_native_format before the Rust side ever sees the file. I reproduced it with a fixed tokenizers version against WeMM's actual files loaded locally: the same tokenizer.json keeps Sequence[Replace, NFC] through the generic TokenizersBackend route and yields plain NFC through the Qwen3_5Tokenizer route.