Repository navigation
Support Llama 3.2 conversion (text models) - #33778
Conversation
Co-authored-by: Omar Sanseviero <osanseviero@gmail.com>
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Initialize from a well-known template. The guidance is that the changes should be applied to 3.1 models as well.
| from transformers import AutoTokenizer | ||
| t = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B-Instruct") | ||
| additional_kwargs["chat_template"] = t.chat_template |
There was a problem hiding this comment.
Is it acceptable to load from a well-known tokenizer @ArthurZucker? This way they'd always be in sync, and having the very long template in the script is a bit noisy.
There was a problem hiding this comment.
If we use a revision, fine by me!
| } | ||
|
|
||
| CONTEXT_LENGTH_FOR_VERSION = {"3.1": 131072, "3": 8192, "2": 4096, "1": 2048} | ||
| CONTEXT_LENGTH_FOR_VERSION = {"Guard-3": 131072, "3.2": 131072, "3.1": 131072, "3": 8192, "2": 4096, "1": 2048} |
There was a problem hiding this comment.
The Llama Guard 3 model uses a different rope scale and requires a different chat template. An alternative would be to use --guard (which would imply --instruct) instead of adding a new version, but I'm not sure if code would be clearer.
|
This is now ready for review @ArthurZucker. I exported the Llama 3.2 instruct, base and guard models and verified that the configuration files were identical to what's currently in the Hub. |
ArthurZucker
left a comment
ArthurZucker
left a comment
There was a problem hiding this comment.
Thanks, LGTM, what's best is usually to try converting at least llama1 7B, and llama2 7B as well! 🤗 this way we are certain there is not regression
|
I converted Llama 2 and only saw a couple of new properties from the default configuration (for example, I found the error message for tokenizer (< 3.x) conversion to be unclear, so I added a explicit reference to requiring I didn't keep a local copy of the Llama 1 research weights, so I can't test that code path now. I re-requested access to download the weights from Meta, not sure how long it'll take. |
|
No worries, LGTM and I think we can merge! 🤗 |
* Support Llama 3.2 conversion (text models) Co-authored-by: Omar Sanseviero <osanseviero@gmail.com> * Fix rope factor * Update chat template Initialize from a well-known template. The guidance is that the changes should be applied to 3.1 models as well. * Remove import * Support Llama Guard 3 conversion * Tokenizer details * Fix eos added token in base models * Fix generation config for base models * Specify revision for known tokenizers * Style * Reuse chat templates for older models * Improve error when converting tokenizer < Llama 3 --------- Co-authored-by: Omar Sanseviero <osanseviero@gmail.com>
…ersion The Llama3Converter hardcodes clean_up_tokenization_spaces=True, which applies BERT-era string replacements (` .` → `.`, ` !` → `!`, etc.) that silently corrupt decoded text from Llama 3's BPE tokenizer. Llama 2's LlamaTokenizer and Llama 4 both use False. The True was introduced in PR huggingface#30334 and hardcoded in PR huggingface#33778 for backward compat. Fixes huggingface#35175
…nversion (#44914) fix: set clean_up_tokenization_spaces=False in Llama 3 tokenizer conversion The Llama3Converter hardcodes clean_up_tokenization_spaces=True, which applies BERT-era string replacements (` .` → `.`, ` !` → `!`, etc.) that silently corrupt decoded text from Llama 3's BPE tokenizer. Llama 2's LlamaTokenizer and Llama 4 both use False. The True was introduced in PR #30334 and hardcoded in PR #33778 for backward compat. Fixes #35175
What does this PR do?
Update the existing Llama conversion script to support the new 1B and 3B models.
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.