[tonywu71]

What does this PR do?

NeoMME is a family of bidirectional multilingual encoders for text and images, including natural images and documents. Text tokens and raw image patches use one shared Transformer encoder, without a separately pretrained vision encoder. The pretrained backbone returns contextual token representations for task-specific fine-tuning.

NeoMME-Retriever is a downstream model fine-tuned for text and document retrieval. It returns token embeddings for MaxSim scoring and mean-pooled embeddings for cosine similarity.

Before submitting

• Did you read the contributor guideline and the
  Pull Request checks?
• Discussed on Slack with @merveenoyan, @tomaarsen, and @zucchini-nlp.
• Did you make sure to update the documentation with your changes according to the guidelines?
• Did you write any new necessary tests?

Who can review?

@tomaarsen and @zucchini-nlp 🤗

Technical details

Non-trivial decisions were discussed on Slack. The main decisions are:

1. Keep NeoMMEForRetrieval so one forward pass can return dense and multi-vector embeddings. It is intentionally absent from MODEL_FOR_RETRIEVAL_MAPPING_NAMES, so AutoModel and Sentence Transformers v6 load NeoMMEModel (you need to load the backbone in ST)
2. NeoMMEProcessor remains generic. Direct processor calls do not require a retrieval task or chat template.
3. Retrieval checkpoints ship a Jinja template that requires an explicit task="query" or task="document". ST forwards this automatically because the template declares task.
4. The template adds retrieval markers, fixed query expansion tokens, and validates retrieval content. Python handles work that depends on token IDs or image geometry, such as truncation checks, image token expansion, and two-axis positions.
5. Retrieval scoring is not part of the processor. Use Sentence Transformers for MeanMaxSim or cosine scoring.
6. The image processor uses torchvision only, preserves native resolution by default, and has no hardcoded processor _defaults.
7. Following @zucchini-nlp's suggestion, this PR extends the shared SizeDict with min_pixels and max_pixels. These fields describe pixel-area limits without overloading the existing edge-length fields (not suited for areas!), which is why the PR modifies shared files outside the NeoMME model folder.
8. Transformers provides one joint checkpoint, Hcompany/NeoMME-{model-size}-Retriever. Sentence Transformers provides separate dense and multi-vector wrappers that reuse the backbone weights and load NeoMMEModel.

Manual tests

• I checked on our internal repository under tolerance 1e-3 that the outputs of NeoMME and NeoMME-Retriever match what our original implementation.
• To validate that NeoMME-Retriever will properly work with ST-v6, I implemented a custom test on this private repo. @tomaarsen @zucchini-nlp I added you as contributors so you can check the passed tests!

🤖 mlinter review state

• hash: fc055c87
• commit: 9962b95b
• review: Add NeoMME and NeoMME-Retriever #47992 (review)

[zucchini-nlp]

Very interesting model, and good use of modular in model classes. There are a few more classes we can modularize imo, left comments where needed

The major concern atm is the processing class, I left a suggestion so lmk if that works for you/ST. I am not really following the latest ST release, would be great to clean-up redundancy where possible

[zucchini-nlp]

Oke, I am not sure when we're merging (didn't get a reply from Arthur) it but overall nothing too important from comments, and all can be fixed afterwards without even changing the state dict

Lmk in slack or here if you absolutely want this merged tonight, and I'll merge. Otherwise we can wait until Monday

[zucchini-nlp]

what Arthur mean is prob to create an output projection layer and tie same way as you tied the lm head

_tied_weights_keys = {"lm_head.weight": "model.embed_tokens.word_embeddings.weight", "unembedding_projection.weight": "model.embed_tokens.embedding_projection.weight"}

[ArthurZucker]

small nits, let's go!

[ArthurZucker]

the heads are dangling if outputs are not needed -> maybe another class to extract embeddings only?

Fine for now, it's clean at least!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, neomme

[github-actions]

---

CI recap

Dashboard: View test results in Grafana
Latest run: 33375260729:2
Result: success | Jobs: 16 | Tests: 184,769 | Failures: 1 | Duration: 15h 9m