[harshaljanjani]

What does the PR do?

Fixes weight tying between lm_head and embed_tokens by exposing tie_word_embeddings at the top-level config, following the canonical repo pattern. Upon inspecting both presets from the AyaVision collection, both presets have text_config.tie_word_embeddings set to True, but this was not being surfaced on the top-level config.

from transformers import AutoConfig

models = [
    "CohereForAI/aya-vision-32b",
    "CohereLabs/aya-vision-8b",
]
for model_id in models:
    config = AutoConfig.from_pretrained(model_id)
    print(f"{model_id} tie_word_embeddings:", config.text_config.tie_word_embeddings)

Fixes #43454.

Output:

CohereForAI/aya-vision-32b tie_word_embeddings: True
CohereLabs/aya-vision-8b tie_word_embeddings: True

Accordingly, I’ve added tie_word_embeddings to AyaVisionConfig, with its default value set to True. As a consequence, the CI failures are also fixed. Run the following command to check the previously failing tests, which should now pass after the weight tying.

!RUN_SLOW=1 pytest -v tests/models/aya_vision/test_modeling_aya_vision.py::AyaVisionIntegrationTest::test_small_model_integration_batched_generate tests/models/aya_vision/test_modeling_aya_vision.py::AyaVisionIntegrationTest::test_small_model_integration_batched_generate_multi_image tests/models/aya_vision/test_modeling_aya_vision.py::AyaVisionIntegrationTest::test_small_model_integration_forward tests/models/aya_vision/test_modeling_aya_vision.py::AyaVisionIntegrationTest::test_small_model_integration_generate_chat_template tests/models/aya_vision/test_modeling_aya_vision.py::AyaVisionIntegrationTest::test_small_model_integration_generate_text_only -s

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that’s the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link
  to it if that’s the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: aya_vision

[harshaljanjani]

Just a slight comment on this fix -- I ran this in the venv listed in the issue (NVIDIA L4).

!RUN_SLOW=1 pytest -v tests/models/aya_vision/test_modeling_aya_vision.py::AyaVisionIntegrationTest::test_small_model_integration_batched_generate tests/models/aya_vision/test_modeling_aya_vision.py::AyaVisionIntegrationTest::test_small_model_integration_batched_generate_multi_image tests/models/aya_vision/test_modeling_aya_vision.py::AyaVisionIntegrationTest::test_small_model_integration_forward tests/models/aya_vision/test_modeling_aya_vision.py::AyaVisionIntegrationTest::test_small_model_integration_generate_chat_template tests/models/aya_vision/test_modeling_aya_vision.py::AyaVisionIntegrationTest::test_small_model_integration_generate_text_only -s

And it does produce perfectly comprehensible outputs after the fix, but there are negligible differences between the output shown in my environment and the expected output; I suspect this is due to minor hardware precision issues, so I haven’t added those changes to the PR yet. If those test failures show up when run in this environment as well, I’ll add them, but they no longer produce gibberish so yeah :)

Output from my environment (env specifications listed in the issue):

tests/models/aya_vision/test_modeling_aya_vision.py::AyaVisionIntegrationTest::test_small_model_integration_batched_generate_multi_image FAILED
tests/models/aya_vision/test_modeling_aya_vision.py::AyaVisionIntegrationTest::test_small_model_integration_forward PASSED
tests/models/aya_vision/test_modeling_aya_vision.py::AyaVisionIntegrationTest::test_small_model_integration_generate_chat_template PASSED
tests/models/aya_vision/test_modeling_aya_vision.py::AyaVisionIntegrationTest::test_small_model_integration_generate_text_only FAILED
=================================== FAILURES ===================================
_ AyaVisionIntegrationTest.test_small_model_integration_batched_generate_multi_image _
       AssertionError: 'Wooden path to water,\nMountains echo in stillness,\nPeaceful forest scene.' != 'Wooden path to water,\nMountains echo in stillness,\nPeaceful forest lake.'
         Wooden path to water,
         Mountains echo in stillness,
       - Peaceful forest scene.?                 ^^^^
       + Peaceful forest lake.?                 ^^^
        : Decoded output: Wooden path to water,
       Mountains echo in stillness,
       Peaceful forest scene.
       Expected output: Wooden path to water,
       Mountains echo in stillness,
       Peaceful forest lake.
tests/models/aya_vision/test_modeling_aya_vision.py:481: AssertionError
___ AyaVisionIntegrationTest.test_small_model_integration_generate_text_only ___
AssertionError: "Whis[14 chars]eeze,\nLeaves dance under moonlit sky,\nNature's quiet song." != "Whis[14 chars]eeze,\nLeaves dance under moonlit skies,\nNature's quiet song."
         Whispers on the breeze,
       - Leaves dance under moonlit sky,
       ?                              ^
       + Leaves dance under moonlit skies,
       ?                              ^^^
         Nature's quiet song.
tests/models/aya_vision/test_modeling_aya_vision.py:318: AssertionError
=========================== short test summary info ============================
FAILED tests/models/aya_vision/test_modeling_aya_vision.py::AyaVisionIntegrationTest::test_small_model_integration_batched_generate_multi_image - AssertionError: 'Wooden path to water,\nMountains echo in stillness,\nPeace...
FAILED tests/models/aya_vision/test_modeling_aya_vision.py::AyaVisionIntegrationTest::test_small_model_integration_generate_text_only - AssertionError: "Whis[14 chars]eeze,\nLeaves dance under moonlit sky,\nNatu...
================== 2 failed, 3 passed, 39 warnings in 58.59s ===================

[ArthurZucker]

This LGTM, errors are expected from different hardwares

[harshaljanjani]

@ArthurZucker Thanks for the review! I was wondering if you might also be able to take a look at this PR whenever you have time. It’s a long standing feature request, and it’s been quite some time since I raised a PR for it since September (with a few follow-up pings); I’d really appreciate it, thank you 🙏