Add support for Nemotron Omni by meatybobby · Pull Request #46509 · huggingface/transformers

meatybobby · 2026-06-08T20:33:15Z

What does this PR do?

Add NVIDIA Nemotron Omni support (This PR depends on radio PR #46425 )

Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
we ask that new users do not submit pure code agent PRs at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read CONTRIBUTING.md.

I confirm that this is not a pure code agent PR.

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

… related token fields explicitly in NemotronH_Nano_Omni_Reasoning_V3_Config

HuggingFaceDocBuilderDev · 2026-06-15T17:37:53Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

meatybobby · 2026-07-13T17:03:10Z

@ArthurZucker This PR is ready to be merged. Could we start the reviewing process?

ArthurZucker · 2026-07-23T06:20:25Z

Yes sorry went off this week

ArthurZucker · 2026-07-23T07:02:31Z

+    pass
+
+
+class EfficientVideoSampling:


this needs the model prefixing

also if its just for a func not sure we need a clas?

keep only func here

ArthurZucker · 2026-07-23T07:07:11Z

+        parakeet_config = AutoConfig.for_model("parakeet_encoder", **config_dict)
+        self.config = parakeet_config
+        self.encoder = AutoModel.from_config(parakeet_config)


the config should just have a config saved. and at this point you create a wrapper around something that should just be used directly this is too many layers

Directly use ParakeetEncoderConfig now

ArthurZucker · 2026-07-23T07:10:21Z

+        return x
+
+    def extract_feature(self, pixel_values):
+        if isinstance(pixel_values, (list, tuple)):


the whole pixel shuffle exists in other models, it should be isolated as this shows it does not belong here. Isolated and done with the mlp1

Isolate in NemotronH_Omni_Reasoning_V3VisionProjector now

ArthurZucker · 2026-07-23T07:11:07Z

+        orig_projection = embeddings.patch_projection
+        embeddings.patch_projection = embeddings.video_patch_projection
+        try:
+            vit_embeds = self.vision_model(x).features


that is not super idea no? why do we have to do try catch it should be predictable

video_patch_projection moved to inside Radio instead of replace patch_projection now.

ArthurZucker · 2026-07-23T07:16:06Z

+            if image_vit_embeds is not None:
+                image_mask = input_ids_copy == self.img_context_token_id
+                assert image_mask.sum() != 0, "No image tokens found in input_ids"
+                inputs_embeds[image_mask] = image_vit_embeds.reshape(-1, C).to(
+                    inputs_embeds.device, inputs_embeds.dtype
+                )
+
+            if video_vit_embeds is not None:
+                if B > 1:
+                    raise NotImplementedError("Video is not supported for batch size > 1")
+                video_mask = input_ids_copy == self.img_context_token_id
+                assert video_mask.sum() != 0, "No video tokens found in input_ids"
+                inputs_embeds[video_mask] = video_vit_embeds.reshape(-1, C).to(
+                    inputs_embeds.device, inputs_embeds.dtype
+                )
+
+            if sound_embeds is not None and self.sound_context_token_id is not None:
+                sound_mask = input_ids_copy == self.sound_context_token_id
+                assert sound_mask.sum() != 0, "No sound tokens found in input_ids"
+                if sound_embeds.dim() == 3 and sound_embeds.shape[0] > 1 and sound_attention_mask is not None:
+                    natural_input_lengths = sound_attention_mask.sum(-1) + 1
+                    output_lengths = self.sound_encoder.encoder._get_subsampling_output_length(natural_input_lengths)
+                    flat = torch.cat(
+                        [sound_embeds[i, : int(n)] for i, n in enumerate(output_lengths.tolist())],
+                        dim=0,
+                    )
+                else:
+                    flat = sound_embeds.reshape(-1, C)
+                assert sound_mask.sum().item() == flat.shape[0], (
+                    f"sound token count ({sound_mask.sum().item()}) != encoder output count ({flat.shape[0]})"
+                )
+                inputs_embeds[sound_mask] = flat.to(inputs_embeds.device, inputs_embeds.dtype)


you should take inspiration from other models on how we usally do this specific part! an example can be found in

Follow Qwen3OmniMoe for generate path now

ArthurZucker · 2026-07-23T07:17:07Z

+    def extract_sound_feature(
+        self,
+        input_features: torch.Tensor,
+        attention_mask: torch.Tensor | None = None,
+    ) -> torch.Tensor:
+        if self.sound_encoder is None:
+            raise RuntimeError("Sound encoder not initialized. Check if sound_config is provided.")
+        sound_embeds = self.sound_encoder(input_features, attention_mask)
+        sound_embeds = sound_embeds.to(dtype=torch.bfloat16)
+        sound_embeds = self.sound_projection(sound_embeds)
+        return sound_embeds
+


why not isolate this as audioProjector and have a single forward for it?

Isolate as NemotronH_Omni_Reasoning_V3SoundProjector now

ArthurZucker

dropping the first review! as always we need to standardize cf the other omni models!

ArthurZucker · 2026-07-23T07:21:40Z

+    from transformers import NemotronH_Omni_Reasoning_V3
+
+
+class NemotronHOmniModelTester:


you can probably inherit from causal LM tester?

see other omni models and which class they inherit from!

Inherit GenerationTesterMixin now

meatybobby · 2026-08-26T20:41:18Z

We need to have this a if else in the init in that case! We try to avoid this, but here it seems the only choice unless you inherit in the modular from it to use directly!

Added if-else in init now.

meatybobby · 2026-09-03T00:15:52Z

@ArthurZucker The PR is ready. Could we merge this PR if there is no other concern?

vasqu · 2026-09-08T15:05:31Z

cc @VI-Arthur

vasqu

Sorry for the delays!

vasqu · 2026-09-09T13:02:04Z

+# NemotronH Omni
+
+NemotronH Omni is a multimodal reasoning model from NVIDIA that pairs the [NemotronH](./nemotron_h) hybrid
+Mamba-Transformer language model with a [RADIO](./radio) vision encoder and an optional Parakeet-based sound encoder.


Suggested change

Mamba-Transformer language model with a [RADIO](./radio) vision encoder and an optional Parakeet-based sound encoder.

Mamba-Transformer language model with a [RADIO](./radio) vision encoder and an Parakeet-based sound encoder.

nit but it sounds a bit off with the optional. I'd assume for omni == always all modalities no?

We will reuse model class for the model without audio encoder

meatybobby · 2026-09-23T19:40:40Z

@vasqu The comments are resolved. Could you review them again? Thanks

ArthurZucker

Let's go! Thanks for you efforts and bearing with us.
Omni models are very hard in general, the hardest to standardize. 🤗

github-actions · 2026-09-24T01:28:10Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, nemotron_h, nemotron_h_omni, radio

ArthurZucker · 2026-09-24T01:36:27Z

run-slow: auto, nemotron_h, nemotron_h_omni, radio

github-actions · 2026-09-24T01:37:51Z

AMD CI

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs on AMD:

models: ["models/auto", "models/nemotron_h", "models/nemotron_h_omni", "models/radio"]

github-actions · 2026-09-24T01:37:53Z

Nvidia CI

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs on Nvidia:

models: ["models/auto", "models/nemotron_h", "models/nemotron_h_omni", "models/radio"]
quantizations: []

github-actions · 2026-09-24T01:43:02Z

CI recap

Dashboard: View test results in Grafana
Latest run: 35942958744:2
Result: failure | Jobs: 16 | Tests: 168,253 | Failures: 1 | Duration: 14h 57m

github-actions · 2026-09-24T01:59:33Z

CI Results (AMD)

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	6bd29722	workflow commit (merge commit)
PR	9b9e05c4	branch commit (from PR)
main	f3247073	base commit (on `main`)

Model CI Report

❌ 10 new failed tests from this PR 😭

nemotron_h_omni:
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronHOmniVision2TextModelTest::test_eager_matches_sdpa_generate (✅ ⟹ ❌)
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronHOmniVision2TextModelTest::test_sdpa_can_compile_dynamic (✅ ⟹ ❌)
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronHOmniVision2TextModelTest::test_sdpa_can_dispatch_on_flash (✅ ⟹ ❌)
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronHOmniAudio2TextModelTest::test_eager_matches_fa2_generate (✅ ⟹ ❌)
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronHOmniAudio2TextModelTest::test_eager_matches_sdpa_generate (✅ ⟹ ❌)
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronHOmniAudio2TextModelTest::test_flash_attn_2_from_config (✅ ⟹ ❌)
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronHOmniAudio2TextModelTest::test_flash_attn_2_inference_equivalence (✅ ⟹ ❌)
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronHOmniAudio2TextModelTest::test_flash_attn_2_inference_equivalence_right_padding (✅ ⟹ ❌)
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronHOmniAudio2TextModelTest::test_sdpa_can_compile_dynamic (✅ ⟹ ❌)
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronH_Omni_Reasoning_V3IntegrationTest::test_video_generation (✅ ⟹ ❌)

github-actions · 2026-09-24T02:01:30Z

CI Results (Nvidia)

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	6bd29722	workflow commit (merge commit)
PR	9b9e05c4	branch commit (from PR)
main	f3247073	base commit (on `main`)

Model CI Report

❌ 21 new failed tests from this PR 😭

nemotron_h_omni:
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronHOmniVision2TextModelTest::test_eager_matches_sdpa_generate (✅ ⟹ ❌)
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronHOmniVision2TextModelTest::test_onnx_export_dynamic (✅ ⟹ ❌)
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronHOmniVision2TextModelTest::test_onnx_export_generate_dynamic (✅ ⟹ ❌)
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronHOmniVision2TextModelTest::test_onnx_export_generate_static (✅ ⟹ ❌)
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronHOmniVision2TextModelTest::test_onnx_export_static (✅ ⟹ ❌)
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronHOmniVision2TextModelTest::test_sdpa_can_compile_dynamic (✅ ⟹ ❌)
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronHOmniVision2TextModelTest::test_sdpa_can_dispatch_on_flash (✅ ⟹ ❌)
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronHOmniVision2TextModelTest::test_torch_export_dynamic (✅ ⟹ ❌)
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronHOmniVision2TextModelTest::test_torch_export_generate_dynamic (✅ ⟹ ❌)
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronHOmniVision2TextModelTest::test_torch_export_generate_static (✅ ⟹ ❌)
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronHOmniVision2TextModelTest::test_torch_export_static (✅ ⟹ ❌)
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronHOmniAudio2TextModelTest::test_eager_matches_sdpa_generate (✅ ⟹ ❌)
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronHOmniAudio2TextModelTest::test_onnx_export_dynamic (✅ ⟹ ❌)
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronHOmniAudio2TextModelTest::test_onnx_export_generate_dynamic (✅ ⟹ ❌)
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronHOmniAudio2TextModelTest::test_onnx_export_generate_static (✅ ⟹ ❌)
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronHOmniAudio2TextModelTest::test_onnx_export_static (✅ ⟹ ❌)
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronHOmniAudio2TextModelTest::test_sdpa_can_compile_dynamic (✅ ⟹ ❌)
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronHOmniAudio2TextModelTest::test_torch_export_dynamic (✅ ⟹ ❌)
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronHOmniAudio2TextModelTest::test_torch_export_generate_dynamic (✅ ⟹ ❌)
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronHOmniAudio2TextModelTest::test_torch_export_generate_static (✅ ⟹ ❌)
tests/models/nemotron_h_omni/test_modeling_nemotron_h_omni.py::NemotronHOmniAudio2TextModelTest::test_torch_export_static (✅ ⟹ ❌)

ArthurZucker · 2026-09-24T02:25:20Z

I’ll check the failures and fixe but it’s ready to merge otherwise !

ArthurZucker · 2026-09-24T06:00:47Z

OK!
The failing tests are somewhat unrelated. Parakeet is at fault here, I am opening this #49070 for the fix and merging

meatybobby added 5 commits June 5, 2026 21:14

Add Nemotron Omni model

ade586c

Declare patch_size, img_context_token_id, video_context_token_id, and…

38f50bf

… related token fields explicitly in NemotronH_Nano_Omni_Reasoning_V3_Config

Use language model from models/nemotron_h

fc1d02e

Remove processing script

710af1a

Move audio model into same model file

a2950d0

Rocketknight1 closed this Jun 9, 2026

Rocketknight1 added the Code agent slop label Jun 9, 2026

meatybobby added 2 commits June 9, 2026 21:32

Change RMSNorm to fp32

bb2cfb4

Fix rms import

6a73a81

ArthurZucker reopened this Jun 15, 2026

Rocketknight1 removed the Code agent slop label Jun 16, 2026

meatybobby added 10 commits July 6, 2026 18:08

Merge branch 'main' into bobchen/nemotron_omni

da5b3c9

Update Radio config and model

3c9c52c

Add Nemotron Omni in AutoConfig

6359783

Remove Nano in class

f0d36e6

Add doc and tests

deb53a2

Fix config check

086dc3d

Fix for consistency pipeline

73d6f22

Change config init instead of add the allow list

25157b7

Merge branch 'main' into bobchen/nemotron_omni

87e173a

Fix consistency

1165137

ArthurZucker reviewed Jul 23, 2026

View reviewed changes

ArthurZucker added the New model label Jul 23, 2026

meatybobby added 3 commits July 24, 2026 01:00

Isolate pixel shuffle

517e06f

Add prefix for some module

fc8299e

Update SoundConfig and SoundEncoder SoundProjector

92db92c

conditional video patch in radio

561e92a

meatybobby added 3 commits September 1, 2026 17:51

Merge branch 'main' into bobchen/nemotron_omni

e3be76e

Route NemotronH-Omni sub-configs through CONFIG_MAPPING/AutoConfig

263aef3

Keep parameter for campability

bb053b0

vasqu reviewed Sep 9, 2026

View reviewed changes

meatybobby added 7 commits September 22, 2026 19:54

Update config file

5aae2ca

Update processor

4071fd2

Merge branch 'main' into bobchen/nemotron_omni

fe3a185

Update nemotron_h_omni modeling file

a24a815

Update radio

63de051

Update tests

f4a909d

Add nemotron_h changes

f0965f3

last nits to merge!

9b9e05c

ArthurZucker approved these changes Sep 24, 2026

View reviewed changes

ArthurZucker merged commit 4b28d51 into huggingface:main Sep 24, 2026
114 of 118 checks passed

ydshieh mentioned this pull request Sep 25, 2026

[NemotronH-Omni] Fix device mismatch in test tensor creation #49102

Merged

stevhliu mentioned this pull request Oct 23, 2026

[docs] Translation workflow #49889

Closed

		from transformers import NemotronH_Omni_Reasoning_V3


		class NemotronHOmniModelTester:

	Mamba-Transformer language model with a [RADIO](./radio) vision encoder and an optional Parakeet-based sound encoder.
	Mamba-Transformer language model with a [RADIO](./radio) vision encoder and an Parakeet-based sound encoder.

Conversation

meatybobby commented Jun 8, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Code Agent Policy

Before submitting

Who can review?

Uh oh!

HuggingFaceDocBuilderDev commented Jun 15, 2026

Uh oh!

meatybobby commented Jul 13, 2026

Uh oh!

ArthurZucker commented Jul 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ArthurZucker left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

meatybobby commented Aug 26, 2026

Uh oh!

meatybobby commented Sep 3, 2026

Uh oh!

vasqu commented Sep 8, 2026

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

meatybobby commented Sep 23, 2026

Uh oh!

ArthurZucker left a comment

Choose a reason for hiding this comment

Uh oh!

meatybobby commented Jun 8, 2026 •

edited by github-actions Bot

Loading