:rotating_light: Vision (2d/3d) rotary embeddings by zucchini-nlp · Pull Request #48105 · huggingface/transformers

zucchini-nlp · 2026-08-19T10:22:23Z

We need a standard way to compute rope freq in vision models which is usually only 2D or 3D. Atm we either interleave the grids inside rope module's forward or inside each attention layer. I think doing one interleave/permute or whatever needed in a rope module is easier and we keep the rest of modeling similar to LLMs, plus it is better in e2e perf (see table below)

I am trying to enforce an extra self.recomposition_2d/3d on all vision RoPE, this method has to be called on freqs before returning them thus permuting grids in the way model was trained. I ran a small bench on Qwen2-VL 2B and 7B version. Asked AI to summarize the numbers, and overall we get better e2e perf when we move the heavy work under ModelVisionRotaryEmbedding. The trend is similar across model and batch sizes, so there is clearly a small win

Metric	Main branch	PR Branch	Difference	Delta (%)
Only RoPE module latency (median)	216.98 µs	290.48 µs	+73.51 µs	+33.88% (Slower)
Only attn layer latency (median)	2988.26 µs	2963.15 µs	-25.11 µs	-0.84% (Slightly Faster)
Prefill Latency (median)	59.48 ms	57.38 ms	-2.09 ms	-3.52% (Faster)
Decode Latency per Call	1976.45 ms	1771.61 ms	-204.84 ms	-10.36% (Faster)
Decode Latency per Token	30.88 ms	27.68 ms	-3.20 ms	-10.36% (Faster)
Prefill Peak Memory	4317.61 MB	4317.61 MB	+0.00 MB	0.00% (Unchanged)
Decode Peak Memory	4274.39 MB	4273.74 MB	-0.66 MB	-0.02% (Negligible)

(btw, gemma4 qk should have been permuted, seems like they gave us an internal research code that simply calls text-rope twice and concats. Under the hood it is 2D rope as in any other ViT 😢 )

cc @IlyasMoutawwakil , should fix #46443

🤖 mlinter review state

hash: 45ab6ae8
commit: 3ade9fef
review: 🚨 Vision (2d/3d) rotary embeddings #48105 (review)

…models

github-actions · 2026-09-03T15:45:28Z

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	3ecab95a	workflow commit (merge commit)
PR	0f94794b	branch commit (from PR)
main	cfa309cf	base commit (on `main`)

✅ No failing test specific to this PR 🎉 👏 !

vasqu

Smaller comments overall, should be good to merge after that 🫡 huge work and dont forget to share internally as well, this is significant and avoids all the headache around weird rope applications

github-actions · 2026-09-04T09:22:42Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: cohere_compass, cosmos3_edge, cosmos3_omni, dinov3_vit, edgetam_video, eomt_dinov3, ernie4_5_vl_moe, exaone4_5, gemma4, glm46v, glm4v, glm4v_moe, glm5_next, glm_image, glm_ocr, idefics

github-actions · 2026-09-04T09:33:25Z

CI recap

Dashboard: View test results in Grafana
Latest run: 33857108983:1
Result: success | Jobs: 2 | Tests: 9 | Failures: 2 | Duration: 3m 55s

…er HPU Pixtral lazily Root cause: transformers 5.17.0 removed PixtralRotaryEmbedding and position_ids_in_meshgrid, so the eager Pixtral import in register_model() aborted plugin registration for every architecture. Upstream: huggingface/transformers#48105 Fix: drop the redundant eager import and keep only the lazy "module:class" registration. Signed-off-by: Paweł Olejniczak <pawelx.olejniczak@intel.com>

…ansformers below 5.17 Root cause: transformers 5.17.0 removed position_ids_in_meshgrid and renamed PixtralRotaryEmbedding to PixtralVisionRotaryEmbedding, both imported at module level by upstream vLLM's vllm/model_executor/models/pixtral.py, so PixtralForConditionalGeneration can no longer be inspected. Upstream: huggingface/transformers#48105 Fix: cap transformers<5.17, the version upstream vLLM tests against, until vLLM adopts the new vision rotary-embedding API. Signed-off-by: Paweł Olejniczak <pawelx.olejniczak@intel.com>

…emoved actorder/g_idx, populate region_num_blocks in (+3 more) (#1788) This PR consolidates 5 hourly-CI fixes against vllm@`d2906091bfc579cebefe3d8e8fb9077397ce9882`. ## Bug 1: drop removed actorder/g_idx kwargs from HPU WNA16 scheme - **State machine id**: compressed_tensors_wna16_actorder_no_longer_supported - **Commit**: 9b4b363 ### Root cause vllm#54809 removed GPTQ group/dynamic activation ordering support, deleting the actorder ctor kwarg, has_g_idx attribute, and weight_g_idx/w_gidx_param_name plumbing from CompressedTensorsWNA16 and MPLinearKernel. HPUCompressedTensorsWNA16 in vllm_gaudi/ops/hpu_compressed_tensors.py still passed/read those removed names, so every WNA16 checkpoint load hit TypeError before reaching the kernel. ### Culprit Regression introduced by [PR #54809](vllm-project/vllm#54809). ### Fix drop actorder/has_g_idx/w_gidx_param_name from the HPU WNA16 scheme and kernel, matching upstream's removal; retire the compressed_w4a16 MoE g_idx e2e job since its checkpoint declares actorder=group, which vLLM now rejects on every backend before hardware dispatch. ## Bug 2: populate region_num_blocks in HPU NIXL KV-cache registration - **State machine id**: nixl_pd_build_fa_local_region_num_blocks_indexerror - **Commit**: 0f8b433 ### Root cause vllm#53780 added a per-region region_num_blocks list to NixlBaseConnectorWorker and made _build_fa_local index it (self.region_num_blocks[i]). The HPU register_kv_caches override (hpu_nixl_connector.py) never populated that list, so the first index access raised IndexError on every NIXL PD job. ### Culprit Probable culprit: [PR #53780](vllm-project/vllm#53780) — pinned by symbol archaeology, bisect not run. Candidate range: [compare](vllm-project/vllm@cd64c2d...d290609). ### Fix append the per-region block count in the same loop that already populates block_len_per_layer/block_stride_per_layer, extend the region-count consistency assert, and mirror upstream's Mamba/hybrid-SSM override of region_num_blocks to the physical block count. ## Bug 3: register HPU Pixtral lazily - **State machine id**: opt_causal_lm_modelconfig_inspect_validation_error - **Commit**: c41afcd ### Root cause transformers 5.17.0 removed PixtralRotaryEmbedding and position_ids_in_meshgrid, so the eager Pixtral import in register_model() aborted plugin registration for every architecture. ### Culprit Regression introduced by [PR #48105](huggingface/transformers#48105). ### Fix drop the redundant eager import and keep only the lazy "module:class" registration. ## Bug 4: populate per-region NIXL registration bookkeeping - **State machine id**: nixl_register_local_xfer_region_mem_types_indexerror - **Commit**: 4ec533f ### Root cause vllm#53780 made the NIXL memory type, transfer group and name per-region, and register_local_xfer_handler now reads region_mem_types[0]; the HPU register_kv_caches override (a pre-#44456 copy kept to restore the K/V region split) never populated any of those lists, so every NIXL PD engine died with IndexError at KV cache registration. ### Culprit Regression introduced by [PR #53780](vllm-project/vllm#53780). ### Fix append region_mem_types / region_group_ids / region_names per region in the override's registration loop, derive _mixed_mem_types and _uses_region_group_mapping from them, and publish the local engine's dst_region_* geometry, matching upstream's post-#53780 tail. ## Bug 5: cap transformers below 5.17 - **State machine id**: mistral3_pixtral_rotary_embedding_import_error - **Commit**: 1e9da63 ### Root cause transformers 5.17.0 removed position_ids_in_meshgrid and renamed PixtralRotaryEmbedding to PixtralVisionRotaryEmbedding, both imported at module level by upstream vLLM's vllm/model_executor/models/pixtral.py, so PixtralForConditionalGeneration can no longer be inspected. ### Culprit Regression introduced by [PR #48105](huggingface/transformers#48105). ### Fix cap transformers<5.17, the version upstream vLLM tests against, until vLLM adopts the new vision rotary-embedding API. --------- Signed-off-by: Paweł Olejniczak <pawelx.olejniczak@intel.com>

* delete * fix repo * . * add video processor docs * seems like this is expected * style * fix repo * i dont understand * ah this was not needed * add docstring back where needed * self reveiw * modular has no way to stop copying docstring... * comments * unpack kwargs * fix tests * oke, at least these are same, let's keep digging * . * second . * and like this * draft * maybe * commit before i break it * i think this is the last one from varity of ropes, now to update all models * vision-only rope qwen * run fix-repo * this will take soo long to fix every model * sooooo long * ! * one by one, hopefully i fix it all by eow * fix the rest * and left-out sam * fix repo most * and now fix repo for special omni models * a bit more of fix repo * new model was merged, ci wont run * suppress TRF, it comes from `main` * wording * . * gemma4, standardize 2D position IDs tmrw * fix repo * testers were working on pure luck, love it * fix repo * failing tests * a bit more * i suppose revert, why i changed it * fix style * lost type annotations * fix Sam copied incorrectly * qwen3 vl fix * move sam positions to buffer * modular convert * what? why CI never complained?! * forgot this * add the decorators where missing * ssome comments * more * axial rope type register separately * forgot * this is way too long * i am making bad decisions, revert * return dtype is fp32 in most models * ooops revert * bit more, rest comes tmrw * i broke smth and i dont know what * need to fix repo again * skip the test for vision rope for now, will add later more * example * Revert "i am making bad decisions, revert" This reverts commit 852a0c8. * axial as type * fix repo * fix repo again * missing * gemma4 * fix * comments * new mrope with 2 sections only, no temporal! * fix repo

zucchini-nlp added 30 commits July 24, 2026 17:19

delete

0dc76b5

fix repo

5c458d8

.

1595f38

add video processor docs

b50faee

seems like this is expected

aa9d08f

main

6a8b9e3

style

1e8b724

fix repo

d15a604

i dont understand

b32da06

ah this was not needed

0eb1271

add docstring back where needed

53aee25

self reveiw

469e706

modular has no way to stop copying docstring...

dde1726

comments

961409a

unpack kwargs

79895ac

fix tests

e1ae9ac

Merge remote-tracking branch 'upstream/main' into rope-vision

2d41492

oke, at least these are same, let's keep digging

4223cdb

.

68e8bee

second .

c226909

and like this

6b6de13

draft

9af201e

Merge remote-tracking branch 'upstream/main' into rope-vision

3490850

maybe

b79c744

commit before i break it

ae5c85a

i think this is the last one from varity of ropes, now to update all …

5aade78

…models

vision-only rope qwen

f6c9dfc

run fix-repo

b3093fa

this will take soo long to fix every model

b092d0c

sooooo long

dd108f9

vasqu approved these changes Sep 3, 2026

View reviewed changes

Comment thread src/transformers/models/cohere_compass/configuration_cohere_compass.py Resolved

Comment thread src/transformers/models/qwen2_vl/modeling_qwen2_vl.py Resolved

Comment thread utils/create_dependency_mapping.py Resolved

Comment thread src/transformers/modeling_utils.py Resolved

zucchini-nlp added 3 commits September 4, 2026 10:45

comments

6e516e2

new mrope with 2 sections only, no temporal!

e5717eb

fix repo

3e03811

zucchini-nlp added this pull request to the merge queue Sep 4, 2026

Merged via the queue into huggingface:main with commit 4177486 Sep 4, 2026
112 checks passed

zucchini-nlp deleted the rope-vision branch September 4, 2026 10:01

sergereview Bot mentioned this pull request Sep 4, 2026

[serge] Fix 1 integration test for model qwen3_vl_moe failing with OOM (other (1)) #48538

Closed

pawel-olejniczak mentioned this pull request Sep 9, 2026

[FIX_FOR_VLLM_CUSTOM=d2906091bfc579cebefe3d8e8fb9077397ce9882] drop removed actorder/g_idx, populate region_num_blocks in (+3 more) vllm-project/vllm-gaudi#1788

Merged

mickaelseznec mentioned this pull request Sep 11, 2026

fix: Pixtral compat with transformers 5.17 vllm-project/vllm#56484

Open

4 tasks

jrp2014 mentioned this pull request Sep 11, 2026

20260911 -- Diagnostic run comparing 0.7.0rc0 → 0.7.0: like-for-like sweep of 33 cached VLMs (mostly "nothing to see") Neywa-Labs/mlx-vlm#2221

Closed

robin-bially mentioned this pull request Sep 25, 2026

[Bug]: v0.30.0 ships transformers 5.17, Pixtral-based models (LightOnOCR, Mistral3) fail with ImportError PixtralRotaryEmbedding vllm-project/vllm#58755

Closed

1 task done

cuichenx mentioned this pull request Sep 28, 2026

Fix MiniMax M3 partial 3D vision rotary embeddings #49164

Merged

zhtmike mentioned this pull request Oct 2, 2026

[RFC][tracking] Upstream patch debt: audited ledger of temporary compatibility patches verl-project/verl-omni#445

Open

38 tasks

This was referenced Oct 6, 2026

Pixtral Flash Attention 2 uses axial RoPE coordinates as packed sequence boundaries (regression in 5.17.0) #49371

Closed

Fix Pixtral Flash Attention image boundaries after axial RoPE refactor #49373

Merged

YaoleWu mentioned this pull request Oct 16, 2026

[Qwen3-VL] Bound vision RoPE inference FP32 intermediates #49665

Closed

2 of 5 tasks

mmangkad mentioned this pull request Oct 18, 2026

Fix CohereCompass text RoPE to use Qwen3-VL interleaved M-RoPE #49715

Merged

2 tasks done

ArthurZucker mentioned this pull request Oct 21, 2026

feat: RoPE-related typing improvements #41967

Closed

4 of 5 tasks

DManetti mentioned this pull request Oct 23, 2026

SAM3: Sam3ViTLayer.position_ids buffer is cast to bf16 on loading, degrading RoPE in global-attention layers (regression since v5.17.0, PR #48105) #49938

Open

sergereview Bot mentioned this pull request Oct 30, 2026

[serge] Fix 2 integration tests for model qwen3_vl_moe failing with OOM, raising OutOfMemoryError (other (2)) #50148

Open

molbap mentioned this pull request Nov 3, 2026

Bump transformers version to 5.22 huggingface/lerobot#5220

Merged

Navigation Menu

Repository navigation

🚨 Vision (2d/3d) rotary embeddings - #48105

🚨 Vision (2d/3d) rotary embeddings #48105
zucchini-nlp merged 87 commits into
huggingface:mainfrom
zucchini-nlp:rope-vision

zucchini-nlp commented Aug 19, 2026 •

edited by github-actions Bot

Loading

Uh oh!

github-actions Bot commented Sep 3, 2026

Uh oh!

vasqu left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

zucchini-nlp commented Aug 19, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 3, 2026

CI Results

Commit Info

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

github-actions Bot commented Sep 4, 2026

CI recap

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zucchini-nlp commented Aug 19, 2026 •

edited by github-actions Bot

Loading