[Yigtwxx]

What does this PR do?

GroundingDinoEncoderLayer.get_text_position_embeddings passes the int64 text_position_ids straight into encode_sinusoidal_position_embedding, which ends with .to(pos_tensor.dtype). Every sin/cos value in [-1, 1] is therefore truncated toward zero and the embedding collapses to {-1, 0, 1} — at the real d_model=256, 93.4% of it becomes exactly zero.

The branch is taken on every forward: GroundingDinoModel.forward calls the encoder with text_position_embedding=None and text_position_ids=position_ids (modeling_grounding_dino.py:2166-2168), where position_ids comes from generate_masks_with_special_tokens_and_transfer_map and is .to(torch.long) (:1854). The other branch, which does .float() before calling, is dead on the model path — and it is the one that shows the intended dtype.

This casts the ids to the text feature dtype instead, which keeps the values and also keeps the embedding in the model dtype under half precision:

-    text_position_ids[..., None], num_pos_feats=self.d_model
+    text_position_ids[..., None].to(text_features.dtype), num_pos_feats=self.d_model

mm_grounding_dino inherits this encoder layer through modular, so the change propagates to its generated file without a separate edit.

Where it came from

c9de1097ee (#41693) replaced the model's own get_sine_pos_embed with the shared encode_sinusoidal_position_embedding. The old helper returned torch.cat(position_embeddings, dim=-1) with no cast, so it produced float32 regardless of input dtype. The new one casts back to the input dtype, which is a no-op for every other caller (conditional_detr, dab_detr, lw_detr, rf_detr pass float anchor coordinates) and destructive for this one, which passes token position ids.

That commit changed the two modeling files and no test file, so the checked-in integration expectations still describe the pre-refactor behavior.

Tests

@slow integration tests against the real IDEA-Research/grounding-dino-tiny checkpoint. These ran on CPU, so they compare against the (None, None) entry of the Expectations blocks; I do not have the A10 the slow CI uses, so the exact deltas there may differ:

	main	this PR
GroundingDinoModelIntegrationTests	3 failed, 1 skipped	2 failed, 1 passed, 1 skipped
test_cross_attention_mask	failed, max abs diff 0.5683	passes
test_inference_object_detection_head	failed, max abs diff 0.3003	failed, max abs diff 0.1401
test_grounding_dino_loss	failed	failed

So one of the repo's own integration expectations goes green and another's divergence halves, without those expectations being touched.

I want to be precise about what this does not do: it does not fully restore them. test_grounding_dino_loss still fails and the detection head is still outside tolerance, which suggests the same refactor changed at least one more thing. I could not identify it, so I have left it rather than guess, and I would rather land the part that is established than bundle it with something unverified. Happy to keep digging if that is preferred.

Fast tests, both models, unchanged:

tests/models/grounding_dino tests/models/mm_grounding_dino
main    191 passed, 323 skipped
branch  191 passed, 323 skipped

ruff check, ruff format --check, modular_conversion and copies are clean.

Note this changes Grounding DINO's numerical outputs, in the direction of the expectations that predate the refactor. Let me know if you would like the title marked accordingly.

Fixes #47674

Disclosure: this fix was implemented with the assistance of a code agent, and reviewed and validated by me.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline and the
  Pull Request checks?
• Was this discussed/approved via a Github issue? Grounding DINO text position embeddings are truncated to int64, collapsing 93% of them to zero #47674
• Did you make sure to update the documentation with your changes according to the guidelines?
• Did you write any new necessary tests?

Who can review?

@yonigozlan @molbap

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: grounding_dino, mm_grounding_dino

[github-actions]

---

CI recap

Dashboard: View test results in Grafana
Latest run: 30636911850
Result: success | Grafana metrics are not available yet.

[molbap]

I think the cast should be at encode_sinusoidal_position_embedding, or tensors be created with the right dtype from init!

[molbap]

coming back to this @Yigtwxx: position_ids also feed the BERT embedding, so they have to stay long. The cast belongs where they become sinusoids now that I look back to this. Can you merge the two branches of get_text_position_embeddings so the arange path and the ids path share one .to(text_features.dtype) call? That also shortens the function.

Maybe something like

if text_position_embedding is not None:
    return text_position_embedding
if text_position_ids is None:
    text_position_ids = torch.arange(seq_length, device=text_features.device).expand(batch_size, -1)
return encode_sinusoidal_position_embedding(
    text_position_ids[..., None].to(text_features.dtype), num_pos_feats=self.d_model
)