[Larslllllll]

Summary

Fixes LukeTokenizer dropping <s></s> special tokens when encoding plain text.

Problem

LukeTokenizer was returning [31414, 232] instead of the expected [0, 31414, 232, 2] for plain text input:

tok("Hello world")["input_ids"]
# Before:  [31414, 232]
# After:   [0, 31414, 232, 2]  ✓

The add_special_tokens=False flag was also being ignored, and special_tokens_mask was returning [0, 0] instead of [1, 0, 0, 1].

Root Cause

LukeTokenizer.__init__ builds a raw BPE tokenizer with a pre-tokenizer and decoder but no post-processor. After #40936 switched to the tokenizers backend, plain-text calls no longer route through build_inputs_with_special_tokens. LUKE is the only tokenizer that defines build_inputs_with_special_tokens without a post-processor set.

Fix

After super().__init__(), set the RobertaProcessing post-processor (if cls/sep token IDs are available), matching how RobertaTokenizer handles this:

if self.cls_token_id is not None and self.sep_token_id is not None:
    self._tokenizer.post_processor = processors.RobertaProcessing(
        (str(sep_token), self.sep_token_id),
        (str(cls_token), self.cls_token_id),
        True,  # add_prefix_space
    )

Testing

• RUN_SLOW=1 pytest tests/models/luke/test_tokenization_luke.py::LukeTokenizerTest::test_sequence_builders — previously failing, now passes
• All 52 non-slow LUKE tokenizer tests pass (5 skipped, 14 deselected)
• No regressions to entity_spans, entity_position_ids, or offset mappings

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: luke

[github-actions]

---

CI recap

Dashboard: View test results in Grafana
Latest run: 34001393243:2
Result: success | Jobs: 2 | Tests: 20 | Failures: 0 | Duration: 2m 20s

[vasqu]

Imo doesnt make sense because that would change existing behavior is there any reason. Seems like an agent maybe :D

[vasqu]

cc @itazap tho!

[itazap]

#48408 will fix this

[ArthurZucker]

PR is merged, closing