[flyingdoubleG]

Summary

The logprob simulator (NeuronSimulator._simulate_with_logprobs, used by the simulation scorer) returns all-zero predictions in two common setups, without raising an error:

tokenized_prompt = self.client.tokenizer.apply_chat_template(prompt, add_generation_prompt=True)
tab_token = self.client.tokenizer.encode("\t")[1]   # "Use [1] to skip BOS token"

1. transformers >= 5. apply_chat_template now returns a BatchEncoding by default (return_dict=True), so tokenized_prompt[-3] in parse_simulation_response raises IndexError. pyproject.toml leaves transformers unpinned, and current vLLM requires transformers>=5.16.1,<5.20, so fresh installs hit this.
2. Explainer tokenizers without a BOS token (e.g. Qwen2.5). encode("\t") is [197], so [1] raises IndexError on any transformers version.

simulate() catches the exception and returns zeros ("Simulation failed: list index out of range, returning zeros"). Every per-sequence and per-quantile correlation then becomes NaN.

Repro with a fake client that returns P("7") = P("0") = 0.5 after every tab, so each token's expected activation should be 3.5:

	Llama-3.1 tokenizer (default explainer)	Qwen2.5 tokenizer
main, transformers 4.57	3.5	0.0
main, transformers 5.20	0.0	0.0
this PR, both versions	3.5	3.5

Changes

• simulator.py: call apply_chat_template(..., return_dict=False), and get the tab token with encode("\t", add_special_tokens=False)[-1].
• clients/offline.py: the Offline client also passes apply_chat_template(..., tokenize=True) to TokensPrompt(prompt_token_ids=...) and takes len() of it for statistics. On transformers >= 5 that is a BatchEncoding, not a list of ids, so return_dict=False is added there as well. I could not run vLLM locally (Windows), so this part is covered by the reasoning above rather than a test.

return_dict exists in transformers 4.x (default False), so behaviour there is unchanged apart from the BOS fix.

Similar in spirit to #176 / #177 (batch_decode on transformers 5).

Tests

tests/test_scorers/test_simulator_tokenization.py runs NeuronSimulator.simulate with the fake client for the Llama-3.1 (BOS) and Qwen2.5 (no BOS) tokenizers, and checks that the expected activations are 3.5.

• transformers 5.20: both cases fail on main and pass with this PR.
• transformers 4.57: the Qwen case fails on main; both pass with this PR.
• ruff check (with the repo's excludes) and black --check pass on the changed files. pyright reports only unresolved imports from my local environment, the same as on main.

Locally I ran the test against stubbed vllm/faiss modules and without tests/conftest.py, since vLLM does not install on Windows. In CI it imports normally.