[Bug]: qwen3 tool parser: strict tool calls pick the wrong function because the grammar forbids the spaces Qwen3 writes #21903

System Info

• CPU architecture: x86_64
• GPU: 1x NVIDIA L40S, 48 GB
• TensorRT-LLM: v1.4.0rc2 (container nvcr.io/nvidia/tensorrt-llm/release:1.4.0rc2), also still present on main
• Libraries: xgrammar 0.1.32, transformers 5.5.4
• NVIDIA driver: 595.91.07
• OS: Ubuntu 24.04
• Models: Qwen/Qwen3-8B and nvidia/Qwen3-8B-FP8
• Server: trtllm-serve <model> --tool_parser qwen3
• Note: the bug is in Python (the qwen3 tool parser), and the CPU check below reproduces it without a GPU

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

For tools sent with "strict": true, trtllm-serve builds an xgrammar structural tag from Qwen3ToolParser.structure_info() (tensorrt_llm/serve/tool_parser/qwen3_tool_parser.py), which forces this prefix:

begin='<tool_call>\n{"name":"' + name + '", "arguments":',

Qwen3's chat template writes tool calls as {"name": "<name>", "arguments": {...}}, with a space after each colon. The grammar forbids both spaces, so at the point where the model picks the function name it can't produce its usual " token. With several tools offered, it often picks the wrong one.

Steps to reproduce the behavior:

1. pip install "xgrammar==0.1.32" "transformers==5.5.4" (the versions 1.4.0rc2 pins)
2. Save the script below as repro.py
3. python repro.py

The script runs on CPU and doesn't import tensorrt_llm. It copies the prefix strings from Qwen3ToolParser.structure_info and builds the same structural tag as _build_tool_strict_guided_decoding_params, then checks whether that grammar accepts the tool call Qwen3's chat template writes. The GPU run below confirms the result with real trtllm-serve.

"""TensorRT-LLM's strict-tools grammar for --tool_parser qwen3 rejects Qwen3's own tool-call format.

pip install "xgrammar==0.1.32" "transformers==5.5.4"
"""

import json

import xgrammar as xgr
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
unit = {"type": "string", "enum": ["celsius", "fahrenheit"]}


def tool(name, props, required):
    params = {"type": "object", "properties": props, "required": required,
              "additionalProperties": False}
    return {"type": "function", "function": {"name": name, "parameters": params, "strict": True}}


TOOLS = [
    tool("get_weather", {"city": {"type": "string"}, "unit": unit}, ["city"]),
    tool("set_thermostat", {"temperature": {"type": "number"}, "unit": unit}, ["temperature", "unit"]),
    tool("set_timer", {"minutes": {"type": "integer"}}, ["minutes"]),
]

# The tool call Qwen3's chat template writes for the correct answer.
msgs = [{"role": "user", "content": "Turn the house temperature down to 19 celsius."}]
call = {"type": "function",
        "function": {"name": "set_thermostat", "arguments": {"temperature": 19, "unit": "celsius"}}}
kw = dict(tools=TOOLS, tokenize=False, enable_thinking=False)
prompt = tok.apply_chat_template(msgs, add_generation_prompt=True, **kw)
full = tok.apply_chat_template(msgs + [{"role": "assistant", "content": "", "tool_calls": [call]}], **kw)
reply = full[len(prompt):].split("<|im_end|>")[0]
ids = tok.encode(reply, add_special_tokens=False)
print("Qwen3 chat template writes:", json.dumps(reply), "\n")

info = xgr.TokenizerInfo.from_huggingface(tok)
compiler = xgr.GrammarCompiler(info)


def check(label, begin):
    # Same structural tag trtllm-serve builds in _build_tool_strict_guided_decoding_params.
    tags = [{"begin": begin.replace("NAME", t["function"]["name"]),
             "content": {"type": "json_schema", "json_schema": t["function"]["parameters"]},
             "end": "}\n</tool_call>"} for t in TOOLS]
    stag = {"type": "structural_tag",
            "format": {"type": "triggered_tags", "triggers": ["<tool_call>"], "tags": tags}}
    matcher = xgr.GrammarMatcher(compiler.compile_structural_tag(json.dumps(stag)))
    for i, t in enumerate(ids):
        if not matcher.accept_token(t):
            mask = xgr.allocate_token_bitmask(1, info.vocab_size)
            matcher.fill_next_token_bitmask(mask)
            allowed = [tok.decode([j]) for j in range(info.vocab_size)
                       if (int(mask[0, j // 32]) >> (j % 32)) & 1]
            print(f"{label}: REJECTED {tok.decode([t])!r} after {tok.decode(ids[:i])!r}")
            print(f"  allowed instead: {sorted(allowed)[:8]}")
            return
    print(f"{label}: ACCEPTED the whole call")


# Qwen3ToolParser.structure_info, as shipped in v1.4.0rc2 and on main:
check("as shipped", '<tool_call>\n{"name":"NAME", "arguments":')
check("only the first space", '<tool_call>\n{"name": "NAME", "arguments":')
check("both spaces (the fix)", '<tool_call>\n{"name": "NAME", "arguments": ')

Output:

Qwen3 chat template writes: "<tool_call>\n{\"name\": \"set_thermostat\", \"arguments\": {\"temperature\": 19, \"unit\": \"celsius\"}}\n</tool_call>"

as shipped: REJECTED ' "' after '<tool_call>\n{"name":'
  allowed instead: ['"', '"g', '"s']
only the first space: REJECTED ' {"' after '<tool_call>\n{"name": "set_thermostat", "arguments":'
  allowed instead: ['{', '{\n', '{\n\n', '{\n\n\n', '{"']
both spaces (the fix): ACCEPTED the whole call

End-to-end on GPU with real trtllm-serve (Qwen3-8B BF16, 1.4.0rc2 container, 45 requests that each offer 3 or 4 tools, greedy). The only change in the third run is the two spaces in structure_info. Script and raw outputs: https://github.com/arulster17/loom/tree/b3c4f15/repro/trtllm-qwen3-strict-tool-prefix

Expected behavior

With strict tools, the grammar should accept tool calls exactly as Qwen3's chat template writes them (<tool_call>\n{"name": "<name>", "arguments": {...}}\n</tool_call>), and the model should pick the same function it picks with plain tools.

For example, "Turn the house temperature down to 19 celsius." with the tools get_weather, set_thermostat and set_timer should call set_thermostat(temperature=19, unit="celsius").

actual behavior

The grammar rejects Qwen3's tool-call format at the space after "name": (see Reproduction), and on multi-tool requests the model often calls the wrong function.

For the example above, from the GPU run linked in Reproduction:

• Strict tools: get_weather(city="New York", unit="celsius")
• Plain tools: set_thermostat(temperature=19, unit="celsius")
• Strict tools with the two spaces restored: set_thermostat(temperature=19, unit="celsius")

vLLM 0.30.0 (hermes parser) answers the same strict request correctly.

On a larger benchmark (135 items, 3 passes each), strict tools scored 84.0% vs 97.8% for plain tools on BF16, and 86.4% vs 98.3% on FP8. Single-tool items were unaffected. The whole drop was on multi-tool items, almost all of it the wrong function.

additional notes

Suggested fix in Qwen3ToolParser.structure_info:

begin='<tool_call>\n{"name": "' + name + '", "arguments": ',

This matches vLLM's hermes structural tag and xgrammar's own built-in qwen_3 tag. SGLang's Qwen25Detector has the same unspaced string as TensorRT-LLM.

Things to check with the fix:

• Named tool_choice reuses the same begin string as a prompt prefix (_build_forced_tool_call_decoding in openai_server.py). With the fix, that prompt would end in a trailing space, while Qwen3 tokenizes that space together with {". That path may need the trailing space stripped.
• With --tool_parser auto, this parser is also used for qwen2, qwen3_moe, qwen3_5, qwen3_5_moe and qwen3_next. I've only tested Qwen3-8B.
• The class docstring and test_structure_info_function use the unspaced format too.

I would like to open a PR with the fix, updated tests, and a regression test that checks the grammar accepts the chat template's output.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.