[Bug]: Streaming tool parser drops all response content emitted after a completed tool call #17740

System Info

Reproduced against main by driving the parser directly; no GPU or model weights involved. The affected code is pure Python in tensorrt_llm/serve/tool_parser/.

• CPU-only, tensorrt_llm/serve/tool_parser/base_tool_parser.py at main
• Applies to any parser that inherits BaseToolParser.parse_streaming_increment and whose eot_token starts with its tool_call_separator — Qwen3ToolParser is the concrete case below

Who can help?

@tongyuantongyu @zhaoyangwang-nvidia

Surfaced during review of #17575 (thread). It is a distinct defect from the one that PR fixes, so filing it separately.

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Stream a Qwen3-format tool call followed by ordinary assistant text:

from tensorrt_llm.serve.openai_protocol import (ChatCompletionToolsParam,
                                                FunctionDefinition)
from tensorrt_llm.serve.tool_parser.qwen3_tool_parser import Qwen3ToolParser

tools = [
    ChatCompletionToolsParam(function=FunctionDefinition(
        name="get_weather",
        description="w",
        parameters={"type": "object",
                    "properties": {"city": {"type": "string"}}},
    ))
]

parser = Qwen3ToolParser()
chunks = [
    "<tool_call>\n",
    '{"name": "get_weather", "arguments": {"city": "Paris"}}',
    "\n</tool_call>",
    " It is sunny.",
]
for c in chunks:
    r = parser.parse_streaming_increment(c, tools)
    print(f"in={c!r:60} normal_text={r.normal_text!r:16} "
          f"calls={[(x.tool_index, x.name, x.parameters) for x in r.calls]}")
print("FINAL _buffer =", repr(parser._buffer))

Expected behavior

The final chunk is ordinary assistant content and should be emitted as normal_text, with the buffer drained:

in=' It is sunny.'   normal_text=' It is sunny.'   calls=[]
FINAL _buffer = ''

This is what happens when the closing tag is not preceded by a newline — replacing "\n</tool_call>" with "</tool_call>" in the script above produces exactly the output above. That control case is the clearest evidence the newline is what makes the difference.

actual behavior

The trailing content is swallowed and the buffer never drains:

in='<tool_call>\n'                                       normal_text='' calls=[]
in='{"name": "get_weather", "arguments": {"city": "Paris"}}'
                                                         normal_text='' calls=[(0, 'get_weather', '')]
in='\n</tool_call>'                                      normal_text='' calls=[(0, None, '{"city": "Paris"}')]
in=' It is sunny.'                                       normal_text='' calls=[]
FINAL _buffer = '\n</tool_call> It is sunny.'

The tool call itself parses correctly. Only the content after it is lost, and every later chunk in the request is lost the same way, since the buffer only grows from that point on.

Cause, as far as I can tell — base_tool_parser.py:136-138 enters the tool-call branch on either of two conditions:

if not (self.has_tool_call(current_text) or
        (self.current_tool_id > 0
         and current_text.startswith(self.tool_call_separator))):

For Qwen3ToolParser, tool_call_separator is "\n" and eot_token is "\n</tool_call>" (qwen3_tool_parser.py:48-49). After a call completes, :249 leaves the remainder starting at the eot_token, so the remainder begins with "\n" — the separator — by construction of the format. The second clause therefore always holds. The parser stays in the tool-call branch, partial_json_loads raises MalformedJSON, and :186 returns early without touching the buffer, so the eot_token stripping at :143-144 is never reached. The </tool_call> scrubbing in Qwen3ToolParser._wrapped_streaming also never runs, because it is gated on result.normal_text being non-empty.

The second clause is presumably meant to catch the start of a subsequent tool call after a separator. It currently also matches a separator that is merely the leading character of the end token.

Notes:

• Independent of [#17574][fix] Complete zero-argument tool calls in the streaming tool parser #17575: it reproduces before and after that change, and with non-empty arguments, so it is not related to the argument-truthiness gate.
• Any parser inheriting the base streaming path whose eot_token begins with its tool_call_separator is exposed; I only verified Qwen3ToolParser.
• User-visible effect on trtllm-serve: with a streamed request, any assistant text the model produces after a tool call is silently dropped from the response, while the same generation parsed through detect_and_parse keeps it.

I am happy to open a PR for this if the approach is agreed. The fix needs a decision on when the separator clause should be allowed to match — e.g. requiring that what follows the separator is not the end token — and it touches a method shared by every parser on the base streaming path, so I would rather confirm the intended semantics first.

additional notes

Verified with a direct-import harness against main; only pure-Python parser code is exercised.