[Performance]: Streamed responses search the whole output text for stop strings on every response #22020

Proposal to improve performance

DetokenizedGenerationResultBase._handle_response checks string stop
sequences after it detokenizes a response. On the streaming path it runs
stop_reason in beam_output.text (and then find) over the whole accumulated
text, for every stop string, on every response. With the default
stream_interval=1 that is one full-text search per generated token, so a
streamed request with stop strings spends time quadratic in its output
length. The search runs on the proxy's asyncio thread (the trtllm-serve
event loop when num_postprocess_workers=0, the default) or in a postprocess
worker, in series with the response handling of every other request.

Incremental detokenization only appends to text, and earlier responses
already searched the text before _last_text_len, so the first match must end
in the newly appended part. Starting the search len(stop) - 1 characters
before that point returns the same index as a full search. The full search
would stay where the whole output is decoded again from its tokens
(non-streaming, beam search, a trimmed stop word) and when a post-processor
hook is configured, because the hook can rewrite the last chunk after it was
searched.

CPU benchmark

Intel Xeon Platinum 8352Y, CPython 3.12.3, transformers 5.5.4, tokenizers
0.22.2, one pinned core. The benchmark drives the repository's
DetokenizedGenerationResultBase._handle_response with the Qwen3-0.6B
tokenizer (HF incremental detokenization), one response per generated token,
on English text from docs/source. The stop strings do not occur, so every
response is searched, as happens until a stop string appears. One warm-up
stream, then five runs per row; medians. Stop strings: ["</answer>"] and
["Question:", "</s>", "<|im_end|>"].

At 32K output tokens the change saves 1.3 s (one string) to 2.8 s (three
strings) of CPU per streamed request. Near 16K tokens each response
carries an extra 39-83 us; across 64 concurrent streams that is 2.5-5.3 ms of
serial work per decode step on the thread that also serves HTTP. Requests
without stop strings are unaffected.

26,000 randomized streams (both incremental detokenization backends, chunk
sizes 1-6, n > 1, beam search, non-streaming, token-level stop-word trims,
post-processor hooks, empty stop strings, multibyte byte-fallback tokens)
produce identical text deltas, final text, finish reasons and stop reasons on
main and with the change.

Current environment

• OS: Linux x86_64

• Python: 3.12.3

• CPU: Intel Xeon Platinum 8352Y

• GPU: not used

• Base: current main at 70fe2b2a6

• I searched open and closed issues and pull requests, including recent
  stop-string, streaming, detokenization and LLM API result-path work.

AI assistance (Cursor) was used to investigate and prepare this report.