[Bug]: Streamed Responses requests on gpt-oss never release their Harmony parser state #22112

System Info

• TensorRT-LLM main at fec3d9191, trtllm-serve with a gpt-oss (Harmony) checkpoint: POST /v1/responses with "stream": true, and POST /v1/chat/completions streams that the client abandons.
• Found while reading the serve code and reproduced on CPU with the repository code (snippet below), openai-harmony==0.0.4 as pinned in requirements.txt. No GPU run.

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

The process-wide HarmonyAdapter (get_harmony_adapter()) keeps one HarmonyStreamState per stream in _stream_states. Each state holds a Harmony StreamableParser with every token and message of its request until cleanup_stream_state removes it.

• Responses API: _generate_streaming_event_harmony creates the state under responses-api-<request_id>, and nothing on the Responses path calls cleanup_stream_state. Every streamed Responses request leaves its state behind for the life of the server process, or of the postprocess worker that handled it.
• Chat Completions: handle_streaming_response releases the state when the stream finishes or fails. Without postprocess workers (the default), the post-processor runs inside the streaming generator, and a client that disconnects closes that generator before the final result is post-processed, so the state of that stream stays too.

The snippet runs 100 streamed Responses requests through the server's post-processor, one token per response:

from tensorrt_llm.executor.result import CompletionOutput
from tensorrt_llm.serve.harmony_adapter import get_harmony_adapter
from tensorrt_llm.serve.openai_protocol import ResponsesRequest
from tensorrt_llm.serve.postprocess_handlers import (
    ResponsesAPIPostprocArgs, responses_api_streaming_post_processor)
from tensorrt_llm.serve.responses_utils import ResponsesStreamingProcessor

adapter = get_harmony_adapter()
tokens = adapter.encoding.encode(
    "<|channel|>analysis<|message|>Look up the order and answer briefly.<|end|>"
    "<|start|>assistant<|channel|>final<|message|>It shipped on Monday.",
    allowed_special="all")


class Result:
    """The fields of a streamed GenerationResult that the post-processor reads."""

    def __init__(self):
        self.outputs = [CompletionOutput(0)]
        self._done = False
        self.prompt_token_ids = [1, 2, 3]
        self.cached_tokens = 0


for _ in range(100):
    request = ResponsesRequest(model="gpt-oss", input="Where is my order?", stream=True)
    sampling_params = request.to_sampling_params()
    processor = ResponsesStreamingProcessor(request, sampling_params, "gpt-oss",
                                            use_harmony=True)
    args = ResponsesAPIPostprocArgs(model="gpt-oss", request=request,
                                    sampling_params=sampling_params, use_harmony=True,
                                    streaming_processor=processor)
    result = Result()
    output = result.outputs[0]
    for i, token in enumerate(tokens):  # one token per response, as the engine streams it
        output._last_token_ids_len = len(output.token_ids)
        output.token_ids.append(token)
        result._done = i + 1 == len(tokens)
        if result._done:
            output.finish_reason = "stop"
        responses_api_streaming_post_processor(result, args)

print(len(adapter._stream_states))

It prints 100 on main.

The memory grows with the number of streamed requests and their length. Same loop with longer outputs (an analysis message followed by the answer), 200 requests, resident memory of the process before and after:

At about 44 KiB per 4K-token response, that is roughly 4 GiB per 100,000 streamed responses.

Expected behavior

A stream's Harmony state is released when the stream ends: after its final result is post-processed, when post-processing fails, and when the client disconnects.

actual behavior

Streamed Responses requests never release their state. Chat Completions streams release it when they finish or fail, but not when the client disconnects.

additional notes

Non-streaming requests are not affected: they parse the whole output with a new parser. I have a fix with CPU tests and will open a PR that references this issue.