Repository navigation
Last message when streaming completion should not be empty #1038
Description
When streaming completions the last message is sometimes empty to ensure that the last message contains a finish_reason. This is correct according to the OpenAI schema but is not expected by all clients and can cause bugs (noticed this in openplayground).
Activity
BlueBlazin commentedon Dec 25, 2023
So what's the alternative?
thiner commentedon Dec 26, 2023
I encountered this problem with the latest docker image for inferencing. When streeaming via /chat/completions API, the stream will be interrupted. The output obviously not complete yet. Call the API with the same input will not encounter this error.
Fix #1038