Repository navigation
npu: the serve answers 503 after 5 s instead of queueing, so a second chat on a single-user appliance fails #1024
Description
What happens
On the appliance the NPU serve refuses a second chat with 503 the NPU is busy with another request instead of
queueing it. Owner, c23 board (4 Nov 16:4x, Lemonade from ryzen): the chat failed while the acceptance check's own
1000-token ladder row held the NPU for 14 s. The same shape happened on c22 at 12:24 (release check holding the device).
Cause is app/unified.cpp: the NPU route serialises generations on one std::timed_mutex and waits at most
--queue-ms, default 5000 ms (app/unified.cpp:81,197-198), then throws DeviceBusy → 503 + Retry-After: 1.
A single-user appliance's first request is routinely a full generation (4096 positions at ~70 tok/s is up to ~60 s), so
5 s is far below the normal hold time.
Why the 5 s is there, and why it cannot just be raised
The 5 s came from #911 arm 5: a second request waited 10.04 s with zero bytes on the wire, and Lemonade's streaming
proxy gives up at ~6 s — that is the board's "0 chunks in 5.9 s" + Backend did not send [DONE] marker. The 503 was
chosen to answer before the proxy died, so a silent queue could not reappear as a hung backend. Raising the wait
without fixing the silence would bring that back.
Fix
- Default
--queue-ms= 300000 (a full generation with headroom); 503 +Retry-Afteronly past it,0still
refuses at once. - While a streaming request waits, put a chunk on the wire every second (
delta: {content: ""}, inert for the
page and for any OpenAI client, and enough for a proxy's no-data timer). That is what makes the long queue safe. - Row: two concurrent chats, both complete — the second one queues and answers, no 503.
Also checked for this issue
Does Lemonade GUI3 fire a second request in parallel with a chat (title/summary generation)? In the fork's renderer there
are two /chat/completions call sites (LLMChatPanel.tsx:654 non-streaming agentic/tool loop, :856 the streaming
chat) plus lemonadeTools.ts:443; nothing generates a title or summary, and the tool loop is sequential within a turn.
So the 503 needs a foreign client (a check, another app, a second tab), which is exactly the board's case — the fix is
still mandatory, since a queue is the single-user behaviour either way.
Activity
os/mini: an NPU row says on the console that it is holding the device…
os/mini: an NPU row says on the console that it is holding the device…
bong-water-water-bong commentedon Nov 4, 2026
Fix + measured row are in #1031 (
326d91c5, branch1bit/npu-1024-queue). The row, onenpu-holdticket(
npu-1024-queue2, 17:34:48–17:35:25, released clean), serve on~/models/npu/Qwen3-0.6Bwith the stick's1024x3072x16x8kernels, chat A streamedmax_tokens: 1000and chat B streamedmax_tokens: 16starting 1 s later:queue_ms5000){"error":{"message":"the NPU is busy with another request"}}frame then[DONE]— the board's shapequeue_ms300000 + keep-alive)reasoning_contentdeltas,finish_reason: length,[DONE], no error frameamdxdnaERROR 0 andArray stopped dispatching0 for the whole ticket; serve alive after both chats; 0 serves leftalive afterwards. B is reasoning-only because thinking is on and 16 tokens is all reasoning — the claim is that it was
answered rather than refused.
GUI3 half, answered in the PR from the fork's code: two
/chat/completionscall sites (LLMChatPanel.tsx:654non-streaming tool loop,
:856the streaming chat) pluslemonadeTools.ts:443; nothing generates a title or summary andthe tool loop is sequential, so the second request has to come from a foreign client — the board's case. Not measured on
the wire.
Two gaps, said rather than implied: the second non-streamed chat was not measured (the row's second pass wrote into a
missing directory, so its figures are curl's failure, not the serve's), and a queue longer than one full generation is
only bounded by
--queue-ms, not exercised.os/v3/puppy: c24 prep - the outbound mDNS rule is stated and asserted…
os/mini/release-check.sh: a row for two concurrent NPU chats, both co…
npu: a second chat queues for the device instead of a 503 after 5 s (#…
os/mini/release-check.sh: a row for two concurrent NPU chats, both co…
npu: a non-streamed request keeps a short wait, because it cannot be …
bong-water-water-bong commentedon Nov 4, 2026
The CHANGES item ("the non-streamed arm waits silently up to 300 s — the #911 shape") is fixed at
adfc6fa4andmeasured on the device (
npu-hold -l npu-1024-plain, 18:28:45–18:29:02, 17 s, released clean):max_tokens: 1000[DONE]— the long chat stayed wholemax_tokens: 16, started 1 s in{"error":{"message":"the NPU is busy with another request","type":"device_busy"}}, headerRetry-After: 1amdxdnaERROR 0,Array stopped dispatching0. The bounds are now split and both explicit:--queue-ms(300000) for astreamed request, which ticks once a second while it queues, and
--queue-ms-plain(5000) for a non-streamed one,which cannot emit progress and is therefore refused promptly instead of left silent.
docs/npu.mdcarries the reasoning;the streamed arm's row (queued 13 s, answered at t=11.50 s) is unchanged from the earlier comment.
npu: say which request waits and which is refused, in the docs and th…
npu: a second chat queues for the device instead of a 503 after 5 s (#…
os/v3/puppy: the NPU busy row also covers a non-streamed client (#1024)…