Repository navigation
server: fix token counting API crash on sleep - #29309
Merged
Merged
Conversation
Contributor
|
Hello, reproduced the crash on master, a SIGSEGV on the first count request after sleep with and without mmproj, and the new test fails there too. With the fix, /chat/completions/input_tokens, /v1/chat/completions/input_tokens, /responses/input_tokens, /v1/responses/input_tokens and /v1/messages/count_tokens all wake the server and return the right count, and test_sleep.py passes. Nit: the three local aliases are not needed, reading ctx_server.vocab, ctx_server.mctx and ctx_server.init_opt inline after create_response() would keep it shorter. |
willweimike
marked this pull request as draft
September 23, 2026 12:09
willweimike
marked this pull request as ready for review
September 23, 2026 12:10
Contributor
Author
|
@ServeurpersoCom Thank you for your kind suggestion! I have refined it. |
ServeurpersoCom
approved these changes
Sep 23, 2026
ServeurpersoCom
left a comment
ServeurpersoCom
left a comment
Contributor
There was a problem hiding this comment.
LGTM, re-tested the updated version: every count route wakes the server without crashing and test_sleep.py passes.
Contributor
|
cc @ngxson |
ngxson
approved these changes
Sep 23, 2026
ngxson
changed the title
4 tasks
despc
added a commit
to despc/llama.cpp
that referenced
this pull request
Sep 26, 2026
Conflicts resolved by hand, keeping the fork's functionality: - allreduce.cu: our mixed-runtime collective kept; upstream's HIP enablement and CUDA_CHECK cleanup applied, #else stub comment taken from upstream. - llama-context.cpp/.h: upstream's per-graph-type gf_res_prev and gf_res_prev_active (ggml-org#28549, CUDA graph for MTP draft) around our ubatch phase profiler; expert heatmap/hotstore members kept. - server-context.cpp/.h: upstream's handle_count_tokens signature (ggml-org#29309) with our reject_unknown_model guard. - fattn-common.cuh: our GGML_CUDA_FATTN_LAUNCH_PROFILE log kept. qwen4exp.cpp auto-merged: our MTP head kept, upstream hc ops and sparse FA in (PR ggml-org#28243 still unmerged upstream, so MTP stays ours). Validation (greedy, temp 0 / top_k 1 / seed 42, hash of reasoning+content): - Qwen3.8-27B 4-GPU MTP: byte-identical to the old build on all prompts. - Qwen3.8-Flash 4-GPU MTP: 1 of 3 identical, 2 diverge late. Cause is the fused hc_pre_gated/hc_post ops from ggml-org#28901 (different rounding order); still diverges with GGML_CUDA_DISABLE_FUSION=1. test-backend-ops passes DSV4_HC_PRE/POST on CUDA0, CUDA1, V100_CUDA0, V100_CUDA1. Perplexity (docs corpus, 24x1024): 5.4967 +/- 0.127 old, 5.4707 +/- 0.126 new. Generation speed unchanged on both models. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Wizard815
pushed a commit
to Wizard815/mx-llama.cpp-Rocm10
that referenced
this pull request
Sep 29, 2026
* server: wake up sleeping server correctly * server: wake up sleeping server correctly (local aliases removed) (cherry picked from commit 42916d8)
frostyautumnleaf
pushed a commit
to frostyautumnleaf/llama.cpp
that referenced
this pull request
Oct 5, 2026
* server: wake up sleeping server correctly * server: wake up sleeping server correctly (local aliases removed)
edwardyoon
pushed a commit
to edwardyoon/focus-llama
that referenced
this pull request
Oct 7, 2026
* server: wake up sleeping server correctly * server: wake up sleeping server correctly (local aliases removed) (cherry picked from commit 42916d8)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Fixed #29188
Fixed that if the server was in a sleeping state, the token counting endpoint read
ctx_server.vocabwithout waking up correctly.Additional information
Add
test_sleep.pyto ensure that the endpoint can successfully wake up a server that is in sleep mode.Requirements