Repository navigation
Eval bug: Gemma4 + Eagle3 crashes llama-server with longer prompts #24637
Description
Name and Version
.\llama-server.exe --version
version: 9637 (aedb2a5)
built with Clang 20.1.8 for Windows x86_64
Operating systems
Windows
GGML backends
CUDA
Hardware
12900F + 4060Ti16GB
Models
Model: gemma-4-26B-A4B-it-UD-IQ3_S.gguf
Draft model: gemma-4-26B-A4B-it-speculator.eagle3-Q8_0.gguf
Problem description & steps to reproduce
Environment:
llama.cpp: 9637 (aedb2a5)
OS: Windows 11
GPU: RTX 4060 Ti 16GB
Model: gemma-4-26B-A4B-it-UD-IQ3_S.gguf
Draft model: gemma-4-26B-A4B-it-speculator.eagle3-Q8_0.gguf
Context size: 8192
Observed behavior:
The server works correctly with:
normal Gemma4 inference
Gemma4 + MTP
Gemma4 + Eagle3 on short prompts
However, with Eagle3 enabled, llama-server exits unexpectedly when the prompt becomes sufficiently long.
The process terminates without any error message, assertion, or stack trace.
The client receives either:
WinError 10054 (connection reset by peer)
WinError 10061 (connection refused on subsequent requests)
because the server process has already exited.
Additional observations:
The issue is reproducible with synthetic prompts such as repeated "你好\n" strings.
Prompt content does not appear to matter.
The failure appears related to prompt length / token count.
Streaming and non-streaming modes both reproduce the issue.
The issue reproduces through both llama.cpp server API and my RAG application.
MTP does not exhibit this behavior.
Relevant log excerpt:
......
28.09.831.381 I srv update_slots: all slots are idle
28.11.607.843 I srv params_from_: Chat format: peg-gemma4
28.11.608.054 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = 1683936586
28.11.608.331 I slot launch_slot_: id 0 | task 155 | processing task, is_child = 0
28.11.608.343 I slot update_slots: id 0 | task 155 | Checking checkpoint with [0, 559] against 0...
28.11.608.343 I slot update_slots: id 0 | task 155 | Checking checkpoint with [0, 12] against 0...
28.11.608.344 W slot update_slots: id 0 | task 155 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
28.11.608.347 W slot update_slots: id 0 | task 155 | erased invalidated context checkpoint (pos_min = 0, pos_max = 12, n_tokens = 13, n_swa = 1024, pos_next = 0, size = 0.741 MiB)
28.11.608.361 W slot update_slots: id 0 | task 155 | erased invalidated context checkpoint (pos_min = 0, pos_max = 559, n_tokens = 560, n_swa = 1024, pos_next = 0, size = 32.003 MiB)
28.11.700.918 I slot create_check: id 0 | task 155 | created context checkpoint 1 of 32 (pos_min = 0, pos_max = 161, n_tokens = 162, size = 9.257 MiB)
PS D:\AI\llama.cpp>
The server exits shortly after checkpoint creation and before prompt evaluation completes.
Prompt A:
564 tokens
Works
Prompt B:
1173 tokens
Works
Prompt C:
Longer prompt (~1500-2000+ tokens)
Server exits
Could this be related to Eagle3 draft context handling or prompt processing for longer inputs?
First Bad Commit
No response
Relevant log output
Logs
0.00.089.787 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.089.792 I device_info:
0.00.168.141 I - CUDA0 : NVIDIA GeForce RTX 4060 Ti (16379 MiB, 15233 MiB free)
0.00.168.150 I - CPU : 12th Gen Intel(R) Core(TM) i9-12900F (65369 MiB, 50991 MiB free)
0.00.168.199 I system_info: n_threads = 4 (n_threads_batch = 8) / 24 | CUDA : ARCHS = 500,610,700,750,800,860,890,900 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.00.168.236 I srv init: running without SSL
0.00.168.252 I srv init: api_keys: ****anae
0.00.168.271 I srv init: using 6 threads for HTTP server
0.00.168.276 I srv init: The UI is disabled
0.00.168.276 I srv init: Use --ui/--no-ui (or deprecated --webui/--no-webui) to enable/disable
0.00.168.371 I srv start: binding port with default address family
0.00.174.021 I srv llama_server: loading model
0.00.174.053 I srv load_model: loading model 'D:\AI\llama.cpp\models\gemma-4-26B-A4B-it-UD-IQ3_S.gguf'
0.00.792.084 I srv load_model: [spec] estimated memory usage of draft model is 832.50 MiB
0.00.792.096 I common_init_result: fitting params to device memory ...
0.00.792.097 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.02.029.694 W load: control-looking token: 1 '<eos>' was not control-type; this is probably a bug in the model. its type will be overridden
0.02.029.881 W load: control-looking token: 50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
0.02.030.192 W load: control-looking token: 212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.02.060.358 W load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
0.05.853.461 W llama_context: n_ctx_seq (8192) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.05.866.384 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
0.05.917.556 I srv load_model: loading draft model 'D:\AI\llama.cpp\models\eagle3\gemma-4-26B-A4B-it-speculator.eagle3-Q8_0.gguf'
0.06.109.425 W load: control-looking token: 212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.06.136.280 W load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
0.06.251.112 W llama_context: n_ctx_seq (8192) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.06.256.724 I srv load_model: initializing slots, n_slots = 1
0.06.289.157 I common_speculative_impl_draft_eagle3: adding speculative implementation 'draft-eagle3'
0.06.289.162 I common_speculative_impl_draft_eagle3: - n_max=4, n_min=0, p_min=0.500000
0.06.292.881 I srv load_model: speculative decoding context initialized
0.06.292.888 I slot load_model: id 0 | task -1 | new slot, n_ctx = 8192
0.06.292.953 I srv load_model: prompt cache is disabled - use `--cache-ram N` to enable it
0.06.292.953 I srv load_model: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
0.06.292.954 I srv load_model: context checkpoints enabled, max = 32, min spacing = 256
0.06.292.985 W srv init: --cache-idle-slots requires --cache-ram, disabling
0.06.300.642 I init: chat template, example_format: '<|turn>system
You are a helpful assistant<turn|>
<|turn>user
Hello<turn|>
<|turn>model
Hi there<turn|>
<|turn>user
How are you?<turn|>
<|turn>model
<|channel>thought
<channel|>'
0.06.301.340 I srv init: init: chat template, thinking = 0
0.06.301.380 I srv llama_server: model loaded
0.06.301.382 I srv llama_server: server is listening on http://127.0.0.1:8999
0.06.301.387 I srv update_slots: all slots are idle
0.24.233.810 I srv params_from_: Chat format: peg-gemma4
0.24.234.062 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = -1
0.24.234.282 I slot launch_slot_: id 0 | task 0 | processing task, is_child = 0
0.24.512.561 I slot create_check: id 0 | task 0 | created context checkpoint 1 of 32 (pos_min = 0, pos_max = 12, n_tokens = 13, size = 0.741 MiB)
0.24.912.899 I slot create_check: id 0 | task 0 | created context checkpoint 2 of 32 (pos_min = 0, pos_max = 559, n_tokens = 560, size = 32.003 MiB)
0.25.957.958 I slot print_timing: id 0 | task 0 | prompt eval time = 731.62 ms / 564 tokens ( 1.30 ms per token, 770.89 tokens per second)
0.25.957.964 I slot print_timing: id 0 | task 0 | eval time = 992.05 ms / 84 tokens ( 11.81 ms per token, 84.67 tokens per second)
0.25.957.965 I slot print_timing: id 0 | task 0 | total time = 1723.67 ms / 648 tokens
0.25.957.966 I slot print_timing: id 0 | task 0 | graphs reused = 11
0.25.957.966 I slot print_timing: id 0 | task 0 | draft acceptance = 0.72881 ( 43 accepted / 59 generated)
0.25.957.981 I statistics draft-eagle3: #calls(b,g,a) = 1 40 24, #gen drafts = 24, #acc drafts = 16, #gen tokens = 59, #acc tokens = 43, dur(b,g,a) = 0.000, 131.564, 0.015 ms
0.25.958.056 I slot release: id 0 | task 0 | stop processing: n_tokens = 647, truncated = 0
0.25.958.061 I srv update_slots: all slots are idle
0.27.705.324 I srv params_from_: Chat format: peg-gemma4
0.27.705.696 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = 20040779
0.27.705.874 I slot launch_slot_: id 0 | task 45 | processing task, is_child = 0
0.27.705.882 I slot update_slots: id 0 | task 45 | Checking checkpoint with [0, 559] against 0...
0.27.705.882 I slot update_slots: id 0 | task 45 | Checking checkpoint with [0, 12] against 0...
0.27.705.883 W slot update_slots: id 0 | task 45 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
0.27.705.886 W slot update_slots: id 0 | task 45 | erased invalidated context checkpoint (pos_min = 0, pos_max = 12, n_tokens = 13, n_swa = 1024, pos_next = 0, size = 0.741 MiB)
0.27.705.888 W slot update_slots: id 0 | task 45 | erased invalidated context checkpoint (pos_min = 0, pos_max = 559, n_tokens = 560, n_swa = 1024, pos_next = 0, size = 32.003 MiB)
0.27.799.640 I slot create_check: id 0 | task 45 | created context checkpoint 1 of 32 (pos_min = 0, pos_max = 161, n_tokens = 162, size = 9.257 MiB)
0.27.975.412 I slot create_check: id 0 | task 45 | created context checkpoint 2 of 32 (pos_min = 0, pos_max = 656, n_tokens = 657, size = 37.547 MiB)
0.28.163.520 I slot create_check: id 0 | task 45 | created context checkpoint 3 of 32 (pos_min = 0, pos_max = 1168, n_tokens = 1169, size = 58.842 MiB)
0.28.520.781 I slot print_timing: id 0 | task 45 | prompt eval time = 482.58 ms / 1173 tokens ( 0.41 ms per token, 2430.69 tokens per second)
0.28.520.786 I slot print_timing: id 0 | task 45 | eval time = 332.31 ms / 19 tokens ( 17.49 ms per token, 57.18 tokens per second)
0.28.520.787 I slot print_timing: id 0 | task 45 | total time = 814.89 ms / 1192 tokens
0.28.520.787 I slot print_timing: id 0 | task 45 | graphs reused = 23
0.28.520.788 I slot print_timing: id 0 | task 45 | draft acceptance = 0.00000 ( 0 accepted / 3 generated)
0.28.520.798 I statistics draft-eagle3: #calls(b,g,a) = 2 58 27, #gen drafts = 27, #acc drafts = 16, #gen tokens = 62, #acc tokens = 43, dur(b,g,a) = 0.001, 162.569, 0.017 ms
0.28.520.866 I slot release: id 0 | task 45 | stop processing: n_tokens = 1191, truncated = 0
0.28.520.878 I srv update_slots: all slots are idle
27.44.486.146 I srv params_from_: Chat format: peg-gemma4
27.44.486.363 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = 22603629
27.44.486.649 I slot launch_slot_: id 0 | task 68 | processing task, is_child = 0
27.44.486.662 I slot update_slots: id 0 | task 68 | Checking checkpoint with [0, 1168] against 0...
27.44.486.663 I slot update_slots: id 0 | task 68 | Checking checkpoint with [0, 656] against 0...
27.44.486.663 I slot update_slots: id 0 | task 68 | Checking checkpoint with [0, 161] against 0...
27.44.486.664 W slot update_slots: id 0 | task 68 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
27.44.486.669 W slot update_slots: id 0 | task 68 | erased invalidated context checkpoint (pos_min = 0, pos_max = 161, n_tokens = 162, n_swa = 1024, pos_next = 0, size = 9.257 MiB)
27.44.486.919 W slot update_slots: id 0 | task 68 | erased invalidated context checkpoint (pos_min = 0, pos_max = 656, n_tokens = 657, n_swa = 1024, pos_next = 0, size = 37.547 MiB)
27.44.487.899 W slot update_slots: id 0 | task 68 | erased invalidated context checkpoint (pos_min = 0, pos_max = 1168, n_tokens = 1169, n_swa = 1024, pos_next = 0, size = 58.842 MiB)
27.44.878.482 I slot create_check: id 0 | task 68 | created context checkpoint 1 of 32 (pos_min = 0, pos_max = 12, n_tokens = 13, size = 0.741 MiB)
27.45.132.737 I slot create_check: id 0 | task 68 | created context checkpoint 2 of 32 (pos_min = 0, pos_max = 559, n_tokens = 560, size = 32.003 MiB)
27.46.127.247 I slot print_timing: id 0 | task 68 | prompt eval time = 693.50 ms / 564 tokens ( 1.23 ms per token, 813.27 tokens per second)
27.46.127.252 I slot print_timing: id 0 | task 68 | eval time = 947.09 ms / 84 tokens ( 11.27 ms per token, 88.69 tokens per second)
27.46.127.253 I slot print_timing: id 0 | task 68 | total time = 1640.59 ms / 648 tokens
27.46.127.254 I slot print_timing: id 0 | task 68 | graphs reused = 33
27.46.127.254 I slot print_timing: id 0 | task 68 | draft acceptance = 0.72881 ( 43 accepted / 59 generated)
27.46.127.269 I statistics draft-eagle3: #calls(b,g,a) = 3 98 51, #gen drafts = 51, #acc drafts = 32, #gen tokens = 121, #acc tokens = 86, dur(b,g,a) = 0.002, 292.346, 0.035 ms
27.46.127.364 I slot release: id 0 | task 68 | stop processing: n_tokens = 647, truncated = 0
27.46.127.376 I srv update_slots: all slots are idle
28.08.524.701 I srv params_from_: Chat format: peg-gemma4
28.08.524.908 I slot get_availabl: id 0 | task -1 | selected slot by LCP similarity, sim_best = 1.000 (> 0.100 thold), f_keep = 0.872
28.08.525.372 I slot launch_slot_: id 0 | task 113 | processing task, is_child = 0
28.08.525.387 I slot update_slots: id 0 | task 113 | Checking checkpoint with [0, 559] against 0...
28.08.555.283 W slot update_slots: id 0 | task 113 | restored context checkpoint (pos_min = 0, pos_max = 559, n_tokens = 560, n_past = 559, size = 32.003 MiB)
28.09.831.246 I slot print_timing: id 0 | task 113 | prompt eval time = 370.56 ms / 5 tokens ( 74.11 ms per token, 13.49 tokens per second)
28.09.831.252 I slot print_timing: id 0 | task 113 | eval time = 935.32 ms / 84 tokens ( 11.13 ms per token, 89.81 tokens per second)
28.09.831.252 I slot print_timing: id 0 | task 113 | total time = 1305.87 ms / 89 tokens
28.09.831.253 I slot print_timing: id 0 | task 113 | graphs reused = 44
28.09.831.254 I slot print_timing: id 0 | task 113 | draft acceptance = 0.68750 ( 44 accepted / 64 generated)
28.09.831.266 I statistics draft-eagle3: #calls(b,g,a) = 4 137 75, #gen drafts = 75, #acc drafts = 48, #gen tokens = 185, #acc tokens = 130, dur(b,g,a) = 0.003, 424.418, 0.051 ms
28.09.831.363 I slot release: id 0 | task 113 | stop processing: n_tokens = 647, truncated = 0
28.09.831.381 I srv update_slots: all slots are idle
28.11.607.843 I srv params_from_: Chat format: peg-gemma4
28.11.608.054 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = 1683936586
28.11.608.331 I slot launch_slot_: id 0 | task 155 | processing task, is_child = 0
28.11.608.343 I slot update_slots: id 0 | task 155 | Checking checkpoint with [0, 559] against 0...
28.11.608.343 I slot update_slots: id 0 | task 155 | Checking checkpoint with [0, 12] against 0...
28.11.608.344 W slot update_slots: id 0 | task 155 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
28.11.608.347 W slot update_slots: id 0 | task 155 | erased invalidated context checkpoint (pos_min = 0, pos_max = 12, n_tokens = 13, n_swa = 1024, pos_next = 0, size = 0.741 MiB)
28.11.608.361 W slot update_slots: id 0 | task 155 | erased invalidated context checkpoint (pos_min = 0, pos_max = 559, n_tokens = 560, n_swa = 1024, pos_next = 0, size = 32.003 MiB)
28.11.700.918 I slot create_check: id 0 | task 155 | created context checkpoint 1 of 32 (pos_min = 0, pos_max = 161, n_tokens = 162, size = 9.257 MiB)
PS D:\AI\llama.cpp>
Activity
ruixiang63 commentedon Jun 16, 2026
Can you try this PR #24707? @ShionWakanae I reproduced the issue and confirmed it’s a bug.
BTW, the low acceptance rate on long prompts has been proven to be caused by the draft model quality issue. More details are included in the PR.
ShionWakanae commentedon Jun 17, 2026
Wow, what did I miss... I saw that the PR has been merged, and then I tested b9686, and eagle3 is working properly now with long prompt (both dense and moe gemma4). Many thanks!