Misc. bug: CUDA: alternating variants sharing one graph cache key cause warmup/replay interference

[Lisiyuan233]

Name and Version

version: 0.4.0-dev (build 0) + built with GNU 13.3.0 for Linux x86_64
reproduced at master @ e2d2c0d6aa9b996d5d3a3c1d5e24c8c19728bb3d（2026-09-09）
real-model A/B：master @ 4d9176092d00586775af140581bb0b558ddc4389（2026-09-04）

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

libllama (core library)

Command line

Problem description & steps to reproduce

issue-first-attach-20260909.zip
Affected component: the ggml CUDA backend's CUDA-graph cache (warmup/update state), reached via libllama / llama-server.

Operating systems

WSL2 Ubuntu 24.04.4（kernel 6.18.33.1-microsoft-standard-WSL2）、RTX 4060 Ti 16GB（driver 560.94 ）、CUDA 12.0 toolkit / sm_89、g++ 13.3.0、CMake 3.28.3、Ryzen 5 7500F 6C12T / 16GB。

Problem description & steps to reproduce

Symptom
Under a single CUDA-graph identity, execution variants that alternate across invocations repeatedly invalidate the shared warmup / update state. As a result, the affected recurring draft-batch variants fail to converge to replay.
Mechanistic evidence (real workload, numbers taken straight from the logs)

Build / control	CUDA graph warmup	Gen throughput	Median / interpretation
vanilla (no patch)	complete=68, reset=68	173.5 / 175.9 / 166.2 t/s	173.5 t/s
variant-keys experiment	complete=7, reset=3	202.4 / 205.3 / 206.0 t/s	205.3 t/s (~+18.3%)
no-spec paired control	—	154.25 vs 155.90 t/s	no measurable regression in this 1024-token steady-state control
GGML_CUDA_DISABLE_GRAPHS=1 control	—	139.1 / 145.9 t/s	graph-disabled control

Counter-intuitive data point worth flagging
On the vanilla build, speculative decoding on this workload ends up slower than no-spec (173.5 vs 188.7 t/s); with the key refinement it reaches 205.3 t/s.
Expected behavior Each stable, looping variant should converge to graph replay independently, instead of repeatedly clobbering the shared state.
Scope limitation the first-node + shape keying granularity cannot distinguish downstream-only attribute changes.

steps to reproduce：

steps：
Route 1 — real model, vanilla master only：

1. Build current master with GGML_CUDA=ON
2. download the models below 

huggingface-cli download Qwen/Qwen2.5-1.5B-Instruct-GGUF qwen2.5-1.5b-instruct-q4_k_m.gguf --local-dir .
huggingface-cli download Qwen/Qwen2.5-0.5B-Instruct-GGUF qwen2.5-0.5b-instruct-q4_k_m.gguf --local-dir .

3.start server (verbose is required: the warmup counters are DEBUG-level logs)

llama-server -m qwen2.5-1.5b-instruct-q4_k_m.gguf -ngl 99 \
    -md qwen2.5-0.5b-instruct-q4_k_m.gguf -ngld 99 \
    --spec-type draft-simple -v --port 8123

4.one greedy completion is enough to trigger it

curl http://127.0.0.1:8123/v1/completions -H "Content-Type: application/json" \
-d '{"prompt":"<any ~150-token prompt>","max_tokens":256,"temperature":0,"seed":42}'

5.Counting the DEBUG output:

• grep -c "CUDA graph warmup complete" ≈ 68
• grep -c "CUDA graph warmup reset" ≈ 68
  both counters grow in lockstep for the whole request and never settle(counts scale with request length; quoted values measured on master @ 4d91760)
  6.Control (no-speculation baseline): counters stay in single digits (complete=3, reset ≤ 2) and growth halts.
  7.Throughput observation: on this workload, speculation on the vanilla build ends up slower than no-spec (173.5 vs 188.7 t/s); with the key refinement it reaches 205.3
  8.Expected: each stable, alternating variant should converge to graph replay independently; in practice, every props flip resets the shared warmup state.

Route 2:

#in a llama.cpp checkout (any current master):
cmake -B build-repro -DGGML_CUDA=ON -DBUILD_SHARED_LIBS=OFF \
    -DCMAKE_CUDA_ARCHITECTURES=89 -DCMAKE_BUILD_TYPE=Release \
    -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_EXAMPLES=OFF -DLLAMA_BUILD_SERVER=OFF \
    -DGGML_BUILD_TESTS=OFF -DGGML_BUILD_EXAMPLES=OFF
cmake --build build-repro --target ggml -j
g++ -O2 -std=c++17 -I ggml/include -I ggml/src /path/to/mmid_repro.cpp -o mmid_repro \
    -Wl,--start-group build-repro/ggml/src/libggml.a \
    build-repro/ggml/src/ggml-cuda/libggml-cuda.a build-repro/ggml/src/libggml-cpu.a \
    build-repro/ggml/src/libggml-base.a -Wl,--end-group \
    -lcudart -lcublas -lcublasLt -lcuda -fopenmp -lpthread -lm -ldl
./mmid_repro

① gate-input shows that both bs=1 and bs=4 pass through gate #19645; ② with fixed shapes, each replays at (0.648/0.672 ms); under strict alternation, the two variants print the same graph key (key=0x723d93f8b4b0, but hashes differ), and the median latency collapses to 1.35/1.38 ms; ③ verdict: warmup_reset=62 / warmup_complete=63; after alternating stops, P5 recovers to 0.633 ms.

Relation to #28549

#28549 is an MTP-scenario, producer-side arena-separation issue. This issue is accompanied by a model-less MMID S1/S4 reproducer that triggers the same shared-state interference outside the MTP path,observable signals: same graph key across batch sizes, warmup_reset climbing during alternation (62 resets vs 63 completes), direct-eval latency collapse in the alternating phase, recovery once the load stabilizes.
The same backend mechanism reproduces independently outside of speculative decoding. The reproducer is single-file, modelless, deterministic, and uses only the public API; verified directly against the current master (e2d2c0d, 2026-09-09). The target op is GGML_OP_MUL_MAT_ID (MoE expert routing) — completely unrelated to the speculative-decoding implementation.

First Bad Commit

not a conventional regression, no single first-bad commit

#18934 introduced the current first-node pointer partitioning mechanism; #19645 empirically broadened the MMID exposure; #19754 changed the failure policy from permanent disable to recoverable warmup starvation,accordingly, the permanent-disable signatures in the attached historical log predate #19754 and no longer occur on current master; #18934 introduced the current first-node-keyed cache partitioning (the single-key state model it partitions predates it) #19645 is the exposure-broadener, not the introducer.

Relevant log output

Logs

Non-MTP MMID reproducer on current master (e2d2c0d, RTX 4060 Ti):

gate-input bs=1: node=node_0 op=MUL_MAT_ID src0_type=q8_0(8) quantized=1 node->ne[2]=1

gate-input bs=4: node=node_0 op=MUL_MAT_ID src0_type=q8_0(8) quantized=1 node->ne[2]=4

--- phase P1: fixed S1 (bs=1)         (24 calls) ---

  => median ms:  bs=1: 0.648 | distinct keys: 1

--- phase P2: fixed S4 (bs=4)         (24 calls) ---

  => median ms:  bs=4: 0.672 | distinct keys: 1

--- phase P4: strict S1,S4 x100       (200 calls) ---

  call   1 bs=1     0.610 ms  key=0x723d93f8b4b0  hash=ffda0f87358a9783

  call   2 bs=4     1.288 ms  key=0x723d93f8b4b0  hash=5702f00072d35383   # same key, different graph

  => median ms:  bs=1: 1.352  bs=4: 1.375 | distinct keys: 1

--- phase P5: recovery S1 (bs=1)      (24 calls) ---   # stable load -> recovers

  => median ms:  bs=1: 0.633 | distinct keys: 1

==== verdict ====

log counters: dis_too_many_updates=0 dis_unsupported_node_type=0 dis_gpu_arch=0 dis_other=0

              warmup_complete=63 warmup_reset=62



Logs

Real-model A/B (llama-server, Qwen2.5-1.5B + 0.5B draft, draft-simple, greedy seed 42; harness summary lines, -v):

== test A: baseline, no spec (steady shapes) ==
  [van-base] gen 188.71 t/s | 256 tokens   warmup complete=3 reset=2
  [fix-base] gen 179.97 t/s | 256 tokens   warmup complete=3 reset=1
== test B: spec decode draft-simple, interleaved x3 ==
  [van-spec1] gen 173.47 t/s               warmup complete=68 reset=68
  [fix-spec1] gen 202.41 t/s               warmup complete=7  reset=3
  [van-spec2] gen 175.87 t/s               warmup complete=68 reset=68
  [fix-spec2] gen 205.29 t/s               warmup complete=7  reset=3
  [van-spec3] gen 166.23 t/s               warmup complete=68 reset=68
  [fix-spec3] gen 206.04 t/s               warmup complete=7  reset=3
== test C: control, GGML_CUDA_DISABLE_GRAPHS=1 ==
  [van-ctl] gen 139.09 t/s                 warmup complete=0 reset=0
  [fix-ctl] gen 145.91 t/s                 warmup complete=0 reset=0
# appended paired run: 1024-token single stream, no spec, decode graph key constant
  [van-1024] gen 154.25 t/s                warmup complete=6 reset=5
  [fix-1024] gen 155.90 t/s                warmup complete=6 reset=4

Non-MTP MMID reproducer (same .cpp vs #19645 parent 667b694 and #19645 ad8207a, round 1):

$ ./mmid_A-667b6942
gate-input bs=1: node=node_0 op=MUL_MAT_ID src0_type=q8_0(8) quantized=1 node->ne[2]=1
gate-input bs=4: node=node_0 op=MUL_MAT_ID src0_type=q8_0(8) quantized=1 node->ne[2]=4
==== verdict ====
log counters: dis_too_many_updates=0 dis_unsupported_node_type=0 dis_gpu_arch=0 dis_other=0

$ ./mmid_B-ad8207af
gate-input bs=1: node=node_0 op=MUL_MAT_ID src0_type=q8_0(8) quantized=1 node->ne[2]=1
gate-input bs=4: node=node_0 op=MUL_MAT_ID src0_type=q8_0(8) quantized=1 node->ne[2]=4

--- phase P2: S1 replays  (median bs=1: 0.665 ms  bs=4: 0.834 ms | distinct keys: 1)
--- phase P4: strict S1,S4 x100
  call   1 bs=1     0.659 ms  key=0x70c874c324b0  hash=ffda0f87358a9783
  call   2 bs=4     0.891 ms  key=0x70c874c324b0  hash=5702f00072d35383   # same key, different graph
  call  24 bs=4     1.465 ms  key=0x70c874c324b0  hash=5702f00072d35383
  => median ms:  bs=1: 1.466  bs=4: 1.516 | hash: bs=1:stable(1) bs=4:stable(1) | distinct keys: 1
--- phase P5: recovery S1 (bs=1)   # pre-#19754: never recovers
  call   1 bs=1     1.408 ms  key=0x70c874c324b0  hash=ffda0f87358a9783
  call  24 bs=1     1.467 ms  key=0x70c874c324b0  hash=ffda0f87358a9783
==== verdict ====
log counters: dis_too_many_updates=1 dis_unsupported_node_type=0 dis_gpu_arch=0 dis_other=0