[bong-water-water-bong]

HRX's GraphProgramCache kept every program it built and never evicted any. A long-running llama-server on HRX therefore grew GPU memory (GTT) without bound. Since engine ggml-org#270 made HRX the default GPU route, this blocks the release: on 2026-10-01 it ran strixhalo out of memory.

Cause

Every new graph shape builds a GraphProgram: each new prompt or ubatch length, and decode shapes as n_kv grows. Each program holds its HRX buffers, its recorded graph and a transient-arena binding.

With ZAYA1-8B that's about 34 programs per request at about 130 MiB each, so roughly +1 GiB per request when prompt lengths vary. Sixty requests of a single length level off at 4.35 GiB.

Fix

New file runtime/graph-program-cache-limit.{h,cpp}. After a lookup that built a new program, if the cache holds more than GGML_HRX_GRAPH_PROGRAM_CACHE programs (default 64; 0 = unbounded, the old behaviour), it:

1. waits for the stream with stream_synchronize_sleeping, so no command of a dropped program is still running;
2. marks the replay state synchronized;
3. drops every program except the current one, plus the prepared programs.

A lookup that reused a program, by uid or by structure, drops nothing, so steady decode never triggers a flush. Each flush logs one WARN line with a running count. (llama-server hides HRX's INFO logs.)

Hookups in AMD's files:

• GraphProgramCache::size() and retain_only(keep)
• one call in GraphExecutor::execute
• the CMake sources

Measured

ZAYA1-8B Q4_K_M, llama-server on HRX0, balanced power mode. 40 greedy requests of varying length (the same 20 lengths sent twice). GTT read from mem_info_gtt_used.

cap	max GTT growth	flushes	median request (new / repeated length)
0 (old)	24.6 GiB, watchdog stop at request 28	0	1.74 s / 1.12 s
64 (default)	8.7 GiB	7	1.84 s / 1.75 s
128	15.9 GiB	3	1.81 s / 1.75 s

• Greedy answers: identical to cap 0 at caps 4, 16 and 32 (23/23 each). Cap 4 flushes several times per request, including in the middle of a request.
• llama-bench: pp512/tg128 unchanged (cap 32 vs cap 0: pp512 1979/1888 vs 1943/1981 tok/s, tg128 93.4 vs 94.2). No flush during tg128.
• Qwen3.8-27B UD-Q4_K_XL with --spec-type draft-mtp -md mtp-Qwen3.8-27B-Q4_0.gguf at cap 16, 30 requests: 3.4 GiB, drafting active (draft_n 15–19).
• GGML_HRX_DISABLE_GRAPH_REPLAY=1 is not a workaround: it still grows to 24.6 GiB, and decode drops to 88.6 ± 10 tok/s.

Not in this PR

• A flush drops the warm working set too: a repeated length takes 1.75 s after a flush, against 1.12 s unbounded. Keeping the most-used programs would need recency tracking inside AMD's cache class.
• get_or_build compares each new uid against every cached program. With many cached programs that's a likely cause of a slow first decode after a prefill. The cap bounds this cost but doesn't remove it.

🤖 Generated with Claude Code

[bong-water-water-bong]

Review (PR-Agent duty): approve.

• The trim runs only after a lookup that built a program, and only over the cap. The current program is kept, and the stream is synchronised (sleeping wait) before retain_only + prepared_programs.clear().
• mark_stream_synchronized() runs the pending host writebacks before any program's buffers can be released, so nothing in flight references a dropped program.
• size() / retain_only() take the cache mutex; last_program_ is reset to a survivor.
• Hooks in AMD's files are minimal; the policy lives in our own file.
• Measurements cover what matters for the release: bounded GTT (24.6 -> 8.7 GiB on ZAYA), greedy answers identical at caps 4/16/32, no flush during tg128, and 27B --mtp drafting under cap 16.
• Known cost, noted in the PR: a repeated length after a flush is 1.75 s vs 1.12 s. Fine for now; an LRU inside the cache class can come later.

Merging after #62, once the hosted Linux/Windows jobs pass. The self-hosted cpu-/gpu- jobs never get a runner on this fork, and the branch is unprotected, so no admin merge is needed.