feat(kvcache): host-resident prefix tier for hybrid GDN models (snapshots + KV pages)

[alvarorsouza-arch]

Note: stacked on #499 (KV host-offload). The GDN-snapshot tier works standalone; the KV-page tier composes with the offloader.

Problem

On hybrid GDN models (qwen4_exp), prefix reuse needs two things alive at once: the KV pages in the radix tree and a GDN state snapshot at that point. The snapshot pool is small and global LRU. Any auxiliary call (WorkBuddy's title generator is our real-world case) drains the main conversation's snapshots between turns, and the match truncates to zero. Measured before the patch: agent turns with 20 to 150 s TTFT. After: stable single-digit seconds.

What the patch does

1. New hooks on HybridRadixCache: on_evict_node (fired before unlink/free) and on_match_dead_snapshot (fired during the match walk). With the tier on, an evicted leaf becomes a tombstone and its KV is no longer cascade-freed.
2. HostPrefixTier (new, kvcache/host_prefix_tier.py): a pinned host bank of GDN snapshots plus an index of cold prefixes with its own KV slot space. Scheduler logical page ids get reused, so content is copied D2H on eviction and never referenced by logical id afterwards.
3. Rehydration at admission: if the tree match falls short, we check the host index; on a hit we restore the GDN snapshot (H2D into a fresh slot) and bring the prefix back. With KVHostOffloader active the copy is host-to-host into its mirror, no bulk H2D; the offloader's lazy fetch covers residency.
4. Eviction copies run in 128-page slices. The first version staged the whole prefix on the GPU and OOMed exactly under memory pressure, which is when eviction runs.

Flags: FT_GDN_HOST_TIER=1, FT_GDN_HOST_SLOTS=32, FT_PREFIX_HOST=1, FT_PREFIX_HOST_GB=16. All off by default.

Measured numbers (RTX 4070 Ti SUPER 16 GB, 251 GB RAM, PCIe Gen4)

Churn test (24k-token conversation, 10 auxiliary calls in between, then back):

	cached-token	wall time
without the tier	0	16.5 s (full recompute)
with the tier	23,936	3.8 s

Total flush test (two ~130k-token prefills through a 200k pool; every page of the conversation leaves the GPU): the conversation comes back with cached-token 24,000 in 4.4 s.

Regressions: our quality suite (40 known-answer questions + 30 cloze) is identical to baseline (39/40 both). Decode steady at 33-37 t/s.

Known limitations

• Rehydrating a prefix larger than the physical pool requires the KVHostOffloader active; without it the conservative path applies.
• The host index is process memory; a server restart clears the tier.
• mr=5 breaks at CUDA graph capture on this build (bs=5 not captured); mr=4 is the safe ceiling. Details in a separate issue.

Files

• New: kvcache/host_prefix_tier.py, tests/kvcache/test_host_prefix_tier.py, tests/kvcache/test_tree_hooks.py, tests/kvcache/test_tier3.py
• Modified: kvcache/hybrid_radix_cache.py, scheduler/cache.py, engine/config.py

[YevheniiKotyrlo]

A measurement that bears on this tier: a client's compaction replays a long conversation up to the tail it keeps, then its instruction, and on a hybrid model that resumes only where a snapshot survives. Serving Qwen3.8-Flash-Next at a 1M window, a 760,534-token replay of a cached 985,577-token prompt prefilled everything again, 798.9 s, because no snapshot below its end existed: the final chunk's is the only one a prompt keeps.

Moving evicted snapshots to host memory, as this PR does, keeps the ones that were made; the replay also needs the intermediate chunks' snapshots to be made at all. #569 describes a patch that does both, keeping the residency on the radix node rather than in a second index, and what it measured (12.3 s for the same replay). I'd be glad to fold it into this work instead if that suits you better.