Repository navigation
voxcpm2 decode path: attention runs over the full max_length KV cache every token, plus ~1.2 GB of per-request cache zero-fills #917
Description
Environment: Jetson Xavier NX (Volta sm_72), CUDA backend, voxcpm2 Q4_K GGUF from audio-cpp/audio.cpp-gguf, audio.cpp main as of 2026-10-14. The numbers below are from that board, but the structure costs are platform-independent.
While profiling a warm short TTS request (2.7 s of audio, ~17 generated patches), two things dominated the decode path:
-
Attention cost scales with max_length, not with the sequence.**
VoxCPM2MiniCPMStepRuntimebuilds both KV caches atconfig.max_length(8192) as f32 (src/models/voxcpm2/minicpm.cpp). Every token, per layer, the full cache is transposed and materialized (ggml_contviaensure_backend_addressable_layout), then ggml's fattn vec path converts the whole f32 K/V to f16 internally (fattn.cu accepts f32 K/V only via conversion), then attention runs over all 8192 rows with a mask. By my count that is ~67 MB of memory traffic per layer-token, across 36 layers (28 base + 8 residual) = ~2.4 GB per token. On a 59 GB/s bus that is ~40 ms/token, versus under 2 ms/token if the cache were sized to the real sequence (~300 rows) or stored f16. -
Per-request KV zero-fills.**
TransformerKVCache::import_statezero-fills and uploads the entire capacity onreset()and again on prefill import (src/framework/runtime/kv_cache.cpp), both LMs, every request. That is ~1.2 GB of memset + copies per request for what is usually a few hundred valid rows. On a small board this also becomes the allocation spike that triggers the OOM killer with other services resident.
Smaller items measured along the way: text embedding runs one graph compute + sync per prompt token (embed_token in a loop); the between-token projection call computes and downloads 8 outputs and uses 2, the per-patch call uses 3; the DiT estimator re-uploads the constant mu/cond conditioning every diffusion step; the AudioVAE snake activation materializes a full-size ggml_repeat of alpha per call.
I fixed all of the above in my tree: step caches sized to prefill+max_tokens (256-step buckets), zero-once-at-build plus prefix-only import, batched embedding, projection graphs split into emit/advance subsets, DiT conditioning uploaded once per patch, broadcast snake ops. Measured on the same box and protocol: warm short reply 5.29 s -> 3.28 s, 15.2 s paragraph 25.4 s -> 16.6 s (RTF 1.67 -> 1.09), KV allocation ~605 MB -> ~20-38 MB per request, cold start ~29 s -> 17.5 s. Output is bit-identical before/after (same fixed seed, per-sample diff on multiple texts).
The patch series applies cleanly to current main (verified against a fresh main tarball this week). Happy to open a PR if you want it, split whichever way is easiest to review.
Activity
wOvAN commentedon Oct 15, 2026
Independently confirmed on x86_64 CPU (4 threads) and RK3588 CPU, from the shipped
q8_0package's embeddedconfig.json(max_length: 8192,num_hidden_layers: 28,residual_lm_num_layers: 8,num_key_value_heads: 2,kv_channels: 128→step_elems = 256):1. Cache sized at
max_length, in f32.VoxCPM2MiniCPMStepRuntimeis constructed withassets_->config.max_lengthfor both LMs (src/models/voxcpm2/generator.cpp:1213-1218), and the per-layer cache tensors areGGML_TYPE_F32at{1, cache_steps_, num_key_value_heads, dim}(src/models/voxcpm2/minicpm.cpp:1143-1151). So the resident KV is36 × 2 × 8192 × 256 × 4 B = 604 MB— matching your ~605 MB.2. Full-capacity zero-fills.
TransformerKVCache's constructor preallocates and zeroesstd::vector<float>(cache_elems, 0.0F)per key and per value (src/framework/runtime/kv_cache.cpp:112-115), andimport_stateresizes the scratch tocache_elemsandstd::fills both to zero before copying only thekeep_elemsprefix, then writes the whole tensor (:164-167,:183-184). voxcpm2 is one of the few families that does not setlazy_import_scratch(comparesrc/models/higgs_audio_tts/ar.cpp:483), so the 604 MB of host scratch is persistent per process, andreset()(minicpm.cpp:1096,step_cache_ = TransformerKVCache()) rebuilds and re-zeroes it. Per request that is ≥1.2 GB of fill/copy traffic for a reply whose valid prefix is a few hundred rows. Nothing in the option table exposes the capacity (src/models/voxcpm2/loader.cpp:38-70has no key for it;max_lengthcomes only from the embeddedconfig.json), so a caller cannot shrink it from the CLI.Measured, host x86_64,
--backend cpu --threads 4, 9.44 s reply = 236 latent frames at 25 Hz (decoder_ratesproduct 1920 atout_sample_rate 48000) = 59 patches atpatch_size: 4:generator_msis one key over base-LM step + residual-LM steps + CFM + local encoder, so the attention share is not separable from these metrics — your own split is the one that isolates it.One nuance for the CPU case, in case it changes how you split the series: on x86_64 the zero-fill traffic is ~1.8 GB per request, which is ~0.1-0.2 s of wall time against 50 s of request — i.e. on CPU the win from (2) is the 1.2 GB resident plus the transient spike (your OOM-killer point), not the memset time. The wall-time win is (1).
Board RK3588 (Orange Pi 5 Plus, 4×A76+4×A55), native aarch64 build, same protocol (
--backend cpu --threads 4 --request-option seed=1234), same package:So on a board the AR path is not "a bit slow", it is the whole request: 84% of wall in
generator_ms, and a quarter of the memory is the cache sized at 8192. Both of your fixes attack exactly that share.Not posting this as a competing patch — the structural fix is yours. Just confirming the two costs reproduce off-CUDA and off-Jetson, and that the arithmetic in the issue matches the tree line by line.
0xShug0 commentedon Oct 15, 2026
@sander110419 Thanks for identifying these issues! A PR would be very welcome, and I look forward to it.
@wOvAN Thanks for testing and helping validate the issues!