[wOvAN]

What

The second half of #917 — the shape of the computed graph, not just the size of the allocation.

VoxCPM2FeatureGeneratorRuntime built both MiniCPM step runtimes in its constructor with
config.max_length (8192 on the shipped q8_0 package) as cache_steps. cache_steps is a graph
dimension, not only an allocation bound:

attention_mask_ = ggml_new_tensor_4d(ctx, GGML_TYPE_F16, cache_steps_, 1, 1, 1);   // minicpm.cpp:1132
...
flash_attention_from_grouped_heads(ctx, q_heads, k_heads, v_heads, dim, attention_mask);  // :935

so every generated patch runs flash attention over all 8192 rows, of which a three-second phrase fills
~114, and 36 layers × 8192 × 256 × 4 B × 2 keys+values stay resident for the life of the runtime
(604 MiB of cache tensors, plus the same again in the per-layer host import scratch — the 1.21 GiB in
the issue). Nothing in the voxcpm2 option table reaches the capacity: it has mem_saver and
prompt_cache_slots, no cache key, so a user cannot shrink it from the CLI.

Fix

Size the step runtimes from the budget the request can fill, bucketed, and grow in place:

• engine::runtime::bucketed_cache_steps() is the bucketing higgs_audio_tts already had as a
  file-local helper (bucketed_initial_cache_steps), lifted into the framework header so both families
  share one copy. higgs' call site is unchanged in behaviour.
• generate_once() builds both step graphs at bucketed(prefill_rows, max_tokens) and rebuilds only
  when the bucket changes, so a fixed-shape workload keeps one pair of graphs. The AR loop doubles the
  bucket when it fills and carries the state through export_state/import_state; the last bucket is
  capped at the request's own prefill_rows + max_tokens, which is the quantity the existing
  CapacityError already compares against max_length.
• VoxCPM2MiniCPMStepRuntime gains cache_steps()/valid_steps() for that check.

Measured — CUDA (RTX 4070 Ti, sm_89, seed=1234, one process per run, device memory = nvidia-smi peak minus desktop baseline)

request	audio ms	wall ms	peak RSS MB	peak device MB
43 chars, 624cf918	3040	1010	1408	4227
43 chars, this	3040	1174	830	3077
214 chars, 624cf918	11360	3048	1407	4431
214 chars, this	11520	2601	832	4319
630 chars, 624cf918	35200	8254	1407	6661
630 chars, this	35360	7275	833	5446

Peak RSS −575 MB at every length; resident device memory −1.1 GiB on the requests that need a long
cache; the short request pays +164 ms (+16%) because the step graphs are now built on the first
request instead of at load (the med/long rows, where the per-step attention saving dominates the one
build, are −15% and −12% wall).

The render changes on long requests — disclosed, and isolated

The 43-character request is bit-identical (md5 00896604 in both builds). The 214- and 630-character
requests gain 4 patches (160 ms) and change md5.

That is not the growth path. With this same code and cache_steps pinned back to max_length, all
three md5s reproduce 624cf918 exactly (00896604 / c6e4c2b5 / 5cbd0784) — so lazy construction,
bucketing, and the export_state/import_state growth round-trip are bit-exact. The delta is the
attention reduction order: flash attention over 135 masked rows and over 8192 masked rows are not the
same floating-point sum, and voxcpm2's generation is stochastic (its durations already differ across
platforms at a fixed seed, unlike f5_tts). Rounding the buckets to powers of two (128/256/512) does
not restore parity — measured, same 4-patch delta — so this is inherent to shrinking a masked
static cache, not to the bucket shape.

If bit-parity on long requests is a hard requirement, the shape that keeps it is a session option that
defaults to max_length (the voxtral_realtime.stream_decode_cache_steps pattern) and opts into the
request-sized buckets — say so and I will re-cut it that way.

Validation

• Both builds are native trees at the same commit, model set voxcpm2,higgs_audio_tts, CUDA/sm_89;
  the higgs target compiles from the lifted helper (pure move, call site unchanged).
• bucketed_cache_steps gets unit coverage in the existing transformer_kv_ring_test (ctest), which
  already links engine_runtime: the exact/clamp cases a caller hits (114+128 → 242, 100+200 → 256, 200+8000 → 512, 8000+0 → 8000, 0+0 → 0) plus bounds and monotonicity in the request over
  a prompt × tokens grid. Passes on this tree (CUDA/sm_89 build, rc=0).
• Measurements are on base 624cf918; the branch is rebased onto current main with an identical
  patch-id, so the numbers describe the shipped diff.
• No ctest coverage exists for voxcpm2 (only the ENGINE_BUILD_WARMBENCH parity harness, which stores
  no md5 goldens), so the check here is the md5 ladder above.
• aarch64 (RK3588 Orange Pi 5 Plus, native build, --backend cpu --threads 4, same three texts, serial
  ladder with pgrep audiocpp_cli empty before every run):

request	audio ms	wall ms	peak RSS MB
43 chars, 624cf918	3840	54826	5041
43 chars, this	3840	30600	3294
214 chars, 624cf918	11520	103962	5855
214 chars, this	11520	95391	4157
630 chars, 624cf918	37920	312792	7053
630 chars, this	36160	274592	5433

Peak RSS −1.7 GiB at every length, wall −44%/−8%/−12%. The short request — the one that pays the
graph build — is 44% faster here, not 16% slower as on CUDA: on 8 A55/A76 cores the attention over
8192 masked rows costs far more than the one graph build it replaces. md5s match on the 43- and
214-character requests (e4598585, 9adec9ed); the 630-character request diverges in both directions
of the same effect (3a396c07 → ae564b26, 37920 → 36160 ms of audio), as on CUDA.

Review follow-up: the long-request drift, measured

Stand: audiocpp_server on an RTX 4070 Ti (CUDA, --threads 4), one live session, 8 requests in the
order med, long, med, med, short, long, short, short, fixed seed, base = 624cf918, head = this
branch.

request	base md5 (repeats)	this branch md5 (repeats)	base wall	this branch wall
43 chars	00896604 ×3	00896604 ×3	0.841 s	0.603 s
214 chars (cold, first)	c6e4c2b5	bae77d91	14.111 s	4.644 s
214 chars (warm)	c6e4c2b5 ×2	bae77d91 ×2	2.787 s	2.227 s
630 chars	5cbd0784 ×2	edf85d5a ×2	7.639 s	6.635 s

#921's failure mode is absent. Every repeat that follows a request of a different length — the
first request after the step caches grow into a larger bucket, and the first after they come back down
to a smaller one — is bit-identical to the warm repeats of the same text, on both sides. The failure
mode in #921 was a per-length prompt-prefill graph rebuild perturbing the draw; this change sizes
the decode-step graphs, and resizing them does not: on the base side, where the capacity never changes
at all, the cold first med request (14.111 s of graph building) and its warm repeats are the same md5,
and the same holds here.

WER does not move. All 16 clips transcribed with faster-whisper small.en (int8, CPU), scored
word-level against the input text:

request	base	this branch
43 chars (7 words)	0.0000 ×3	0.0000 ×3
214 chars (40)	0.0000 ×3	0.0000 ×3
630 chars (117)	0.0171 ×2	0.0171 ×2

The transcripts are word-for-word identical on both sides (118 hypothesis words against a 117-word
reference at the long length, on both sides). The long-request md5 change is the disclosed
summation-order effect of a wider attention graph — the audio differs, the words do not.

[0xShug0]

@wOvAN Thanks for the PR. I'll be AFK and will test the PR later today. Some drift in long outputs is fine, but you may want to check the WER to see if it's moving in the right direction. Meanwhile, could you check that it doesn't have the same failure mode as #921?

[wOvAN]

Both checked, added to the PR body under «Review follow-up».

The #921 failure mode is absent. One live audiocpp_server (RTX 4070 Ti, CUDA), 8 requests in the
order med, long, med, med, short, long, short, short, fixed seed, base 624cf918 vs this branch.
Every repeat that follows a request of a different length — the first request after the step caches
grow into a larger bucket, and the first after they come back down — is bit-identical to the warm
repeats of the same text (00896604 ×3, c6e4c2b5/bae77d91 ×3, 5cbd0784/edf85d5a ×2). #921's
mode was a per-length prompt-prefill rebuild; the decode-step graphs can be re-sized between
requests without moving the draw. Control from the base side, where capacity never changes at all: the
cold first 214-char request (14.111 s of graph building) and its warm repeats (2.787 s) share one md5.

WER does not move. All 16 clips through faster-whisper small.en (int8, CPU), word-level against
the input text: 0.0000 (7 words), 0.0000 (40), 0.0171 (117) — the same three numbers on both sides, and
the transcripts are word-for-word identical. So the long-request md5 change is the disclosed
summation-order effect of a wider attention graph: the audio differs, the words do not.

The stand also gave a server-side timing row the CLI stand in the body did not cover (one session,
warm requests): 43 chars 0.841 → 0.603 s, 214 chars 2.787 → 2.227 s, 630 chars 7.639 → 6.635 s, and
the cold first request 14.111 → 4.644 s.