[Fishballllllll]

How this was made: most of the code and the investigation behind this PR were done by an AI coding assistant (Claude Code), with me directing the work and checking the results. I validated it as described below, but it may still contain mistakes that neither of us caught, so please review it with that in mind. The speedup is real on my machine; I'm happy to rerun anything, share the scripts and logs, or rework the change.

Summary

• New session option breeze_tts.perf_mode=off|cuda_perf (default off), following the Drama3 / MiniMax Music3 pattern, with three individual options it presets: breeze_tts.fused_depth, breeze_tts.direct_prefill and breeze_tts.skip_cache_round (all default false). Explicit options override the preset; other backends ignore them.
• With the default off, BreezeTTS runs the same code path as on main: output is byte-identical, and speed and peak VRAM match main.
• With cuda_perf, CUDA output is also byte-identical to main for the same seed, peak VRAM is no higher than main's, and offline RTF is 1.82x faster at CFG 4 (0.802 → 0.442) and 1.70x faster at CFG 1 (0.577 → 0.340).

Problem

On CUDA, most of a BreezeTTS frame was spent outside the actual GPU work:

• The depth decoder ran the 15 residual codebooks of each frame as separate graph launches. After every codebook the logits went back to the host for sampling and the next embedding came back up. Because its cache offsets kept changing, ggml-cuda could not replay a captured CUDA graph for it.
• Offline generation prefilled the backbone, exported the KV state to host floats and imported it back into the decode cache. In my timing logs, starting the decode from that state (breeze_tts.ar.start_decode_ms) took 460–680 ms per request. The streaming path has prefilled straight into the decode cache since Improve Qwen3 and Breeze TTS streaming and Breeze longform voice consistency #858.

Change

All model code is in src/models/breeze_tts/ (generator.cpp, session.cpp), plus include/engine/models/breeze_tts/generator.h, model_specs/breeze_tts.json and docs/models/breeze_tts.md. Each of the three options below is off unless set explicitly or through cuda_perf, and takes effect on CUDA only.

breeze_tts.fused_depth: BreezeFusedDepthRuntime builds the whole depth loop of a frame as one fixed-shape ggml graph. The K/V of earlier codebooks are carried inside the graph, and the CFG mix, temperature, top-k/top-p and sampling run on the device. Each frame is one graph launch plus a download of 15 tokens. It matches the per-codebook path exactly:

• Codebooks after the first attend over the in-graph K/V with the same flash attention the per-codebook path uses for cached decoding (DecoderPrefixAttentionMode::FlashWithPrefix).
• Sampling uses the per-codebook sampler's own torch-CUDA exponential draws. The host computes them with torch_cuda_tensor_iterator_exponential_element_at_offset at the same seed and RNG offsets, uploads -log(E) per token, and the graph picks argmax(logits / T - log E). The torch RNG offsets advance exactly as before.
• When the fused graph is enabled, the per-codebook depth runtimes are not created, and the graph embeds tokens from the backbone's own token-embedding tensor when it holds the exact F32/F16/BF16 values (quantized tables still get an F32 copy).

breeze_tts.direct_prefill: offline generation prefills into the decode cache, as streaming already does. This is the same computation without the host round trip. With it off, offline prefill goes through the same export/import calls as on main.

breeze_tts.skip_cache_round: with the bf16 reference path, an F16 cache and flash attention, K/V rows are already bf16-rounded before they are written, and an F16 cache stores those values exactly. Re-rounding the whole cache after each update is therefore an identity, and this option skips it.

What changed since your review

Thanks for testing it. Both trade-offs you saw were real, and I could reproduce them once I measured like you did: at the default guidance_scale 1, and with peak VRAM sampled finely (my earlier 200 ms nvidia-smi sampling missed the peaks).

• Defaults: everything this PR adds is now behind perf_mode, off by default. That includes direct prefill and the cache-round skip, which were unconditional in the first revision (and direct prefill applied to every backend); they are now separate options, CUDA only.
• Recognition accuracy: two differences in the fused graph made its tokens diverge from the per-codebook path. Its later codebooks used eager attention instead of the per-codebook path's flash attention, and it sampled with its own std::mt19937 Gumbel noise instead of the torch-CUDA stream. Both are fixed as described above. cuda_perf now reproduces main byte for byte, so accuracy is unchanged by construction.
• Peak VRAM: the original revision measured +13.5% over main on the longest requests. About half came from keeping the decode graphs resident between requests. That saved at most ~2.5% RTF, so I removed it: graphs are released after each request, as on main. The other half came from the fused graph itself: a second 256 MB F32 copy of the audio embedding table, and the per-codebook depth runtimes that were still created alongside it. Both are gone. Peak VRAM with cuda_perf is now within 0.4% of main on the longest requests and below main on the 10-request set.

Validation

Setup: RTX 5090 Laptop GPU (24 GB, sm_120, driver 616.92), WSL2 Ubuntu 24.04, CUDA 12.8, conda-forge g++ 13.4. main (2f6b42d) and this branch (based on efdf97e) built with the same compiler and flags. The two commits between them (#1085, #1086) do not change BreezeTTS: #1085 adds a KV-cache import option that only VoxCPM2 enables.

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=120 \
  -DENGINE_ENABLE_CUDA=ON -DAUDIOCPP_MODEL_SET=custom -DAUDIOCPP_MODELS=breeze_tts
ninja -C build audiocpp_cli

Requests: audiocpp_cli --task tts --family breeze_tts --backend cuda --request-sequence <file> --metrics, one warm-up request excluded. The 10-request set is Mandarin instruction-controlled voice cloning (one 6 s reference clip, seed 1234, default sampling, guidance_scale 1 or 4). The long set is the 8 longest of a 47-request set (32–93 characters, guidance_scale 4). Warm RTF is sum(wall_ms) / sum(audio_duration_ms). Peak VRAM is the device's used memory sampled through NVML about every 0.2 ms over the whole process, including loading and warm-up.

Speed, peak VRAM and byte identity (Q8_0, 10 requests, all runs back to back)

	main	this PR, off (default)	this PR, cuda_perf
CFG 4, warm RTF	0.802	0.805	0.442 (1.82x)
CFG 4, peak VRAM	5817 MiB	5827 MiB	5161 MiB
CFG 4, byte-identical to main		10/10	10/10
CFG 1, warm RTF	0.577	0.602	0.340 (1.70x)
CFG 1, peak VRAM	5195 MiB	5199 MiB	5181 MiB
CFG 1, byte-identical to main		10/10	10/10

off runs the same code as main; the CFG 1 difference is within this laptop's run-to-run variation (main itself measured 0.572–0.594 at CFG 1 across runs the same day).

Further byte-identity checks with cuda_perf

These were run on earlier commits of this branch that compute the same thing with cuda_perf (the later commits only release the decode graphs after each request and move direct prefill and the cache-round skip behind options):

• BF16 package, 10 requests at CFG 1 and at CFG 4: 10/10 each.
• Q8_0, the full 47-request set at CFG 4: 47/47.
• Q8_0, the long set: 8/8 (one commit before the last).

Peak VRAM on the long set (Q8_0, CFG 4)

	Peak VRAM	vs main
main	6235 MiB
original revision of this PR	7075 MiB	+13.5%
this PR, cuda_perf	6257 MiB	+0.4%

Not tested

• Only one machine (RTX 5090 Laptop, WSL2). I could not test other NVIDIA GPUs, and I'd be grateful if you could rerun it on yours.
• HIP, Vulkan, Metal and CPU take none of the new paths (the options are CUDA only), so they run the same code as main; I did not rebuild or run them for this revision.
• MUSA counts as BackendType::Cuda in audio.cpp, so the options would also take effect there, but I have no MUSA device.
• Not tested: the Q4_0 package, multi-chunk longform text, concurrent server requests, streaming with cuda_perf, and Windows builds.
• This PR adds no unit test. On this revision I ran the existing tests that touch the BreezeTTS spec and options, model_spec_system_test and attention_fallback_test, in the CUDA build, and both pass; I did not run the full ctest.

🤖 Generated with Claude Code

[0xShug0]

@Fishballllllll Thanks for the PR! I'm actually backporting optimizations right now: VoxCPM2 (#1047), Qwen3-TTS (#1055), YuE2 (#1056), Drama3 (#1068), ACE-Step (#1073), MiniMax Music3 (#1077).

Please follow the same patterns and add an option breeze_tts.fused_depth, default false. You can also add perf_mode with a cuda_perf preset that enables fused_depth in this PR. In my tests, the tradeoff includes both recognition accuracy and peak VRAM usage (>12%), so I'd keep it disabled by default.

Changing defaults can unexpectedly break configurations that currently work. Main's defaults are at least compatibility baseline. I've changed optimization defaults before and quickly received regression reports (#1016 #1071).

[Fishballllllll]

@0xShug0 Thanks for testing it and for pointing me to the other cuda_perf PRs. I've updated this PR to follow the same pattern:

• breeze_tts.perf_mode=off|cuda_perf, default off. Everything this PR adds is now behind it: fused_depth, plus direct_prefill and skip_cache_round for the two smaller changes that used to be always on. With off, BreezeTTS runs exactly the code path it runs on main (byte-identical output, same speed), on every backend.
• The two trade-offs you saw were real, and I reproduced both once I measured at the default guidance_scale 1 with fine-grained VRAM sampling:
  • Accuracy: the fused graph used eager attention for the later codebooks and its own Gumbel noise. It now uses the same flash attention and the same torch-CUDA random draws as the per-codebook path, so cuda_perf is byte-identical to main for the same seed (Q8_0 and BF16, CFG 1 and 4).
  • Peak VRAM: the original revision was +13.5% on my longest requests. Resident decode graphs (removed) and a duplicate embedding table plus unused per-codebook runtimes (removed) accounted for it. Now +0.4% on the same requests, and at or below main on the 10-request set.
• Speed with cuda_perf on my RTX 5090 Laptop: 0.802 → 0.442 RTF at CFG 4 (1.82x), 0.577 → 0.340 at CFG 1 (1.70x).

The PR description has the full numbers and how I measured them. To try it: --session-option breeze_tts.perf_mode=cuda_perf.

If it's useful, I can also mention cuda_perf in the BreezeTTS Quick Start, but I'm just as happy to leave that until it has been tried on other GPUs.

[0xShug0]

@Fishballllllll I've decided to include all three of your PRs in the next release, so if any issues come up, I'll just fix them directly.

[0xShug0]

@Fishballllllll Thanks! PR merged. The updated PR works great! Since the parity and VRAM concerns have been addressed, I’ll run more WER checks and may enable it by default before the release.

[Fishballllllll]

@0xShug0 Thanks so much for merging it and for considering it as a default! And thanks for building audio.cpp and for the careful review. Working on this PR taught me a lot. I’m just getting started with open source and GitHub, so this was a really valuable first experience.

If your WER checks or any other testing turn up a problem, just ping me and I’m happy to fix it. Beyond these PRs, I’d also love to help with other work on the project to keep learning, so if there’s anything you’d like a hand with, feel free to point me to it.

I also have a few more ideas for BreezeTTS that might make cuda_perf another 20–30% faster (a rough, untested estimate). If you think it’s worth it, I can work on them over the next few days and send a follow-up PR, in time for this release if it hasn’t gone out yet, or for the next one otherwise.

I’m also rebasing #1091 and #1092 onto the latest main to resolve the conflicts, and will push the updated branches shortly.

[0xShug0]

@Fishballllllll You've done a great job, especially for someone just getting started with open source. Really glad to have you contributing to audio.cpp!

The BreezeTTS optimization ideas sound interesting. Looking forward to the follow-up PR! No rush for the next release, which is planned for tomorrow... Depending on the scope of the changes, we can work toward a more systematic way to integrate optimizations that aren’t model-local. For example, we introduced exec_plan in the last release to organize backend- and hardware-specific execution paths. #922. It can help if you want to bypass GGML and implement a component directly in CUDA .cu.

[Fishballllllll]

@0xShug0 Thank you, that really means a lot! I’ll take a look at #922 and see how the follow-up could fit into it.

I’m a bit busy with school right now, so the next PR might take a little while, but I’ll work on it when I can and keep you posted.

Thanks again!