sanoTTS heart-nano: hand-written ARM NEON decoder, 1.4-2x faster on Raspberry Pi 400 — would you want it?

[tarelerulz]

Hi, and thanks for audio.cpp. I use sanoTTS heart-nano every day on a Raspberry Pi 400 (Cortex-A72, 4 GB, CPU only) and wanted to ask before sending code.

What I did

I wrote a hand-written ARM NEON version of heart-nano's decoder (the part after the tokens, up to and including the complex spectrum) and a small CLI around it. Everything else is audio.cpp's library unchanged: phonemes (eSpeak), sentence splitting, pauses, seed, FFT, WAV writing. Same model file, same command-line options as audiocpp_cli's tts call.

Decoder alone, 7 s sentence, 3 threads: 72 ms (audio.cpp) -> 32 ms.

Whole process, 3 cores, best of 3 (based on audio.cpp ff1bcc45, Sept 13):

text (audio length)	audiocpp_cli	NEON decoder
short greeting (2.8 s)	82 ms	59 ms
quotes (5.0 s)	120 ms	76 ms
one sentence (7.0 s)	153 ms	96 ms
numbers and dates (12.1 s)	236 ms	139 ms
6-sentence email (17.7 s)	316 ms	168 ms
run-on, no punctuation (20.9 s)	379 ms	205 ms
80-word paragraph (32.4 s)	563 ms	287 ms

Output is the same

On all 7 texts: same sample count; 99.68-99.76% of 16-bit samples identical, the rest differ by 1 step (float rounding); all word timings identical.

Where the speed comes from

1. 1x1 convolutions: inputs and weights repacked so the inner loop reads one sequential stream, 4x16 NEON register blocks.
2. The embedding convolutions merged into one pass.
3. All threads work on every layer with a barrier between layers (instead of splitting work across layers).
4. NEON GELU (erf approximation) and NEON exp/sin/cos for the complex spectrum.

The question

The code is plain C with NEON intrinsics, outside GGML, so it is ARM-only and specific to this one model. I know that may not fit how you want models implemented. Would any of these be welcome?

• an opt-in ARM CPU path for heart-nano, model-local, with the GGML path unchanged as the default;
• or just the ideas, applied to the existing GGML graph so every platform benefits (I can share the code as a reference);
• or neither, which is fine too.

If you're interested, I'd rebase on current main and re-run the timing and output checks before opening a PR.

(Not tested: GPU backends, x86, other sanoTTS voices. The GPU (Vulkan on the Pi's V3D) was slower than the CPU for this model in my tests.)

[0xShug0]

@tarelerulz Thanks for offering to contribute and for sharing the detailed benchmarks! I've been really into edge AI models lately. Just merged microWakeWord, sherpa kws, and ced. I definitely want to see audio.cpp run more efficiently on Raspberry Pi.

I'd suggest two PRs: one for heart-nano and another for GGML changes (which usually take longer to review and test). I'm extremely interested in building a small, reusable library in audio.cpp for ARM optimizations. Once I see the code, I can iterate on the right abstractions to make these optimizations easier to reuse across other models. Please enable maintainer edits on the PR so we can collaborate on the implementation.

[0xShug0]

@tarelerulz I'm really excited about this direction! I did some tests rewriting one component each in OpenWakeWord and Moonshine ASR, and saw a 2× speedup 😮 for OpenWakeWord and a 15% speedup for Moonshine ASR. I won't be able to do a full rewrite myself, but I'm working on a library and subsystem to make it easier. Are there any other models you'd recommend testing on the Pi (maybe one or two per task)? I'd love to see how much we can improve their performance with these ARM optimizations.

[tarelerulz]

@0xShug0 Those are great results, 2x on OpenWakeWord is a lot!

Here's what I actually use on the Pi every day:

• Speech-to-text: Cactus Whistle is my main dictation, running live so words show up while I talk. I'm also trying FUTO asr4all-s right now.
• Text-to-speech: heart-nano, now with the new NEON decoder from feat(sanotts): opt-in ARM NEON CPU decoder for nano voices (sanotts.cpu_decoder=neon) #860.

Beyond those, please use the faster ARM code on any model where you think it will help. You know the code base better than I do.

I think this is all great. It's really exciting to see audio.cpp getting faster on small ARM boards like the Pi, and it's been a pleasure working with you on it. Thank you!

[0xShug0]

Thanks!