[Ryu1845]

Summary

Adds zonos3, a community port of Zyphra's ZONOS3-Mini TTS model. It is ported from the native ggml runtime in Zyphra/ZONOS3 (servers/ggml, MIT) and built on audio.cpp framework modules.

• Backbone (backbone.cpp): 11 layers at width 1024. Two layers use global GQA attention and nine use Gated DeltaNet. Every layer is joined by Full AttnRes. Layer 0 has a dense SwiGLU FFN; layers 1-10 use a latent top-2-of-64 MoE. Output is 10 delayed 4096-entry codebooks at 15 Hz, with a softcap of 15.
• Generation (prompt.cpp, sampler.cpp, generator.cpp): the prompt layout, the per-codebook sampler (repetition penalty, temperature, top-k, top-p, min-p) and the delayed-codebook loop with the speaker-CFG twin.
• Decoder (codec_decoder.cpp): the KEEP-2 MR-iSTFT decoder with a host iSTFT/resampler tail, producing 44.1 kHz audio. It streams incrementally.
• Speaker encoder (speaker_encoder.cpp): MECT-B2, which produces a 192-d embedding. It uses the framework Kaldi fbank and resampler.
• Raw-reference encoder (reference_encoder.cpp): the ZAC encoder with a w2v-BERT teacher, which turns a clip into reference codes.
• Session and loader: offline and streaming TTS and cloning, voice enrollment to .npy, condition buckets, speaker CFG, long-form chunking, and an LRU voice cache.
• Text: the SuperBPE tokenizer runs through the framework LlamaBpeTokenizer. Text normalization uses zyphra-textnorm and is opt-in (AUDIOCPP_ZONOS3_TEXTNORM).

MoE routing and expert compute run through MoeModule, and the optional expert cache (zonos3.disk_experts) runs through DiskExpertWeightStore (decode) and StagedMoeRuntime (prefill), as requested in #1231 (reply in thread).

Framework change

MoeScoring::SelectedSoftmax (commit framework: MoE routing can weigh selected experts by the softmax of their logits). ZONOS3 ranks experts by raw router logits plus a selection bias, then weighs the selected experts by the softmax of their logits. That is top-k first, then softmax. The existing modes apply softmax or sigmoid before ranking, so they weigh experts differently. The new mode is an additional enum value: existing callers are unchanged. moe_module_test covers it against the host reference:

$ ctest --test-dir build/zonos3 -R moe_module_test
1/1 Test #54: moe_module_test ..................   Passed    0.97 sec

The branch is rebased on eeeca9e8 (#1270, MoE hybrid CPU/GPU expert execution). On the rebased tree, prepared_graph_test, disk_expert_weight_store_test and moe_module_test pass, and the zonos3 parity numbers are unchanged. zonos3 does not opt into the new hybrid placement.

Build

cmake -S . -B build/zonos3 -DCMAKE_BUILD_TYPE=Release -DAUDIOCPP_MODEL_SET=custom -DAUDIOCPP_MODELS=zonos3 \
  -DENGINE_ENABLE_OPENMP=OFF -DGGML_OPENMP=OFF            # macOS / Apple clang
cmake --build build/zonos3 -j 8 --target audiocpp_cli audiocpp_server
# optional text normalization (needs cargo):
#   -DAUDIOCPP_ZONOS3_TEXTNORM=ON -DAUDIOCPP_ZONOS3_TEXTNORM_DIR=<zyphra-textnorm v0.1.0 checkout>

Run

M=models/ZONOS3-Mini-GGUF/backbone-ptq.gguf
build/zonos3/bin/audiocpp_cli --task tts  --family zonos3 --model $M --text "Hello." --out out.wav
build/zonos3/bin/audiocpp_cli --task clon --family zonos3 --model $M --text "Hello." --voice-ref voice.wav --out clone.wav
build/zonos3/bin/audiocpp_cli --task clon --family zonos3 --model $M --text "Hello." --voice-ref voice.wav \
  --request-option speaker_mode=reference --out clone_ref.wav
build/zonos3/bin/audiocpp_cli --task tts  --family zonos3 --model $M --mode streaming --text "Hello." --out stream.wav

Models

Package	Files (from Zyphra/ZONOS3-Mini gguf/)	Size
zonos3_mini_ptq (default)	backbone-ptq.gguf, zac-keep2.gguf, spk-encoder.gguf	1.16 GB
zonos3_mini_ptq_raw_reference	+ zac-encoder.gguf, ssl-teacher.gguf	6.74 GB

Parity

These run against outputs of the ZONOS3 native tools (tests/community_models/zonos3/make_oracle.sh). They are registered as CTest tests when AUDIOCPP_ZONOS3_MODEL_DIR and AUDIOCPP_ZONOS3_PARITY_DIR are set:

$ bash tests/community_models/zonos3/make_oracle.sh ../ZONOS3/servers/ggml/build models/ZONOS3-Mini-GGUF \
    ../ZONOS3/servers/ggml/gguf/voices/Cillian_en-Irish_M_1.wav /tmp/zonos3-oracle
$ cmake -S . -B build/zonos3 ... -DENGINE_BUILD_TESTS=ON -DENGINE_BUILD_MODEL_TESTS=ON \
    -DAUDIOCPP_ZONOS3_BACKBONE=$PWD/models/ZONOS3-Mini-GGUF/backbone-ptq.gguf \
    -DAUDIOCPP_ZONOS3_MODEL_DIR=$PWD/models/ZONOS3-Mini-GGUF -DAUDIOCPP_ZONOS3_PARITY_DIR=/tmp/zonos3-oracle
$ ctest --test-dir build/zonos3 -R '^(zonos3_|moe_module_test$)'
zonos3 backbone parity: PASS cos_min=1.000000 argmax=1.0000 steps=100
zonos3 backbone parity (disk experts): PASS max_diff_vs_resident_plain=0 oracle_cos_min=0.997770 oracle_argmax=0.9320 steps=100
zonos3 speaker parity: PASS cos=1.000000
zonos3 decoder parity: PASS snr=113.81 seam_max=0 samples=261660 oracle_samples=261660 frames=92
zonos3 reference parity: PASS cb0=1.0000 all=1.0000 frames=145 oracle_frames=145 seconds=328.1

• Backbone: teacher-forced over 100 steps against zonos3-cli --dump-logits. The pass threshold is cosine ≥ 0.9999 and argmax ≥ 0.99.
• Backbone, disk experts: the expert cache must produce bit-identical logits to resident experts run with the same plain kernels. The resident CPU path uses ggml's repacked Q4_0/Q8_0 kernels, as ZONOS3 does. Partial expert uploads can't go into repacked buffers, so the cache cannot use them, and its oracle numbers are shown for reference only.
• Speaker: cosine ≥ 0.999.
• Decoder: SNR ≥ 40 dB, same length, and chunked decoding identical to a single pass.
• Reference: same frame count, with ≥ 95% of codebook-0 codes and ≥ 90% of all codes identical. All 1450 codes matched.
• Text and tokenizer: zonos3_text_test covers language codes, .npy voice files, line-break collapse and phoneme-marker rejection. zonos3_tokenizer_test checks token ids against ZONOS3's tokenizer fixture.

Quality

Qwen3-ASR-1.7B on the generated audio: 4 English prompts × seeds {1, 2}, CPU. Text normalization was off. Speaker cosine is measured with the MECT-B2 encoder against the reference clip.

	ZONOS3 zonos3-cli	audio.cpp
WER, TTS	3.08%	3.08%
WER, clone (speaker embedding)	3.08%	3.08%
Speaker cosine, clone	0.815	0.835
WER, TTS, Metal	-	1.54%
WER, TTS, Metal, disk_experts	-	1.54%

Every error is on the prompt containing "7" and "3": the ASR transcribed them as "seven" and "three", which the scorer counts as different words.

• Long form: a 1,235-character paragraph with text_chunk_size=300 produced 6 chunks and 73.9 s of audio, with reached_limit:false and 0.45% WER.
• Same output as ZONOS3: on the CPU, with identical inputs and seed, TTS, speaker-embedding cloning, reference_codes_file, cfg_scale=1.5 and condition_buckets=4,10,30 all produce the same audio as zonos3-cli: same length, max |diff| 3.05e-5 (1 LSB). Streaming output is bit-identical to offline output.
• Text normalization: with AUDIOCPP_ZONOS3_TEXTNORM=ON, "It costs $42." comes back from ASR as "It costs forty-two dollars."

Performance

Apple M3, 8 CPU threads, 27-word sentence. Wall time is the session wall time after load, and clone rows include encoding a 10-second --voice-ref clip.

Backend	Mode	Wall	Audio	RTF	Peak RSS
CPU	TTS	2.67 s	8.87 s	0.30	2.06 GiB
CPU	Clone	5.87 s	7.00 s	0.84	2.05 GiB
CPU	TTS, disk_experts	3.38 s	8.80 s	0.38	1.81 GiB
Metal	TTS	0.97 s	9.07 s	0.11	1.71 GiB
Metal	Clone	4.43 s	6.93 s	0.64	1.71 GiB
Metal	TTS, disk_experts	1.58 s	8.67 s	0.18	1.42 GiB

• Speaker encoder: 10-second clip in 3.2 s; ZONOS3's spk-encoder-cli takes 3.38 s.
• Long-lived server (CPU, audiocpp_server, streaming mode): 5 POST /v1/audio/speech requests (3 plain, 2 voice_ref, one of them SSE/PCM) all returned 200. The SSE response had 7 speech.audio.delta events before speech.audio.done. The second voice_ref request took 6.6 s instead of 23.4 s because the voice cache skipped the encoder. Peak RSS was 1.29 GiB (/usr/bin/time -l, after Ctrl-C).
• Expert cache: the plain CPU run with expert_cache_size=8 logged disk_experts.expert_loads 1530 and disk_experts.cache_hits 949.

Backends tested

• CPU (Apple M3, 8 threads): everything above.
• Metal (Apple M3): TTS, speaker-embedding cloning (the encoders stay on CPU by default) and disk_experts all work. WER on the 8 plain takes is 1.54% both resident and with disk_experts; the performance table is above. Raw-reference encoding has not been run on Metal.
• CUDA and Vulkan: not tested; no hardware here.

Known limitations

• Not implemented: phoneme input, voice-design steering, accents, emotion directions, LACI and speech editing. These need extra GGUFs (steer.gguf, accents.gguf, backbone-ptq-edit.gguf) or Python workers.
• Expert cache on CPU: its takes differ from the resident CPU path's, as described under Parity. The minimum per-step logit cosine against ZONOS3 is 0.99777 and argmax agreement is 93.2%.
• Raw-reference encoder: a 10-second clip uses about 1,740 CPU-seconds (226 s wall on an 8-thread M3), the same as ZONOS3's zac-encoder-cli (1,738 CPU-seconds), and peaks at about 3.4 GiB RSS. It is released after each encode by default. On non-CPU backends its Snake activation falls back to sin/mul ops, and that path is untested.
• Untested: CUDA and Vulkan.
• WebUI: the catalog is not updated in this PR.

License

The weights are Apache-2.0 according to the model card. The runtime code is ported from Zyphra/ZONOS3 (MIT). zyphra-textnorm (MIT) is linked only with AUDIOCPP_ZONOS3_TEXTNORM=ON. A row is added to docs/model_licenses.md.

[0xShug0]

@Ryu1845 Thanks for the PR! I tested the PR on CUDA and Vulkan, but both failed to run:

Failure	Suggested fix
CUDA/Vulkan abort during TTS: unsupported strided sigmoid/SiLU operations terminate the server.	Use framework SigmoidModule and SiluModule for layout handling.
Vulkan abort during embedding lookup: unaligned index views trigger an assertion and terminate the server.	Materialize index views with ensure_backend_addressable_layout() before ggml_get_rows.
CUDA abort during raw-reference encoding: the SSL teacher’s 31-tap SSM_CONV is unsupported.	Replace it with existing DepthwiseConv1dModule, preserving BF16 rounding.
CUDA raw-reference encoding fails with an invalid launch configuration: padding a five-second reference exceeds the kernel’s grid-Y limit.	Transpose before padding so time occupies the contiguous dimension, then restore layout.

For an initial model port PR, the expectation is simply that it works correctly end to end across backends. There are some optimization opportunities (e.g., using framework modules instead of raw GGML ops can often improve performance), but those can be addressed in follow-up PRs.

[Ryu1845]

Thank you for the review I'll try to get this fixed ASAP

[Ryu1845]

OK this should be fixed

[0xShug0]

@Ryu1845 Thanks for the update! I'll try to review the PR later today.