Model request: Haru-4B — native full-duplex speech-to-speech #1278

Could we add support for Haru-4B? It’s an Apache-2.0 spoken-dialogue model that listens and speaks simultaneously, rather than chaining separate STT, LLM and TTS models.

I’ve run the upstream PyTorch demo locally on an RTX 4060 Ti 16 GB. A native C++/ggml implementation would simplify deployment, with potential for lower memory use through validated quantization.

Model overview

• Backbone: Qwen3.5-4B, 32 layers, hidden size 2560.
• Streaming: 7.5 Hz—one 3,200-sample frame every ~133 ms at 24 kHz.
• Codec: causal VibeVoice acoustic tokenizer, with continuous 64-dimensional audio latents.
• Audio generation: four-layer MeanFlow MLP head with one sampling step; no discrete audio tokens or separate audio transformer.
• Conditioning: character card plus a reference voice recording.
• Languages: English, Spanish, Portuguese, German, French, Chinese, Japanese, Russian and Korean.
• License: code and weights are Apache-2.0.

Haru is a pretrained research base model, not an instruction-tuned assistant.

Requested support

The main goal is a persistent full-duplex session: feed microphone frames continuously while receiving generated audio and text, preserving conversation and codec state. A turn-based S2S wrapper alone would miss the model’s main capability.

A staged port could start with offline, frame-by-frame inference and parity checks against upstream, followed by live streaming and CLI/server integration. Reference-voice and character-card conditioning would be useful from the outset; quantization could follow once baseline parity is established.

There may be reusable pieces in audio.cpp’s existing Qwen3.5 decoder runtime and VibeVoice implementation, though compatibility would need checking rather than assuming this is a drop-in variant.

Would this be a suitable community-model port under the project’s model-spec-v1 guidelines?

Upstream references

• Code, architecture and usage
• Weights and model card
• Technical write-up