Repository navigation
Irodori-TTS support #6
Description
Irodori-TTS is a Japanese TTS model based on EchoTTS, but it uses Facebook Research's DAC-VAE instead of Fish-Speech S1-DAC autoencoder.
https://github.com/Aratako/Irodori-TTS
Irodori-TTS is a Japanese TTS model based on EchoTTS, but it uses Facebook Research's DAC-VAE instead of Fish-Speech S1-DAC autoencoder.
https://github.com/Aratako/Irodori-TTS
Activity
0xShug0 commentedon Jun 27, 2026
Sure! I’m a big fan of Aratako's work. MioTTS and MioCodex are amazing, and they’re already included in audio.cpp.
Update: In case anyone is interested in performance compared with other cpp implementations:
check #6 (comment).
0xShug0 commentedon Jul 1, 2026
@nazodane Irodori-TTS Done. Stay tuned for release!
0xShug0 commentedon Jul 10, 2026
@nazodane @bakamomi You can try a preview version in the dev branch.
bakamomi commentedon Jul 10, 2026
How do you know that I'm interested in Irodori? I don't recall saying it in this repo. lol
Anyway, Vulkan backend simply crashes.
I also tried voice reference with Vulkan and it crashed on encoding with:
As far as I know, this means that it ran out of memory.
Cuda backend lacks any memory management. A line of 41 characters needs 12 GB of VRAM (when running with voice reference). "本日はお越しいただき、誠にありがとうございます。どうぞごゆっくりお過ごしください。" Generating longer lines on 16 GB VRAM is impossible.
Edit: for comparison, CrispASR needs only 2 GB of VRAM for the same line and same voice reference. Also, Crisp is twice as fast... though it might be attributed to ref voice caching.
audio.cpp - real 0m3.037s
crispasr - real 0m1.644s
0xShug0 commentedon Jul 10, 2026
@bakamomi Based on your last issue, I thought Japanese TTS might interest you. Models on the
devbranch are previews and still under testing, but some Reddit users wanted to try them, so I put them all ondev. I'm testing memory now, and you can check the latest commit (5e91ece).Would you be willing to test long text in server mode, or multiple CLI requests after warm-up, rather than one-shot runs? audio.cpp caches many graphs and also speaker embedding for reuse. Input length has very little impact on VRAM usage. Short requests and 6,000 characters use almost the same amount of VRAM. Check the table below.
(weights: FP32 safetensor)
Vulkan issue is fixed. Tested on Vulkan and the resulting wavs
{
"requests": [
{
"id": "short_emoji",
"text": "今日は短い確認です。😊やさしく、聞き取りやすい声でお願いします。",
"language": "ja",
"seed": 1234,
"num_inference_steps": 40,
"options": {
"no_ref": true
}
},
{
"id": "style_sound",
"text": "📞お電話ありがとうございます。少しお待ちください。⏸️ただいま担当者へおつなぎします。",
"language": "ja",
"seed": 1234,
"num_inference_steps": 40,
"options": {
"no_ref": true
}
},
{
"id": "clone_reference",
"text": "この声の雰囲気を保ったまま、落ち着いて自然に読み上げます。",
"language": "ja",
"voice_ref": "resources/sample.wav",
"seed": 1234,
"num_inference_steps": 40,
"options": {
"no_ref": false
}
}
]
}
{
"requests": [
{
"id": "voice_design_caption",
"text": "本日はお越しいただき、誠にありがとうございます。どうぞごゆっくりお過ごしください。",
"language": "ja",
"seed": 1234,
"num_inference_steps": 40,
"options": {
"caption": "落ち着いた大人の男性。フォーマルな場で、深く響く声で丁寧かつ歓迎の意を込めて話している。",
"no_ref": true
}
},
{
"id": "voice_design_ref_caption_emoji",
"text": "あははっ🤭、それ本当に言ってるの？…😮💨まぁ、君らしいけどね。",
"language": "ja",
"voice_ref": "resources/clone_ref2.wav",
"seed": 1234,
"num_inference_steps": 40,
"options": {
"caption": "余裕のある大人の男性。親しい相手に対して、くだけた雰囲気で呆れながらも楽しそうに話している。",
"no_ref": false
}
}
]
}
Ref wavs:
clone_ref2.wav
sample.wav
clone_reference.wav
short_emoji.wav
style_sound.wav
voice_design_caption.wav
voice_design_ref_caption_emoji.wav
0xShug0 commentedon Jul 10, 2026
@bakamomi Process wall time can be misleading:
CrispASR: Q8 GGUF, audio.cpp: FP32 Safetensor. 40 steps
本日はお越しいただき、誠にありがとうございます。どうぞごゆっくりお過ごしください。
inference breakdown
Long-text or multi-request tests also amortize the model-loading and warm-up costs.
Longer: 朝の港町には、まだ薄い霧が残っていた。駅前の時計台が七時を告げるころ、商店街のシャッターが一つずつ上がり、パン屋の窓からは焼きたての香りが通りへ流れ出した。古い図書館では、司書の美咲が返却箱に積まれた本を丁寧に分けながら、週末に開く朗読会の準備を進めていた。朗読会のテーマは「遠くへ出かけなくても見つかる冒険」で、子どもから年配の人まで楽しめるように、海辺の伝説、山里の手紙、町工場で働く職人の記録など、さまざまな物語を選んでいた。そこへ、漁師の健太が濡れた帽子を片手にやって来た。彼は沖で見つけた小さな木箱を図書館に預けたいと言い、中には百年前の日付が書かれた航海日誌と、色あせた写真が入っていた。写真には、現在は倉庫になっている赤レンガの建物の前で、若い女性が笑って立っている。美咲はその建物が、昔は旅人のための音楽ホールだったことを思い出し、日誌の文字を追いながら、町の記憶が静かにほどけていくのを感じた。午後になると、学校帰りの子どもたちが図書館に集まり、木箱の話を聞いて目を輝かせた。ある子は写真の女性が有名な歌手だったのではないかと言い、別の子は日誌に残された地図が宝探しの暗号だと主張した。美咲は笑いながら、まずは資料を調べ、古い新聞を読み、町の人たちに話を聞くことが大切だと説明した。その言葉を聞いた健太は、祖母が赤レンガの建物について何か知っているかもしれないと言い、夕方までに話を聞いて戻ると約束した。日が傾くころ、図書館の窓は橙色に染まり、ページをめくる音だけが静かに響いていた。美咲は朗読会の最後に、この木箱から始まった調査の途中経過を紹介しようと決めた。完成した答えではなく、みんなで手がかりを集めていく過程そのものが、きっと一番わくわくする物語になると思ったからだ。夜になり、港の灯台がゆっくり光を回し始めると、町は昼間とは違う表情を見せた。風は少し冷たくなったが、図書館の中には小さな熱が残っていた。美咲は机の上に日誌を置き、破れやすい紙を傷つけないよう手袋をはめた。最初のページには、若い船乗りが初めてこの町に着いた日のことが書かれていた。彼は嵐で予定の航路を外れ、偶然この港に避難したらしい。ところが、そこで聴いた一夜限りの演奏会に心を奪われ、翌朝出発するはずだった船を見送り、数日だけ町に残ることにした。日誌には、演奏していた女性の声が、波の音と混ざって夜明けまで忘れられなかったと記されている。美咲は写真の女性と日誌の文章を見比べ、これが単なる拾い物ではなく、町に眠っていた小さな歴史の入口なのだと確信した。
inference breakdown:
bakamomi commentedon Jul 10, 2026
I tried the server with new updates.
VRAM situation is better. At least things don't just crash with OOM on long lines.
But still,
📖🐢衝動は、もはや抑えられなかった。二乃は淹れたての熱いコーヒーが乗ったトレイを掴むと、迷わず男のテーブルへと向かった。
「😱きゃっ！」
📖🐢わざとらしく足をもたつかせ、二乃は持っていたトレイを男の膝の上に「うっかり」ひっくり返した。
「😖熱っ！ あ、あつっ！」
📖🐢安物のスラックスに染み込んでいく熱いコーヒーに、男――ヒロシは悲鳴を上げる。店内の客の視線が一斉に集まる中、二乃は勝 ち誇ったように仁王立ちになり、声を張り上げた。
Garbled output sometimes appears earlier when using reference audio. I suppose it just goes over maximum line length. So, how about using Irodori speech length predictor to chunk text into sentences of about 15 seconds long?
P.S. It would be nice to add gguf support. audio.cpp idles at 2.6 GB, while CrispASR with q8 takes just 1.2 GB at diffusion (more at decoding).
bakamomi commentedon Jul 10, 2026
I retested speed with your long text. Chunking works for your test line, and audio.cpp is faster than CrispASR. But VRAM usage for 0.5B model is just too much. People use 0.5B because they want small footprint.
0xShug0 commentedon Jul 10, 2026
@bakamomi Thanks for testing! This is exactly the kind of feedback I need before releasing the model. I don’t speak Japanese, so I really can’t tell whether the Japanese output sounds right.
For (2), I couldn’t reproduce the issue. For (3), I can’t tell, but I collected the audio outputs for your examples from both audio.cpp and the Python implementation (under random seeds). Let me know if the Python output sounds correct but the audio.cpp output doesn’t.
audio.cpp also supports long-form generation by default. There is no limit on text len. You don’t need to configure anything. It automatically chunks long text.
Using text input for the multiline example: input.txt
Update: I messed up the naming and shared duplicated wavs. Maybe I should let AI handle this task. Wavs re-upload.
cuda_1.wav
vulkan_1.wav
python_1.wav
cuda_2.wav
vulkan_2.wav
python_2.wav
0xShug0 commentedon Jul 10, 2026
As for VRAM vs performance, there’s always a trade-off. I’ve tried layer-wise and chunking to reduce memory usage, but performance can drop by up to 50%, depending on the model. My current priority is keeping memory usage stable across requests in a long-lived session, so the current mem-saver mode is less aggressive to better balance performance and VRAM usage (of course, there may still be easier ways to reduce memory usage that I haven’t found yet).
GGUF support will be rolled out model by model.
bakamomi commentedon Jul 10, 2026
I bet it's because you're testing on a dGPU with lots of vram. I'm testing Vulkan on iGPU that has just 512 Mb dedicated VRAM (it's almost always completely filled), while working memory is allocated dynamically from RAM pool. Here is full log:
All your examples for my test line have garbled speech in the later part (including python implementation). Some sentences are just missing. The issue is most probably the fact that there is not enough time (30s) to say all text. It needs to be chunked, but it's not chunked.
0xShug0 commentedon Jul 10, 2026
That’s interesting. Chunking is based on text length, not audio duration. There is no minimum text length requirement. As shown in the example WAVs I shared, some outputs are only about 5 seconds long. The text you pasted is about 89 Japanese characters, while the default chunk size is 256, so it will not be chunked. From the logs, the behavior looks correct.
I ran the exact cmd on iGPU and the wav sounds okay.
output_audio.wav
bakamomi commentedon Jul 11, 2026
I was talking about this text:
📖🐢衝動は、もはや抑えられなかった。二乃は淹れたての熱いコーヒーが乗ったトレイを掴むと、迷わず男のテーブルへと向かった。
「😱きゃっ！」
📖🐢わざとらしく足をもたつかせ、二乃は持っていたトレイを男の膝の上に「うっかり」ひっくり返した。
「😖熱っ！ あ、あつっ！」
📖🐢安物のスラックスに染み込んでいく熱いコーヒーに、男――ヒロシは悲鳴を上げる。店内の客の視線が一斉に集まる中、二乃は勝 ち誇ったように仁王立ちになり、声を張り上げた。
It's 220 characters. You posted it as
cuda_2.wav
vulkan_2.wav
python_2.wav
It's not chunked, but the
🐢emoji tells the model to speak slowly. On top of that, it needs pauses between narrative and direct speech. Together, these things make text much longer than 30 sec. CrispASR generates 40 seconds for this text.Another important note, chunking by character number is very unreliable for Japanese because kanji can unpack to multiple kana characters. As far as I know, Irodori TTS has built-in length predictor (CrispASR uses it to decide how many latent frames to allocate for diffusion) specifically because it's hard to predict text length.
I don't know what to do about Vulkan issues.
Is there some way to get more debug output?
0xShug0 commentedon Jul 11, 2026
@bakamomi Yes. audio.cpp uses duration predictor.
audio.cpp/src/models/irodori_tts/condition_encoder.cpp
Line 547 in 5e91ece
Does python have the same issue with the same input?
0xShug0 commentedon Jul 11, 2026
Also I don't know if this helps, but you can try to force the duration.
./audiocpp_cli
--task tts
--family irodori_tts
--model /path/to/Irodori-TTS-500M-v3
--backend cuda
--threads 8
--text "..."
--option duration_seconds=12.0
--out out.wav
./audiocpp_cli --task tts --family irodori_tts --model models/Irodori-TTS-500M-v3 --backend cuda --threads 8 --text "本日はお越しいただき、誠にありがとうございます。どうぞごゆっくりお過ごしください。" --request-option [xxx]=[yyy] --out out.wav0xShug0 commentedon Jul 11, 2026
I think you can try different seeds. I tested the official Python implementation with random seeds, and some produced nonsensical output when
duration_scale = 0.5. The same happens with audio.cpp.bakamomi commentedon Jul 11, 2026
I tried the duration options. None of them allow setting duration longer than 30 seconds, but that's understandable. The model probably wasn't trained on longer continuous audio. Besides, manual duration setting won't work for automatic TTS.
The python version also fails to chunk the provided text example. You posted it here python_2.wav It's missing a chunk of text at the end because there is not enough time to say it.
I guess your implementation follows the reference one. It's up to you if you want to make it better than the reference.
0xShug0 commentedon Jul 11, 2026
@bakamomi did you try max seconds?
{
"max_seconds": 60,
"trim_tail": false
}
In both audio.cpp and python, 30s is the default cap of duration predictor but can be changed.
bakamomi commentedon Jul 11, 2026
Alright. max_seconds works to extend audio, but the result is gibberish. Speech breaks down even if I set max_seconds=35.
0xShug0 commentedon Jul 11, 2026
I want to make sure I am not chasing wrong target. Could you please run the failed case with a fixed seed, and share the exact cmd you used and the wav?
bakamomi commentedon Jul 11, 2026
Output audio: https://litter.catbox.moe/asl7ka5zb6z2rg30.wav
Around 0:20, the speech jumps from
ひっくり返した。to店内の客の視線が一斉に集まる中、二乃は勝ち誇ったように仁王立ちになり、声を張り上げた。It's missing「😖熱っ！ あ、あつっ！」📖🐢安物のスラックスに染み込んでいく熱いコーヒーに、男――ヒロシは悲鳴を上げる。The speech uses up all 30 seconds, so clearly the missing part is dropped simply because there is no time for it.0xShug0 commentedon Jul 11, 2026
Thanks. Could you also share (1) any correct WAV output produced by other implementations for this input, along with the command you used (2) the gibberish wav you mentioned in #6 (comment) and the cmd? I’d like to determine whether this is a limitation of the reference Python implementation, a bug in audio.cpp, or a limitation of the model itself.
bakamomi commentedon Jul 11, 2026
CripsASR can do this. BUT it automatically chunks text into three parts of 11, 11, and 20 seconds (it's visible in console output).
https://litter.catbox.moe/33mulu.wav
This produced garbled output.
0xShug0 commentedon Jul 11, 2026
@bakamomi I think I understand whats going on. Check the latest commit on the dev branch 85d4a65. You can change the default text chunk mode now. The default mode for Irodori is bin packing, and since the budget is 256, it won’t chunk the text. Also I checked CrispASR and it does NOT do duration-based chunking for Irodori. It still uses text chunk, and the 11s / 11s / 20s shown in logs are the generated audio durations after synthesis, not the rule used to split the text.
./audiocpp_cli --task tts --request-option text_chunk_mode=endline --model models/Irodori-TTS-500M-v3 --out [xxx] --backend cuda --threads 8 --log --text [xxxx]
Don't set duration_second.
Seed matters because Irodori a stochastic diffusion/flow-matching TTS model. Check the wavs attached. The seeds 2282869626/1935717885/1849449564 (one for each chunk generated randomly) produced garbled output (based on ASR), while other random seeds work better.
irodori_endline_seed3131938595-2382678063-2557918360.wav
irodori_endline_seed3026951487-949633697-2041888617.wav
irodori_endline_seed2282869626-1935717885-1849449564.wav
ci: defer arm64 ctest and HIP job (FT-1/FT-2)
ci: defer arm64 ctest and HIP job (FT-1/FT-2)