Vulkan backend: Japanese generation often never emits EOS and runs to the frame cap (CUDA/CPU do not reproduce) #10

Summary

On the Vulkan backend, Japanese generation frequently fails to emit EOS and runs all the way to the frame cap. The audio stays fluent-sounding but stops being the requested text, and ASR often labels it as a different language (Korean/Vietnamese). Rebuilding the exact same commit with -DGGML_CUDA=ON and changing nothing else cuts the failure rate by ~3.5x and eliminates the language drift entirely.

Same zonos2-q6_k.gguf, same speaker embedding, same sampling parameters, same two sentences, 20 generations per backend.

I mention this partly because every prebuilt GPU binary in the release assets is Vulkan (zonos2-windows-x64-vulkan.zip, zonos2-linux-x64-vulkan.tar.gz), while docs/INTERNALS.md's numerical-validation table covers CPU and CUDA only — so the shipped GPU path looks like the one path that isn't covered by that table.

Environment

Vulkan build = the released zonos2-windows-x64-vulkan.zip toolchain; CUDA build = same tree configured with -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86.

Reproduction

Two Japanese sentences, one per request:

• A (32 chars): 本日は晴天なり。これは音声合成エンジンの長文読み上げテストです。
• B (45 chars): 文章が長くなった場合でも、最後まで正しく安定して音声が生成されるかどうかを確認しています。

{
  "text": "<A or B>",
  "stream": false,
  "format": "wav",
  "max_tokens": 1000,
  "temperature": 0.85,
  "min_p": 0.25,
  "speaker_embedding_id": "<uploaded via /tts/speakers>"
}

I count a response as runaway when its duration reaches ≥ 90 % of the requested cap (max_tokens * 512 / 44100 s) — i.e. EOS never fired. Expected speech for A is ≈ 4.6 s against an 11.6 s cap, so it is unambiguous, and consecutive runaways come back with byte-identical lengths pinned at the ceiling.

What I ruled out first

Quantization — 10 generations each, all on Vulkan:

Raising the quant does nothing; lowering it makes things worse, consistent with the lower top-1.

Temperature — 8 generations per setting, Vulkan:

No trend anywhere in the range.

Reference audio — it helps rather than hurts, so this is not a speaker-conditioning problem (4 generations of A, Vulkan):

Port logic — I diffed the C++ against the Python reference and found no behavioural difference:

• EOS detection in zonos2-graph.cpp (last codebook index whose token is eoa_id → eos_frame = max(0, step - max_eoa) → countdown = ncb + 1) matches python/zonos2/tts/sequence.py::_check_eos exactly.
• The sampler matches too: same order (repetition penalty → temperature → top-k → softmax → top-p → min-p) and the top-p/min-p masks are mathematically equivalent to tts/sampler.py.
• Every default matches server/api_server.py: temperature 1.15, topk 106, top_p 0.0, min_p 0.18, repetition_window 50, repetition_penalty 1.2, repetition_codebooks 8.

That left the compute backend as the only remaining variable, which is where the difference showed up.

In the Q6_K Vulkan run, the three non-Japanese detections were exactly the three runaways (all ko), so "it started speaking another language" and "it never stopped" are the same event: once it overruns EOS it keeps producing plausible audio that is no longer Japanese. (Language detection is faster-whisper base on the returned WAV.)

Given this is an MoE model and the docs already note that small perturbations can flip the router's top-k expert choice and decorrelate the output, a slight numerical difference in the Vulkan kernels seems like a plausible mechanism, though I have not bisected which kernel diverges. Happy to run --validate / models/compare.py against Vulkan if that would be useful — the existing golden-comparison tooling looks like it would localize this quickly.

Secondary: max_tokens is clamped silently

parse_gen_req does out.max_frames = (req_max > 0) ? std::min(req_max, cap) : cap; with cap = s.max_frames (default 2000 ≈ 23 s), and nothing in the response says the request was reduced. A client asking for 4560 frames silently gets 2000, so a client-side "did this hit the cap?" check computed from the requested value misses every runaway. Surfacing the effective cap in the response would make this detectable.

Raising the ceiling isn't a workaround, at least on a 12 GB card: --max 6000 makes DAC decode attempt a ~7 GB allocation and die with vk::Device::allocateMemory: ErrorOutOfDeviceMemory.

Workarounds I'm using

Switching to a CUDA build is what actually helped. On top of that, client-side: split input so each request stays well under the cap, and regenerate on a fresh seed when the returned duration indicates a runaway.

Thanks for open-sourcing this — happy to run further experiments or test a patch.