Repository navigation
fix(server): align b10621 audio transcription route semantics #1446
Description
Parent
Problem
Both servers register /v1/audio/transcriptions and /audio/transcriptions, but b10621 converts multipart audio through the loaded multimodal chat model, accepts a prompt and language, supports only the json response format, and rejects models without audio capability. mlxcel uses a dedicated STT worker, omits the prompt field, and supports additional formats, so route-name overlap alone is not behavioral compatibility.
Scope
- Align the shared multipart field set, required file behavior, model/capability checks, prompt/language handling, response-format validation, and error envelopes.
- Decide whether a dedicated mlxcel STT model is a correctly aliased implementation or whether the compatibility route must use the loaded multimodal chat model.
- Preserve mlxcel-only transcription/translation formats and speech routes under clearly documented extensions when they exceed b10621.
- Apply upload byte, decoded-duration, channel/rate, multipart-part, timeout, and concurrent preprocessing limits.
- Align authentication and native
/audio/transcriptionsalias behavior.
Acceptance criteria
- Differential multipart tests cover file, model, prompt, language, temperature, response format, unknown fields, and duplicate parts.
- Unsupported audio capability and non-
jsoncompatibility requests return b10621-compatible errors. - Extra mlxcel formats are documented as extensions and do not change compatibility-route defaults.
- Oversized, malformed, truncated, and decompression-amplifying inputs fail within bounded memory/time.
- The same short clip produces equivalent language/prompt routing through a real audio-capable checkpoint.
Status: delivered, kept open for two recorded divergences
PR #1484 merged and satisfies every acceptance criterion above. The design question the issue asked to settle was settled by measurement rather than by preference: mlxcel's dedicated Whisper worker is not a correct alias of b10621's route, because a Whisper -m leaves the chat worker unloaded, so mlxcel-server -m <audio-capable chat model>, the only server shape b10621 can express, answered 501 audio model kind not loaded: stt. The compatibility route now dispatches through the loaded multimodal chat model as upstream does, and the Whisper worker remains the fallback for the shape b10621 cannot express. Coverage is src/server/routes/transcription_compat_tests.rs, 18 tests, validated on gemma3n-e2b-4bit, whisper-tiny and qwen3-0.6b-4bit.
The issue stays open because both route entries in compat/llama-server/b10621/multimodal-and-audio.json remain deferred against it, and the compatibility gate requires a deferred entry's issue to be open. Two divergences are left, neither of which was in the acceptance criteria above; both were found while measuring against the pinned binary.
- Container support. Only RIFF/WAVE is decoded, on both the chat-model and the Whisper path, so a non-WAV clip is a 400 here and a transcript on b10621, whose mtmd front-end takes mp3, flac and the rest. Closing this needs an audio decoder under
src/audio/and the removal of theformat != "wav"refusal inread_audio_input. It is not a route change. - Stream granularity.
stream=truereturns onetranscript.text.deltaframe carrying the whole transcript, then thedoneframe anddata: [DONE]. The frame shapes and the terminator are upstream's; incremental delivery needs an ASR variant ofstream_chat_completioninsrc/server/routes/chat.rs, which today builds chat chunks rather than transcript events.
A third residue is recorded on the entries but is unreachable from a b10621 command line: on the mlxcel-only Whisper-server shape the usage counts are zeros and prompt steers nothing, because the STT worker reports neither.
Activity
test(server): add pinned b10621 compatibility manifest and gates
fix(server): translate model-source flags and reject GGUF semantics (#…
fix(server): classify projector flags and confine request media (#1481)
fix(server): align the b10621 audio transcription route (#1484)
inureyes commentedon Aug 28, 2026
PR #1484 merged and delivers most of this issue, but deliberately does not close it: two divergences remain, both route entries stay
deferredagainst this issue incompat/llama-server/b10621/multimodal-and-audio.json, and the entries'divergencelists now record exactly what is left.The design decision, settled by measurement
b10621 has no speech-to-text model; its transcription route is a translation layer over
/v1/chat/completions. mlxcel's route was served by a dedicated Whisper worker populated only when-mnames a Whisper checkpoint, and that same-mleaves the chat worker unloaded, so the two were mutually exclusive server shapes. Before the change,mlxcel-server -m models/mlx/whisper-tinytranscribed and answered/v1/chat/completionswith a 503, whilemlxcel-server -m models/mlx/gemma3n-e2b-4bitanswered chat and answered/v1/audio/transcriptionswith501 audio model kind not loaded: stt. The second command line is the only shape b10621 can express, and it was the one that did not work. The dedicated STT worker is therefore not a correctly aliased implementation, and the compatibility route now uses the loaded multimodal chat model, keeping the Whisper worker as the fallback for the shape b10621 cannot express.Delivered
The capability refusal and its precedence over every field check, the missing-file refusal,
response_formatrestricted tojson, the ASR preset prompt with the" (language: xx)"suffix,temperatureandmax_tokensretyped from strings,streamcompared against the literal"true", the multipart duplicate-key rules, unknown fields carried and ignored, thetranscript.text.done/transcript.text.delta/data: [DONE]response shapes, and the multipart-part, upload-byte and WAV-geometry limits. mlxcel'stextandverbose_jsonformats moved to/v1/audio/translations, which b10621 does not mount. Validated withgemma3n-e2b-4bit,whisper-tinyandqwen3-0.6b-4biton M1 Ultra.Remaining, and what each would take
src/audio/, plus lifting theformat != "wav"refusal inread_audio_input; it is not a route change.stream=truereturns onetranscript.text.deltaframe carrying the whole transcript, then thedoneframe anddata: [DONE]. The frame shapes and the terminator are upstream's; incremental delivery would mean an ASR variant ofstream_chat_completioninsrc/server/routes/chat.rs, which builds chat chunks rather than transcript events.A third residue is recorded on the entries but is unreachable from a b10621 command line: on the Whisper-server shape the
usagecounts are zeros andpromptsteers nothing, because the STT worker reports neither.feat(server): implement b10621 resumable streams and completion contr…
feat(compat): add the by_design b10621 compatibility-policy state
feat(server): decode mp3 and flac and stream ASR per token (lablup#1523)
test(server): add pinned b10621 compatibility manifest and gates
fix(server): translate model-source flags and reject GGUF semantics (l…
fix(server): classify projector flags and confine request media (labl…
fix(server): align the b10621 audio transcription route (lablup#1484)
feat(server): implement b10621 resumable streams and completion contr…