feat(vlm): add MiniMax-M3-VL multimodal support

docs/supported-models.md

@@ -95,6 +95,7 @@ Implemented VLM variants include:
 - ERNIE-4.5 MoE VL (`ernie4_5_moe_vl`): Baidu's vision-language MoE. A DFNRope ViT (linear patch embedding over 588-wide merge-window rows, 2D vision RoPE, `cu_seqlens`-packed attention, quick_gelu MLP) feeds a variable-resolution resampler (2x2 spatial fold, temporal pair fold with single-frame duplication, GELU MLP stacks, RMSNorm) whose rows replace the `<|IMAGE_PLACEHOLDER|>` tokens. The text decoder extends ERNIE-4.5 MoE with modality-split expert banks: separate text and multimodal routers and expert stacks selected per token by token type, with a correction bias that shifts expert selection but never the mixing weights, plus a fused shared-experts MLP. Position encoding is interleaved 3D MRoPE (`[T, H, W]` axes assigned per frequency index, adjacent-pair rotation) that degenerates to traditional RoPE for text. Validated against `mlx-community/ERNIE-4.5-VL-28B-A3B-Thinking-4bit` (28B-A3B; text and resampler 4-bit, vision tower bf16). Best for general image chat and grounded reasoning; the checkpoint is a thinking variant and emits reasoning before the answer.
 - Qwen3-Omni MoE (`qwen3_omni_moe`, thinker): Alibaba's omni-modal MoE. Stage 1 covers the thinker: text output conditioned on text, image, and audio inputs. The vision tower and MoE text decoder are the Qwen3-VL-MoE stack (DeepStack feature injection, interleaved MRoPE) reused unchanged; the new audio tower converts 16 kHz audio to a 128-bin log-mel spectrogram, downsamples it through three stride-2 convolutions (13 output frames per second of audio), and runs 32 windowed-attention encoder layers whose output rows scatter into the token stream exactly like image features. Audio arrives via `--audio file.wav` on the CLI (combinable with `--image`). Stage 2 adds speech output: `mlxcel generate --output-audio out.wav` runs the talker and code2wav after text generation and writes 24 kHz mono PCM16. The talker is a 20-layer Qwen3-MoE codec decoder conditioned on the projected thinker token embeddings of the chat-role segments; per frame it emits the first of 16 codebooks and a 5-layer code predictor fills in the residual 15, then the code2wav vocoder (causal pre-transformer, ConvNeXt upsampling, BigVGAN-style SnakeBeta decoder) renders 1920 samples per 12.5 Hz frame. `--speaker` selects the voice (ethan default; chelsie and aiden also ship in the released checkpoints). The speech stack loads lazily and only when requested, so text/vision use keeps its memory footprint; speech currently requires a text-only, chat-templated prompt. Validated against `mlx-community/Qwen3-Omni-30B-A3B-Instruct-4bit` (text and talker 4-bit; vision, audio tower, code predictor, and code2wav bf16).
 - Hunyuan-VL (`hunyuan_vl`, e.g. HunyuanOCR): Tencent's vision-language family. A ViT with a per-patch conv embedding, bilinearly interpolated learned position embeddings, and full attention over the packed patch sequence feeds a `perceive` merger: a stride-2 conv pair over the raster grid, a learned `image_newline` column, a linear to the decoder width, and learned `image_begin` / `image_end` rows. Per image that yields `mh * (mw + 1) + 2` feature rows, matching the prompt placeholder count exactly. The decoder is the Hunyuan dense stack (per-head Q/K RMSNorm after the rotation, DynamicNTK-alpha rope base) with XD-RoPE at prefill: 4D `[P, T, H, W]` position ids split across the frequency dims, degenerating to the standard rotation for text; decode uses sequential positions. Validated against `hadeseus/HunyuanOCR-mlx-4bit` (text 4-bit, vision bf16). Best for OCR: text spotting, document parsing, and grounded extraction.
+- MiniMax-M3-VL (`minimax_m3_vl`): MiniMax's vision-language model on top of the MiniMax-M3 text backbone. A CLIP-style ViT (hidden 1280, 16 heads, 32 layers, patch 14, `pre_layrnorm`, LayerNorm + exact-GELU blocks with separate `q/k/v/out` projections) runs native-resolution packing: Qwen2-VL-style dynamic `smart_resize` to `patch_size * spatial_merge_size = 28`-aligned dimensions, `image_grid_thw` patchify, per-image `cu_seqlens` variable-length attention, and 3D (t, h, w) vision RoPE (temporal axis inert for images, trailing dims unrotated). A two-stage projector then maps features to the text width: a per-patch `multi_modal_projector` (`linear_1 -> GELU -> linear_2`, into `projection_dim` 6144) followed by a `patch_merge_mlp` that folds each `spatial_merge_size^2 = 4` adjacent patches (`linear_1` [6144, 24576] `-> GELU -> linear_2`). Each `]<]image[>[` placeholder expands to `grid_t * (h/2) * (w/2)` tokens (`vision_start` 200029, `vision_end` 200030), which the merged features replace LLaVA-style; the MiniMax-M3 hybrid dense/MoE decoder then runs its standard partial 1D RoPE. The vision tower runs in f32 (non-quantized in the checkpoint) while the text tower may be quantized in community exports. Image and multi-image inputs are wired end to end (CLI and server); video is out of scope for this port. The only public checkpoint (`MiniMaxAI/MiniMax-M3`, 427B) exceeds the development machine, so image Q&A parity is deferred to a runtime validation once a fitting quantized conversion exists; the merge gate is the unit tests plus the real nested-config parse.
 - FastVLM (`llava_qwen2` / `fastvlm`): Apple's low-latency VLM. A FastViTHD hybrid encoder runs entirely on channels-last maps: a conv stem, three RepMixer stages (depthwise token mixing plus a BatchNorm ConvFFN), two attention stages with channel LayerNorm and `head_dim` 32 self-attention, inter-stage large-kernel PatchEmbed downsamples, RepCPE position encoders, and a squeeze-excite `conv_exp` head. Each 1024x1024 pad-to-square image becomes a `(16, 16, 3072)` map flattened to 256 tokens, projected to the Qwen2 decoder width by an `mlp2x_gelu` MLP. The `<image>` placeholder is the fixed `-200` sentinel (not a vocabulary token); the runtime splices one sentinel per image, expands it to 256 tokens, and scatters the image embeddings (LLaVA merge). The text decoder is stock Qwen2, reused unchanged. The loader accepts both the genuine (`apple/FastVLM-0.5B`) and converted (`mlx-community/FastVLM-0.5B-bf16`) weight layouts. Best for fast image description and grounded chat.
 - GLM-OCR (`glm_ocr`): document-OCR sibling of GLM-4V. A 24-block ViT (3D patch embedding, per-head q/k RMSNorm on the packed `cu_seqlens` attention, 2D vision RoPE, Conv2d spatial downsample, SwiGLU patch merger) feeds a 16-layer GLM-4 text decoder driven by full-width even/odd MRoPE (`rope_parameters` with `mrope_section [16, 24, 24]`, `partial_rotary_factor 1.0`). The tower has no learned position embedding or post-conv norm, and the loader drops the next-n prediction (MTP) layer. Patches are reordered from the processor's raster order into spatial-merge-window order so the rotary, downsample, and merged-token grid stay spatially aligned (OCR reads scrambled patches wrong). Best for plain text, tables, and formula recognition.
 - Youtu-VL

src/distributed/tensor_parallel/inference.rs

@@ -171,6 +171,7 @@ fn fallback_architecture(model_type: ModelType) -> &'static str {
         ModelType::GptOss => "gpt_oss",
         ModelType::MiniMax => "minimax",
         ModelType::MiniMaxM3 => "minimax_m3",
+        ModelType::MiniMaxM3VL => "minimax_m3",
         ModelType::Mixtral => "mixtral",
         ModelType::Qwen2Moe => "qwen2_moe",
         ModelType::OLMoE => "olmoe",

src/loaded_model.rs

@@ -84,6 +84,7 @@ pub enum LoadedModel {
     FastVLM(vision::VisionLanguageModel),
     Ernie45MoeVLM(vision::ernie4_5_moe_vl::Ernie45MoeVlModel),
     HunyuanVLM(vision::hunyuan_vl::HunyuanVlModel),
+    MiniMaxM3VL(vision::MiniMaxM3VlModel),
     GraniteVisionVLM(vision::GraniteVisionVLModel),
     Granite4VisionVLM(vision::Granite4VisionVLModel),
     DeepSeekOcrVLM(vision::deepseekocr::DeepSeekOcrVlModel),
@@ -229,6 +230,7 @@ macro_rules! delegate_language_model {
             LoadedModel::FastVLM(inner) => LanguageModel::$method(inner, $($arg),*),
             LoadedModel::Ernie45MoeVLM(inner) => LanguageModel::$method(inner, $($arg),*),
             LoadedModel::HunyuanVLM(inner) => LanguageModel::$method(inner, $($arg),*),
+            LoadedModel::MiniMaxM3VL(inner) => LanguageModel::$method(inner, $($arg),*),
             LoadedModel::GraniteVisionVLM(inner) => LanguageModel::$method(inner, $($arg),*),
             LoadedModel::Granite4VisionVLM(inner) => LanguageModel::$method(inner, $($arg),*),
             LoadedModel::DeepSeekOcrVLM(inner) => LanguageModel::$method(inner, $($arg),*),

src/loaded_model_capabilities.rs

@@ -72,6 +72,8 @@ pub enum VlmRuntimeRef<'a> {
     Ernie45MoeVl(&'a vision::ernie4_5_moe_vl::Ernie45MoeVlModel),
     Qwen3OmniMoe(&'a vision::qwen3_omni_moe::Qwen3OmniMoeModel),
     HunyuanVl(&'a vision::hunyuan_vl::HunyuanVlModel),
+    /// MiniMax-M3-VL runtime (CLIP ViT + MiniMax-M3 hybrid dense/MoE text).
+    MiniMaxM3Vl(&'a vision::MiniMaxM3VlModel),
     /// Pixtral / Mistral3 dynamic aspect-ratio runtime. Shares the generic
     /// `VisionModule` storage but preserves each image's aspect ratio and emits
     /// `[IMG] / [IMG_BREAK] / [IMG_END]` row structure (see `pixtral_layout`).
@@ -206,6 +208,7 @@ impl LoadedModel {
             Self::FastVLM(vlm) => Some(VlmRuntimeRef::FastVLM(&vlm.vision)),
             Self::Ernie45MoeVLM(model) => Some(VlmRuntimeRef::Ernie45MoeVl(model)),
             Self::HunyuanVLM(model) => Some(VlmRuntimeRef::HunyuanVl(model)),
+            Self::MiniMaxM3VL(model) => Some(VlmRuntimeRef::MiniMaxM3Vl(model)),
             _ => None,
         }
     }

src/loading/mod.rs

@@ -186,6 +186,7 @@ fn try_load_vlm_model_from_dir(
         ModelType::FastVLM => Some(load_fastvlm_vlm(model_path)?),
         ModelType::Ernie45MoeVLM => Some(load_ernie4_5_moe_vlm(model_path)?),
         ModelType::HunyuanVLM => Some(load_hunyuan_vlm(model_path)?),
+        ModelType::MiniMaxM3VL => Some(load_minimax_m3_vl(model_path)?),
         ModelType::LlavaBunnyVLM => Some(load_llava_bunny_vlm(model_path)?),
         ModelType::AyaVisionVLM => Some(load_aya_vision_vlm(model_path)?),
         ModelType::PaliGemmaVLM => Some(load_paligemma_vlm(model_path)?),

src/loading/vlm.rs

@@ -67,6 +67,8 @@ mod kimi_vl_loader;
 mod lfm2_vl;
 #[path = "vlm_llava.rs"]
 mod llava;
+#[path = "vlm_minimax_m3_vl.rs"]
+mod minimax_m3_vl;
 #[path = "vlm_mllama.rs"]
 mod mllama;
 #[path = "vlm_nemotron_h_nano_omni.rs"]
@@ -103,6 +105,7 @@ pub(crate) use internvl::load_internvl_vlm;
 pub(crate) use kimi_vl_loader::load_kimi_vl_vlm;
 pub(crate) use lfm2_vl::load_lfm2_vl;
 pub(crate) use llava::{load_llava_bunny_vlm, load_llava_vlm};
+pub(crate) use minimax_m3_vl::load_minimax_m3_vl;
 pub(crate) use mllama::load_mllama_vlm;
 pub(crate) use nemotron_h_nano_omni::load_nemotron_h_nano_omni_vlm;
 pub(crate) use paddleocr::load_paddleocr_vl;

src/loading/vlm_minimax_m3_vl.rs

@@ -0,0 +1,106 @@
+// Copyright 2025-2026 Lablup Inc. and Jeongkyu Shin
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! MiniMax-M3-VL loader (`model_type: "minimax_m3_vl"`).
+//!
+//! The only public checkpoint (`MiniMaxAI/MiniMax-M3`, 427B) is itself the VL
+//! model: a top-level `minimax_m3_vl` config with nested `text_config` and
+//! `vision_config`. Text weights live under `language_model.model.*` and the
+//! CLIP-style vision tower under `vision_tower.vision_model.*`, with the
+//! two-stage projector at `multi_modal_projector.*` / `patch_merge_mlp.*`.
+//!
+//! The tower is built from the raw weight map first (its verbatim keys are
+//! intact there), after which the map is moved into the MiniMax-M3 text
+//! sanitizer, which drops the vision/projector tensors and rewrites
+//! `language_model.model.*` -> `model.*` for the text decoder. Vision weights
+//! are f32/bf16 non-quantized while the text tower may be quantized in
+//! community exports; the text `ModelArgs` inherits the top-level quantization
+//! block when its own `text_config` omits it.
+
+use anyhow::Result;
+use std::path::Path;
+
+use crate::LoadedModel;
+use crate::models::MiniMaxM3Model;
+use crate::models::minimax_m3::{ModelArgs, sanitize_weights};
+use crate::vision;
+use crate::vision::encoders::minimax_m3_vl::{MiniMaxM3VisionConfig, MiniMaxM3VisionEncoder};
+use crate::vision::processors::minimax_m3::MiniMaxM3Processor;
+
+use super::{load_vlm_weights_common, read_sanitized_vlm_config};
+
+pub(crate) fn load_minimax_m3_vl(model_path: &Path) -> Result<LoadedModel> {
+    let (_config_str, full_config) = read_sanitized_vlm_config(model_path)?;
+
+    // Text config lives under `text_config`. Community 4-bit exports commonly
+    // keep the quantization block at the config root, so inject it when the
+    // nested block omits it.
+    let mut text_value = full_config
+        .get("text_config")
+        .cloned()
+        .ok_or_else(|| anyhow::anyhow!("Missing text_config in MiniMax-M3-VL config.json"))?;
+    if let (Some(obj), Some(quant)) = (text_value.as_object_mut(), full_config.get("quantization"))
+    {
+        obj.entry("quantization").or_insert_with(|| quant.clone());
+    }
+    let text_args: ModelArgs = serde_json::from_value(text_value)
+        .map_err(|e| anyhow::anyhow!("Failed to parse MiniMax-M3-VL text_config: {}", e))?;
+    if let Some(sparse) = text_args.sparse_attention_config.as_ref() {
+        sparse.validate().map_err(|e| anyhow::anyhow!(e))?;
+    }
+
+    let vision_value = full_config
+        .get("vision_config")
+        .cloned()
+        .ok_or_else(|| anyhow::anyhow!("Missing vision_config in MiniMax-M3-VL config.json"))?;
+    let vision_config: MiniMaxM3VisionConfig = serde_json::from_value(vision_value)
+        .map_err(|e| anyhow::anyhow!("Failed to parse MiniMax-M3-VL vision_config: {}", e))?;
+
+    let weights = load_vlm_weights_common(model_path, None)?;
+
+    // Build the vision tower from the raw keys first (borrowing), then move the
+    // map into the text sanitizer.
+    let vision_encoder = MiniMaxM3VisionEncoder::from_weights(&weights, &vision_config)
+        .map_err(|e| anyhow::anyhow!("Failed to load MiniMax-M3-VL vision tower: {}", e))?;
+
+    let text_weights = sanitize_weights(weights, &text_args);
+    let text_model = MiniMaxM3Model::from_weights(&text_weights, &text_args)
+        .map_err(|e| anyhow::anyhow!("Failed to load MiniMax-M3-VL text model: {}", e))?;
+
+    let read_token = |key: &str, default: i64| -> i32 {
+        full_config
+            .get(key)
+            .and_then(|v| v.as_i64())
+            .unwrap_or(default) as i32
+    };
+
+    let mut eos_token_ids = crate::loading::read_eos_token_ids(model_path);
+    if eos_token_ids.is_empty() {
+        // MiniMax `[e~[` sentinel at 200020 in its 200k vocab.
+        eos_token_ids = vec![200020];
+    }
+
+    let vlm = vision::minimax_m3_vl::MiniMaxM3VlModel {
+        text_model,
+        vision_encoder,
+        processor: MiniMaxM3Processor::default(),
+        image_token_id: read_token("image_token_id", 200025),
+        video_token_id: read_token("video_token_id", 200026),
+        vision_start_token_id: read_token("vision_start_token_id", 200029),
+        vision_end_token_id: read_token("vision_end_token_id", 200030),
+        spatial_merge_size: vision_config.spatial_merge_size() as i32,
+        eos_token_ids,
+    };
+    Ok(LoadedModel::MiniMaxM3VL(vlm))
+}

src/model_metadata.rs

@@ -165,6 +165,7 @@ macro_rules! for_each_model_registration {
             PhiMoe => { kind: Text, directory: ConfigBacked, weight: Some(WeightLoadRoute::ConfigBacked), adapter: None, config_backed: { dir_loader: models::PhiMoeModel::load, args: models::phimoe::ModelArgs, weight_builder: models::PhiMoeModel::from_weights, wrap: LoadedModel::PhiMoe } };
             MiniMax => { kind: Text, directory: ConfigBacked, weight: Some(WeightLoadRoute::ConfigBacked), adapter: None, config_backed: { dir_loader: models::MiniMaxModel::load, args: models::minimax::ModelArgs, weight_builder: models::MiniMaxModel::from_weights, wrap: LoadedModel::MiniMax } };
             MiniMaxM3 => { kind: Text, directory: ConfigBacked, weight: Some(WeightLoadRoute::ConfigBacked), adapter: None, config_backed: { dir_loader: models::MiniMaxM3Model::load, args: models::minimax_m3::ModelArgs, weight_builder: models::MiniMaxM3Model::from_weights, wrap: LoadedModel::MiniMaxM3 } };
+            MiniMaxM3VL => { kind: Vlm, directory: Vlm, weight: None, adapter: Some("MiniMax-M3-VL cannot be loaded with LoRA adapters yet") };
             Mixtral => { kind: Text, directory: ConfigBacked, weight: Some(WeightLoadRoute::ConfigBacked), adapter: None, config_backed: { dir_loader: models::MixtralModel::load, args: models::mixtral::ModelArgs, weight_builder: models::MixtralModel::from_weights, wrap: LoadedModel::Mixtral } };
             OLMoE => { kind: Text, directory: ConfigBacked, weight: Some(WeightLoadRoute::ConfigBacked), adapter: None, config_backed: { dir_loader: models::OlmoeModel::load, args: models::olmoe::ModelArgs, weight_builder: models::OlmoeModel::from_weights, wrap: LoadedModel::OLMoE } };
             DeepSeek => { kind: Text, directory: ConfigBacked, weight: Some(WeightLoadRoute::ConfigBacked), adapter: None, config_backed: { dir_loader: models::DeepSeekModel::load, args: models::deepseek::ModelArgs, weight_builder: models::DeepSeekModel::from_weights, wrap: LoadedModel::DeepSeek } };

src/models/detection.rs

@@ -160,6 +160,7 @@ pub fn get_model_type(model_path: &Path) -> Result<ModelType> {
         "phimoe" => Ok(ModelType::PhiMoe),
         "minimax" => Ok(ModelType::MiniMax),
         "minimax_m3" => Ok(ModelType::MiniMaxM3),
+        "minimax_m3_vl" => Ok(ModelType::MiniMaxM3VL),
         "gpt_oss" => Ok(ModelType::GptOss),
         "mixtral" => Ok(ModelType::Mixtral),
         "olmoe" => Ok(ModelType::OLMoE),

src/models/minimax_m3.rs

@@ -167,7 +167,30 @@ impl MiniMaxM3Model {
         caches: &mut [KVCache],
         mask: Option<&MlxArray>,
     ) -> UniquePtr<MlxArray> {
-        let mut h = self.embed_tokens.forward(input_ids);
+        self.forward_with_embeddings_impl(input_ids, None, caches, mask)
+    }
+
+    /// Token embedding lookup, exposed for the VL wrapper's LLaVA-style merge
+    /// (`src/vision/minimax_m3_vl.rs`).
+    pub fn get_embed_tokens(&self, input_ids: &MlxArray) -> UniquePtr<MlxArray> {
+        self.embed_tokens.forward(input_ids)
+    }
+
+    /// Decoder forward that optionally starts from precomputed input
+    /// embeddings. When `input_embeddings` is `Some`, the embedding lookup is
+    /// skipped and the provided (already vision-merged) embeddings are decoded
+    /// directly; `input_ids` is then only used for shape/consistency by callers.
+    pub fn forward_with_embeddings_impl(
+        &self,
+        input_ids: &MlxArray,
+        input_embeddings: Option<&MlxArray>,
+        caches: &mut [KVCache],
+        mask: Option<&MlxArray>,
+    ) -> UniquePtr<MlxArray> {
+        let mut h = match input_embeddings {
+            Some(embeds) => mlxcel_core::copy(embeds),
+            None => self.embed_tokens.forward(input_ids),
+        };
 
         for (i, layer) in self.layers.iter().enumerate() {
             h = layer.forward(&h, &mut caches[i], mask);