Part of #493
Context
Multimodal / VLM models add vision or audio encoders and projectors that feed embeddings into the language model. The OpenXLA path is text-only today: the serve worker rejects multimodal input. Target set: Qwen2-VL / Qwen2.5-VL / Qwen3-VL, Gemma3n, Phi4MM, Molmo / Molmo2, Youtu-VL.
Scope
- Write a design doc covering encoder execution (running the vision/audio tower on the XLA/IREE path or alongside it), projector handling, embedding injection into the LM token stream, and the serve-path changes needed to accept multimodal requests.
- Then spin out a dedicated follow-up epic for implementation.
Acceptance criteria
Resolution
Design doc merged with a recorded decision to defer implementation. spike/openxla/MULTIMODAL_VLM_DESIGN.md covers all four deliverables the scope asks for, grounded in the current codebase: encoder execution (host-encode-first reusing the MLX vision stack, then an on-XLA ViT/SigLIP graph), projector handling, embedding injection into the LM token stream (the load-bearing emitter change: a prefill-from-embeddings graph entry, since the text graph only accepts token ids and gathers params['embed'] internally today), and the serve-path change to accept multimodal requests instead of rejecting them at admit. Implementation is a foundation-sized effort spanning the emitter, the IREE shim, the engine, and the serve worker, so it is scoped into the follow-up epic #566 rather than landed here.
Dependencies
None hard. Builds on the text-only LM path already in place. Spun out into follow-up epic #566.
Part of #493
Context
Multimodal / VLM models add vision or audio encoders and projectors that feed embeddings into the language model. The OpenXLA path is text-only today: the serve worker rejects multimodal input. Target set: Qwen2-VL / Qwen2.5-VL / Qwen3-VL, Gemma3n, Phi4MM, Molmo / Molmo2, Youtu-VL.
Scope
Acceptance criteria
spike/openxla/MULTIMODAL_VLM_DESIGN.md; recorded decision to DEFER; follow-up epic created: epic: OpenXLA multimodal / VLM architecture support (Qwen2-VL, Gemma3n, Phi4MM, Molmo, and more) #566.Resolution
Design doc merged with a recorded decision to defer implementation.
spike/openxla/MULTIMODAL_VLM_DESIGN.mdcovers all four deliverables the scope asks for, grounded in the current codebase: encoder execution (host-encode-first reusing the MLX vision stack, then an on-XLA ViT/SigLIP graph), projector handling, embedding injection into the LM token stream (the load-bearing emitter change: a prefill-from-embeddings graph entry, since the text graph only accepts token ids and gathersparams['embed']internally today), and the serve-path change to accept multimodal requests instead of rejecting them at admit. Implementation is a foundation-sized effort spanning the emitter, the IREE shim, the engine, and the serve worker, so it is scoped into the follow-up epic #566 rather than landed here.Dependencies
None hard. Builds on the text-only LM path already in place. Spun out into follow-up epic #566.