Repository navigation
server: Bring back multimodal support #8010
Description
Multimodal has been removed since #5882
Current llama.cpp multimodal roadmap
(update 9th april 2025)
mtmd(MulTi-MoDal) library (top prio 🔥 )- Implement
libmtmd: llava : introduce libmtmd #12849 - Support more models via
libmtmd(top prio 🔥 ) : mtmd : merge llava, gemma3 and minicpmv CLI into singlellama-mtmd-cli#13012 - Support M-RoPE models via
libmtmd(Qwen2VL, Qwen2.5VL) : mtmd : add qwen2vl and qwen2.5vl #13141 - Support audio input
- Use smart pointer in
clip.cppto avoid mem leak: clip : use smart pointer (⚠️ breaking change) #12869 -
Add wrapper for--> Probably don't need since we're already having some helper instb_imageto avoid polluting project with the big header filelibmtmdacting as wrapper for stb_image - Unify conversion scripts --> best case scenario: having
convert_hf_to_gguf.pythat can output both text + vision GGUF files --> introduced in convert : experimental support for--mmprojflag #13023 - Remove BOI / EOI token embeddings from clip.cpp (used by glm-edge): clip : remove boi/eoi embeddings for GLM-edge model (⚠️ breaking change) #13081
- Refactor documentations (find a way to reduce number of README files): llava : update documentations #13055
- Implement
- Implement
libmtmdin server API and server web UI (top prio 🔥 )- Publish first proposal: server : vision support via libmtmd #12898
- User can upload image from UI ( + drag-and-drop)
- Nice-to-have: Better KV caching strategy (TBD)
- Nice-to-have: allow loading remote image (may come with security risk)
- Update the security policy, make it clear that bugs related to 3rd party lib (like
stb_image) should be reported to upstream, not in llama.cpp
- Unify all vision CLI (like
minicpmv-cli,gemma3-cli, etc) into a single CLI - Add deprecation notice for
llava.h(we will remove libllava) andclip.h(clip is now internal-only) - Experimental support for audio input: (wip) support ultravox audio input #12745
- (far in the future) implement
llama_multimodalAPI that supports image, audio and more!
Activity
ZhenyaPav commentedon Jul 20, 2024
Can someone advise me on what is the latest (working) commit before multimodal was removed?
HAV0X1014 commentedon Jul 21, 2024
It is rather annoying that multimodal support was removed for the server and has not been re-implemented for such a long time now (4 months?). Multimodal LLMs and interleaved image and text models are growing in capability recently, and not being able to run models that used to work before is unfortunate. Seemingly, the only way to restore this functionality is to downgrade to a version that loses support for most new models and improvements.
I am not trying to demand multimodal/llava support to return, but show that this feature on the server is missed.
micoraweb commentedon Aug 16, 2024
Hello, is there still no multimodal support in llama-server ? According to ReadMe in LLaMA.cpp HTTP Server it should be supported?
How to use it with OpenAI API format?
its-ven commentedon Aug 22, 2024
Have there been any updates on this?
HAV0X1014 commentedon Aug 31, 2024
So far, it appears that there hasn't been any updates. This really stinks because there were updates to llava recently to support new models.
micoraweb commentedon Sep 12, 2024
So, this functionality seems to be unavailable for months and there is no hope to get it running? With all the amazing new models we could work with, such as MiniCPM, even Pixtral, etc. Can someone point us at a working server software that allows working with the newer multimodal models ? We just need something like llama-server that should run these multimodal GGUFs. Perhaps one of the llama.cpp files allows to run a server? It's also important to have a standard (OpenAI) API to support standard interactions.. It's so frustrating to wait months and months for such an important feature with no one even bothering to reply!
ggerganov commentedon Sep 12, 2024
Not much has changes since the issue was created. We need contributions to improve the existing vision code and people to maintain it. There is interest to reintroduce full multimodal support, but there are other things with higher priority that are currently worked upon by the core maintainers of the project.
ngxson commentedon Sep 12, 2024
Just to remind: Currently, llama-cpp-python has a server implementation that supports vision models (with OAI compat API). You can use it as an alternative.
Of course it's much better to bring vision support into llama.cpp itself (instead of staying as
llavaexample). The problem is that the current code requires a big clean up. We will eventually do that, as vision capability is becoming more and more streamline.chigkim commentedon Sep 26, 2024
@ggerganov, Meta released Llama-3.2 with multimodal capabilities. Does this affect the priority for core maintainers? I hope this question doesn’t come across as entitled...
ggerganov commentedon Sep 26, 2024
@chigkim My PoV is that adding multimodal support is a great opportunity for new people with good software architecture skills to get involved in the project. The general low to mid level patterns and details needed for the implementation are already available in the codebase - from model conversion, to data loading, backend usage and inference. It would take some high-level understanding of the project architecture in order to implement support for the vision models and extend the API in the correct way.
We really need more people with this sort of skillset, so at this point I feel it is better to wait and see if somebody will show up and take the opportunity to help out with the project long-term. Otherwise, I'm afraid we won't be able to sustain the quality of the project.
36 remaining items
AIWintermuteAI commentedon Mar 10, 2025
@ngxson I probably won't be sharing the .gguf files - the process to obtain them on my fork (or your branch) is simple enough and I don't have any good place to share them anyway.
But nice idea for future!
For benchmarks, I'm traveling this week and don't have Raspberry Pi with me. But I'll gladly run some next week once I'm back home. One thing I can tell for sure is that it times faster than onnx runtime. The 256M model image encoder alone took 35 seconds using onnx runtime (INT8 quantized model), while with llama.cpp it was closer to 5 seconds, maybe even less.
jkbe commentedon Mar 12, 2025
Some experimental support for using Gemma 3's vision was introduced here: #12344
ikcii commentedon Mar 13, 2025
Tested it just now and it works flawlessly, wish it was supported on the server...
BahamutRU commentedon Apr 24, 2025
What about Gemma 3 or Qwen2.5 VL vision in server? Very needed, don't want to use ollama.
Is it planned? Or should we not wait?
pwilkin commentedon Apr 24, 2025
@BahamutRU from what I understand, all the models that are currently available under llama-mtmd-cli will be available on the server once this PR is done.
sinand99 commentedon Apr 25, 2025
Pixtral 12B support for the server please.
Cerno-b commentedon May 8, 2025
Can I just say thank you for adding Qwen2.5-VL support to llama-mtmd-cli a short while ago. This came at precisely the perfect time for me.
Server mode would be amazing for that as well, but I wanted to contribute something positive instead of just demanding new stuff. Keep up the good work!
ngxson commentedon May 9, 2025
Server now supports image input, please have a look on this documentation
PR: #12898
AIWintermuteAI commentedon May 9, 2025
@ngxson , great work, will try it out shortly!
asif166037 commentedon Feb 16, 2026
Hi @ngxson , can we support multimodal on ios xCFramework as well.
day-dreaming-guy commentedon Apr 19, 2026
Hey @asif166037, did you manage to solve it somehow?
asif166037 commentedon Apr 20, 2026
Hi @day-dreaming-guy ,
Yes I have created the ios xCFramework that supports the multimodal. I created separate .sh file in that added existing command that present in llama.cpp repo (to create the xCFramework)
In that I have enable the LLAMA_BUILD_TOOLS=ON and created the libmtmd.a and attached in to the final dynamic framework
Since mtmd written in c++ internally but Swift/Objective-C can only consume pure C headers I copy pasted this headers in my plugin (ios/Sources/LlamaCppPlugin/mtmd-headers) and created the bridge to communicate with mtmd.
It wraps the C++ mtmd API behind extern "C" so Swift can call.
_void * cap_mtmd_init(...) // init vision/audio context
bool cap_mtmd_supports_vision(...)
bool cap_mtmd_supports_audio(...)
int32_t cap_mtmd_eval(...) // run multimodal inference
Swift → CapMtmdBridge.h (pure C) → mtmd.h (C++) → libmtmd.a (compiled into xcframework)_
I am working on ionic capacitor plugin and this is example -
#20057
day-dreaming-guy commentedon Apr 20, 2026
Awesome @asif166037 !! Thank you brother. This is really helpful!