Repository navigation
[BUG]: Error loading the LLava model #1136
Description
Models:
https://huggingface.co/benxh/Qwen2.5-VL-7B-Instruct-GGUF
https://huggingface.co/KBlueLeaf/llama3-llava-next-8b-gguf (from here #897)
https://huggingface.co/second-state/Llava-v1.5-7B-GGUF
Error:
External component has thrown an exception.
System.Runtime.InteropServices.SEHException (0x80004005): External component has thrown an exception.
at LLama.Native.SafeLlavaModelHandle.clip_model_load(String mmProj, Int32 verbosity)
at LLama.Native.SafeLlavaModelHandle.LoadFromFile(String modelPath, Int32 verbosity)
at LLama.LLavaWeights.LoadFromFile(String mmProject)
What could be the problem?
I wanted to use a multimodal model to convert image2text.
Environment & Configuration
- Operating system: Windows
- .NET runtime version: NET 9.0.3
- LLamaSharp version: 0.23.0
- CUDA version (if you are using cuda backend): CPU
- CPU & GPU device: CPU
Reacted by Ondřej Kubíček
Activity
martindevans commentedon Mar 22, 2025
Does the model you're using work with the corresponding version of llama.cpp? Maybe also test in the previous version of LLamaSharp, to check for regressions.
aropb commentedon Mar 22, 2025
I'll try. Can you tell me if llava should work with any multimodal models?
I also noticed that the context is created only through llama weight, why is that?
aropb commentedon Mar 22, 2025
0.21.0 - the same mistake
aropb commentedon Mar 22, 2025
CPU
Yes, loaded!
SignalRT commentedon Mar 22, 2025
I tested the master with the model used in the unit test successfully.
Tested on Windows with CUDA.
aropb commentedon Mar 22, 2025
and on the CPU?
aropb commentedon Mar 22, 2025
Models/llava-v1.6-mistral-7b.Q3_K_XS.gguf ?
SignalRT commentedon Mar 22, 2025
As I explain I tested in GPU. The unit test are running successfully so it should work. I'm talking about Llava. Qwen-VL is not tested, and should not work.
aropb commentedon Mar 22, 2025
Qwen-VL?
aropb commentedon Mar 22, 2025
Does LLava not work on the CPU?
SignalRT commentedon Mar 22, 2025
Is that LlamaSharp documentation?
aropb commentedon Mar 22, 2025
mmproj-model-f16.gguf - works!
llava-v1.6-mistral-7b-Q5_K_S.gguf - dont works!
https://huggingface.co/bartowski/Qwen2-VL-7B-Instruct-GGUF - dont works!
aropb commentedon Mar 22, 2025
Yes
https://github.com/ggml-org/llama.cpp
aropb commentedon Mar 22, 2025
Should the LLava model be mmproj only?
Is it really impossible to use even Qwen2-VL for image2text?
Maybe I don't understand correctly how to work with VLM.
SignalRT commentedon Mar 22, 2025
You need both files. Check https://scisharp.github.io/LLamaSharp/0.23.0/QuickStart/ and the Llava example in the examples project
SignalRT commentedon Mar 22, 2025
That's llama.cpp documentation
That's llama.cpp documention. In the list of supported models in LlamaSharp documentation https://github.com/SciSharp/LLamaSharp qwen-vl is not in the list.
If I find some time I will test it.
aropb commentedon Mar 22, 2025
Thanks. Yes, I get it.
aropb commentedon Mar 23, 2025
I found model (main + mmproj):
https://huggingface.co/second-state/Qwen2-VL-7B-Instruct-GGUF
and here many mmproj models:
https://huggingface.co/koboldcpp/mmproj/tree/main
All models are loaded, but the output is not working. The models from the example work, but they are weak and old. I would like to try Qwen2-VL or gemma3 (it doesn't work yet, apparently a new version is needed llama.cpp, ggml-org/llama.cpp#12344).
SignalRT commentedon Mar 23, 2025
@aropb,
After conducting several tests and reviewing the current status of llama.cpp in relation to multimodal models, my understanding is as follows:
Regarding LlamaSharp, only Llava and similar models are currently compatible. I don’t believe that replicating the work done in qwen2vl-cli or gemma3-cli would be the best approach. Instead, I recommend waiting for llama.cpp to introduce a vision API and then updating LlamaSharp’s multimodal support to integrate with that API.
aropb commentedon Mar 23, 2025
Thanks.
Very useful information. I've been trying these models for the second day, really nothing works except llava :) And which of llava is considered the most recent and powerful?
Apparently it is, but it is weak by modern standards for Vision:
https://huggingface.co/xtuner/llava-llama-3-8b-v1_1-gguf
aropb commentedon Mar 24, 2025
I haven't found any information about this, can you show me where they discuss it?
I found:
ggml-org/llama.cpp#9687
ggml-org/llama.cpp#11292
SignalRT commentedon May 1, 2025
#1178 That's the information
aropb commentedon Jun 20, 2025
In llama.cpp does it look like it's ready?
ggml-org/llama.cpp#12898
https://github.com/ggml-org/llama.cpp/blob/master/docs/multimodal.md
github-actions commentedon Aug 20, 2025
This issue has been automatically marked as stale due to inactivity. If no further activity occurs, it will be closed in 7 days.