You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Is it possible to use a LoRA with a K-quantized model on GPU #936
I would've sworn it was possible at one point, but perhaps I misremembered it. Currently it doesn't work, as cuda get_rows and vulkan ggml_copy do not support K quants.
Speaking of which, do you know what's the best way to create all these gguf quants for image models?
I know the recent readmes in wan.md and qwen_image.md have links to GGUFs made by QuantFactory and Phil2Sat but I don't think they are using stable-diffusion.cpp itself to make the quants (especially since they offer k-quants as well).
Is there a pipeline or standard tool for this, or just private conversion scripts?
FWIW, mainline sd.cpp can create a few k-quants, even in cases they end up not working (like k-quants with LoRAs). But I believe most available ggufs follow that patched llama.cpp route.
Activity
stduhpf commentedon Nov 3, 2025
It works on CPU backend. Maybe it could make sense to add a way to apply LoRA on CPU backend and then move the model back to VRAM for inference?
leejet commentedon Nov 3, 2025
I'm currently refactoring the LoRA part. Once the refactoring is complete, it will support using LoRA with a K-quantized model.
LostRuins commentedon Nov 4, 2025
Speaking of which, do you know what's the best way to create all these gguf quants for image models?
I know the recent readmes in
wan.mdandqwen_image.mdhave links to GGUFs made by QuantFactory and Phil2Sat but I don't think they are using stable-diffusion.cpp itself to make the quants (especially since they offer k-quants as well).Is there a pipeline or standard tool for this, or just private conversion scripts?
stduhpf commentedon Nov 4, 2025
I think they're using an old patched version of llama.cpp (such a patch can be found there https://github.com/city96/ComfyUI-GGUF/tree/main/tools).
Maybe I should update my imatrix branch eventually, results seemed somewhat promising.
wbruna commentedon Nov 4, 2025
FWIW, mainline sd.cpp can create a few k-quants, even in cases they end up not working (like k-quants with LoRAs). But I believe most available ggufs follow that patched llama.cpp route.
LostRuins commentedon Nov 4, 2025
It would be good to formalize the conversion process sometime.
leejet commentedon Nov 11, 2025
This PR #969 should fix the issue.