perf: parallelize quantization #906

Activity

1. sw

   added

   performanceSpeed related topicsSpeed related topics

   on Apr 12, 2023
2. sw commented on Apr 12, 2023

   

   sw

   on Apr 12, 2023

   Contributor

   More actions

   @ikawrakow did that in #896, see kQuantizeQ4 in ggml_extra.cpp, but that's for a new quantization scheme. https://github.com/ggerganov/llama.cpp/blob/6bfb00a53b1a06e209f1b814356dd79ee96b89af/ggml_extra.cpp#L287-L291

   It did indeed speed things up. This could probably be integrated into llama_model_quantize_internal so that a new cpp module isn't necessary.
3. jon-chuang commented on Apr 12, 2023

   

   jon-chuang

   on Apr 12, 2023

   Last edited by jon-chuang

   ContributorAuthor

   More actions

   Is the new quantization scheme the one that minimizes MSE against the original weights?
4. sw commented on Apr 22, 2023

   

   sw

   on Apr 22, 2023

   Contributor

   More actions

   Resolved by #1075
5. sw

   closed this as completedon Apr 22, 2023
6. phuongncn

   added a commit that references this issue on Apr 28, 2026

   Bug fixes for completions and prompt caching in server (ggml-org#906)

   ...

   18f5a6c