Repository navigation
Conversation
|
This is a very interesting idea and makes me think of video compression. In video we can use a variable bitrate algorithm that allocates more bits to scenes with lots of detail and less bits for say a still image, all while targeting a preset birate. I'm just thinking here but maybe in the future we can consider performance as well and automatically juggle error and speed with some sort of slider like what they have for video. 
|
EAddario
changed the title
|
Sharing some like-for-like test results showing that this approach produces, in the majority of the cases, better quality models compared to naive quantisation (i.e. simply running standard To reduce the duration of the tests, I have chosen two small but representative models: Llama-3.2-1B ("classic" transformer architecture) and Huihui-MoE-1.2B-A0.6B (typical Mixture or Experts). The test protocol for each is:
Llama-3.2-1B results:
Huihui-MoE-1.2B-A0.6B results:
PPL: the smaller the better; 𝜌PPL: the higher the better; KLD: the smaller the better; In bold: best quality NoteAlthough these are very encouraging results, more testing with different model architectures and sizes will be required before categorically concluding this functionality consistently yields higher quality models. Comments, feedback and, in particular, bug reports are very much welcome |
|
This has an effect on the file size of the quants too, right? |
|
That's correct @ThiloteE, file size is directly and proportionally influenced by the chosen bpw |
|
Latest version introduces a new algorithm to select candidates resulting in models that are much closer to the bpw target and, in the large majority of the models tested, with better quality metrics than the equivalent naive version or very close: ERNIE-4.5-21B-A3B-PT
Huihui-MoE-1.2B-A0.6B
Huihui-MoE-5B-A1.7B-abliterated
Llama-3.1-8B
Llama-3.2-1B
NVIDIA-Nemotron-Nano-9B-v2
NVIDIA-Nemotron-Nano-12B-v2
Qwen3-14B
|
EAddario
changed the title
|
/bot review |
Automated code reviewReview of PR #15550:
|
|
Mostly valid points but a few missed the mark (e.g. point 3 is plainly wrong but I'll add a comment to clarify what's going on; on 15, I added the cast because CI failed somewhere, windows if I recall correctly; etc.). |
|
@pwilkin, issues flagged by the review bot should be addressed now. New |
|
ubuntu-webgpu has compile errors: FAILED: [code=1] src/CMakeFiles/llama.dir/llama-quant.cpp.obj
ccache /opt/hostedtoolcache/wasi-sdk/bin/clang++ --target=wasm32-wasip3 -DGGML_USE_CPU -DGGML_USE_WEBGPU -I/home/runner/work/llama.cpp/llama.cpp/src/. -I/home/runner/work/llama.cpp/llama.cpp/build-wasi/src -I/home/runner/work/llama.cpp/llama.cpp/src/../include -I/home/runner/work/llama.cpp/llama.cpp/ggml/src/../include -msimd128 -D_WASI_EMULATED_SIGNAL -fwasm-exceptions -mllvm -wasm-use-legacy-eh=false -O3 -DNDEBUG -Xclang -fno-pch-timestamp -Wmissing-declarations -Wmissing-noreturn -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wunreachable-code-break -Wunreachable-code-return -Wmissing-prototypes -Wextra-semi -MD -MT src/CMakeFiles/llama.dir/llama-quant.cpp.obj -MF src/CMakeFiles/llama.dir/llama-quant.cpp.obj.d -o src/CMakeFiles/llama.dir/llama-quant.cpp.obj -c /home/runner/work/llama.cpp/llama.cpp/src/llama-quant.cpp
/home/runner/work/llama.cpp/llama.cpp/src/llama-quant.cpp:51:62: warning: implicit conversion from 'unsigned long long' to 'const size_t' (aka 'const unsigned long') changes value from 8589934592 to 0 [-Wconstant-conversion]
51 | static const size_t LLAMA_QUANT_MAX_BUF_SIZE = 8ull*1024*1024*1024;
| ~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~^~~~~
/home/runner/work/llama.cpp/llama.cpp/src/llama-quant.cpp:2380:28: error: variable has incomplete type 'std::exception_ptr'
2380 | std::exception_ptr w_exception;
| ^
/opt/hostedtoolcache/wasi-sdk/bin/../share/wasi-sysroot/include/wasm32-wasip3/eh/c++/v1/__exception/operations.h:38:33: note: forward declaration of 'std::exception_ptr'
38 | class _LIBCPP_EXPORTED_FROM_ABI exception_ptr;
| ^
/home/runner/work/llama.cpp/llama.cpp/src/llama-quant.cpp:2410:39: error: calling 'current_exception' with incomplete return type 'exception_ptr'
2410 | w_exception = std::current_exception();
| ^~~~~~~~~~~~~~~~~~~~~~~~
/opt/hostedtoolcache/wasi-sdk/bin/../share/wasi-sysroot/include/wasm32-wasip3/eh/c++/v1/__exception/operations.h:40:59: note: 'current_exception' declared here
40 | [[__nodiscard__]] _LIBCPP_EXPORTED_FROM_ABI exception_ptr current_exception() _NOEXCEPT;
| ^
/opt/hostedtoolcache/wasi-sdk/bin/../share/wasi-sysroot/include/wasm32-wasip3/eh/c++/v1/__exception/operations.h:38:33: note: forward declaration of 'std::exception_ptr'
38 | class _LIBCPP_EXPORTED_FROM_ABI exception_ptr;
| ^
1 warning and 2 errors generated.
[75/81] Building CXX object src/CMakeFiles/llama.dir/llama-model-loader.cpp.obj
/home/runner/work/llama.cpp/llama.cpp/src/llama-model-loader.cpp:1107:57: warning: implicit conversion from 'unsigned long long' to 'const size_t' (aka 'const unsigned long') changes value from 4294967296 to 0 [-Wconstant-conversion]
1107 | constexpr size_t auto_min_size = 4ull * 1024 * 1024 * 1024;
| ~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~^~~~~~
1 warning generated.
[76/81] Building CXX object src/CMakeFiles/llama.dir/llama-sampler.cpp.obj |
|
Thanks for this work. I compared per-tensor cost estimates against a degrade-one ground truth (each block matrix alone in Q2_K, ΔKL vs BF16, everything else Q8_0) on Qwen3-1.7B and SmolLM2-1.7B. The per-tensor error estimated by this branch, read from its log, reaches a Spearman correlation of 0.45 / 0.44 with the measured ΔKL. An output-side cost (KL at the model output, Fisher approximation) reaches 0.93 / 0.95. With the same imatrix and the same file size, maps built from that cost have lower KL than Details, reproduction commands and the code: #31549 If it's useful, I'd be happy to help see whether an output-side cost could fit into |
|
Hi @RedOxer, I like the imatrix angle! it's an interesting approach which I think is useful, but in a different direction to This PR takes an alternative angle in that the lowest error recipe is estimated through measuring quant/dequant error in relation to each tensor compared to itself but in the previous layer. The upside is it doesn't take much CPU but the downside of course is that the resulting quality is not the same as an exhaustive search. IMO, producing a If you haven't yet, recommend to check out Thireus' GGUF-Tool-Suite and its companion website (it may be down momentarily) and Per-tensor layout maps for GGUF quantization |
|
Hello @EAddario ! One clarification on cost, since it's the main point of the approach: llamap doesn't search permutations or evaluate the model once per variant. It estimates every tensor × type cost from a single forward/backward pass on 512 calibration tokens (Fisher approximation, 128 samples), plus the in-memory quantization of each candidate. On an 8B that's ~15 min for 11 types, mostly spent quantizing candidates, then ~1 min per extra budget. For comparison, I'll look at Thireus' GGUF-Tool-Suite and add it to the related work. Thanks again! |
|
@pwilkin, anything else that's needed? |
This PR adds
target_bpw_type(), a function to determine an optimal per-tensor quantization mix to achieve a user-specified total file size (i.e.,--target-size 1.5g) or a global bits-per-weight (bpw) target (i.e.,--target-bpw 4.5678).An evaluation of 34 models across 238 tests shows that
target_bpw_type()optimization matched or exceeded the quality of standard quantization in 93% of cases, as indicated by lower KLD metrics.The function solves a constrained optimization problem to minimize quantization error, subject to a global size budget. It estimates per-tensor error for each layer, and dynamically allocates the bit budget where is needed the most.
High level flow:
--state-fileis set, target computations are saved to a file. If the quantization is interrupted (e.g., Ctrl+C), it can resume where it left off in the next run.Advantages
Target arbitrary size models
--target-size 23.85gwill generate a 23.85 GiB file to utilize the hardware fully.Data-driven mixed precision often can improve quality at fixed size
Allows better like-for-like comparisons between models and families
Standard quantization uses hardcoded rules like: "use Q4_K_M, except bump some tensors up/down, except fall back if incompatible, except keep some tensors unquantized..." and consquently, two different models quantized at the same Q4_K_M level can end up with very different bpw (e.g. 4.75 and 4.30).
Model performance generally scales with size; larger models typically outperform smaller ones. A model quantized with more bits will usually perform better (exhibiting lower perplexity and better evaluation scores) than a smaller version, even when the same underlying quantization method is used. This makes performance comparisons between models not a controlled experiment, as the models being compared have different effective compression ratios.
--target-bpwhelps to normalize experiments by forcing models to be quantized to a roughly equal overall byte budget. This standardization allows performance variations between models to be more accurately linked to underlying factors such as architectural or training differences, the effect of quantization error at the same compression level, or the decisions made by the optimizer regarding allocation.Disadvantages
Quantization process is significantly slower than standard
This approach can take 5x-10x longer as it quantizes a sample of most tensors into 15 different formats, dequantizes them back, computes error diffs, and selects the best size/error option that fits the target file size or global bpw budget.
However, the
--state-fileoption will save the above-mentioned computations to disk so that future quantizations can be generated at normal speed. It also allows to interrupt the computation process and resume it at a later time.The optimization target is only a proxy for the model's performance quality
An imatrix with activations data is required for best results
llama-imatrixcommit 0504396 or newer--target-bpwand--target-filewill refuse to run.Design considerations
The
target_bpw_type()function is implemented as a container for several lambdas providing all the logic for serialization, multithreading, math/stats, and optimization. Although there are clear downsides to this approach (i.e. cognitive load, testability and maintainability), given that all structs and helper lambdas are highly specific to this exact algorithm and have no reuse value elsewhere in the library, a self-contained "God function" seemed the best option to keep modifications to existing code to a minimum.Test results
Target BPW test results
238 tests: 195 (82%) WIN, 26 (11%) TIE, 16 (7%) LOSS, 1 NAAI usage disclosure
AI was used to validate the mathematical approach and calculations, and to optimize and debug the code.
Special thanks to @AesSedai, @am17an, @compilade, @ddh0 and @ngxson for their contributions during the development of this PR.