Repository navigation
[Question]How do I check whether kv cache int8 is effective? #394
Description
Currently I want to test the specific benchmark of weight only int8 + kvcache int8
The build command is as follows
python build.py --model_dir $model-path \
--dtype float16 \
--remove_input_padding \
--use_gpt_attention_plugin float16 \
--enable_context_fmha \
--use_gemm_plugin float16 \
--output_dir $output_path \
--use_inflight_batching \
--paged_kv_cache \
--world_size 2 \
--tp_size 2 \
--max_batch_size 128 \
--max_input_len 2048 \
--max_output_len 2048 \
--int8_kv_cache \
--use_weight_onlyHow should I check that kv cache int8 has taken effect?
In addition, I would also like to ask if I want to use my own data set for kv cache quantification, how should I do this? Is there any guidance document? Thanks~
Activity
wjj19950828 commentedon Nov 16, 2023
@juney-nvidia
Do you have any suggestions? Thanks~
wm2012011492 commentedon Nov 16, 2023
@wjj19950828 please check the verbose building log which shows whether kv-cache (past_key_value_0 and present_key_value_0) is quantized.
Layer(PluginV2): PLUGIN_V2_GPTAttention_1, Tactic: 0x0000000000000000, BloomForCausalLM/layers/0/attention/qkv/PLUGIN_V2_WeightOnlyQuantMatmul_0_output_0 (Half[-1,-1,6144]), Reformatted sequence_length (Int32[-1]), Reformatted host_past_key_value_lengths (Int32[-1]), Reformatted context_lengths (Int32[-1]), Reformatted cache_indirection (Int32[-1,1,-1]), Reformatted host_request_types (Int32[-1]), past_key_value_0 (Int8[-1,2,16,-1,128]), BloomForCausalLM/layers/0/attention/CONSTANT_1_output_0 (Float[1]), BloomForCausalLM/layers/0/attention/CONSTANT_2_output_0 (Float[1]), BloomForCausalLM/layers/0/attention/CONSTANT_0_output_0 (Half[1,16,1,1]), BloomForCausalLM/layers/0/attention/qkv/SHUFFLE_0_output_0 (Half[1,1,6144]) -> BloomForCausalLM/layers/0/attention/PLUGIN_V2_GPTAttention_1_output_0 (Half[-1,-1,2048]), present_key_value_0 (Int8[-1,2,16,-1,128])wjj19950828 commentedon Nov 16, 2023
@wm2012011492
Thank you for your reply~
If I want to use my own data set for calibration and collect the corresponding scale, how should I do this?
wjj19950828 commentedon Nov 17, 2023
@byshiue @wm2012011492
Hello, I currently measured some benchmark data of
weight only int8 + kv cache int8VSfp16at 70b tp=4, as follows:The build command is as above, modified to tp4
The environment is A100, CUDA 11.8
The test command is as follows：
mpirun --allow-run-as-root -n 4 ./benchmarks/gptSessionBenchmark \ --model llama \ --engine_dir $MODEL_PATH \ --batch_size "32;64" \ --input_output_len "1536,512;1024,1024;512,1536"The current fp16 data converted from latency to generation throughput is almost in line with the official data.
However, the performance of INT+KVQuant obviously does not meet expectations and is worse than FP16. Is this reasonable?
wm2012011492 commentedon Nov 17, 2023
The weight-only solution is more suitable for smaller batchsize*beamwidth. You're recommended to use INT8 SmoothQuant + INT8 kv-cache for large batch size as for your case.
wjj19950828 commentedon Nov 17, 2023
@wm2012011492
Thank you for your reply~
But why is A16W8 slower than A16W16 under large bs?
By the way, If i use SQ, how to use our own dataset?
wm2012011492 commentedon Nov 18, 2023
Because A16W8 also uses FP16 tensor core and has the overhead of W8->W16. When batch size is small, the GEMM kernel is memory-bounded and W8 helps save memory-loading time.
wm2012011492 commentedon Nov 18, 2023
Just change your dataset and following datapoint usage
jdemouth-nvidia commentedon Nov 23, 2023
I’m closing that issue as all the questions have been answered. Feel free to reopen if needed. Thanks.
wang90063 commentedon Dec 13, 2023
why is NT8 SmoothQuant + INT8 kv-cache is more suitable for large batch size