Will this repo support A10 in the future?

I saw that A10 is not listed as supported GPU in reader, may I ask why and if it's possible to support in the future?

A10 should be supported although it is not listed in readme. Please let us know if you encounter any issue on A10.

@Fangzhou-Ai

Hi @Fangzhou-Ai , as @byshiue said, we support the Ampere architecture. It’s just that our QA farm does not include A10 GPUs but the SM 8.6 CUDA architecture is supported. If you find bugs on A10, we will work with you to fix them. Thanks!

@jdemouth

@jdemouth hi, I'm trying to build the baichuan2-13b model on single A10 machine, when i run build.py, both fp16 and int8 mode suffers memory error, my command was:

python build.py --model_version v2_13b --model_dir ./Baichuan2-13B-Base/ --dtype float16 --use_gemm_plugin float16 --use_gpt_attention_plugin float16 --use_weight_only --output_dir ./tmp/baichuan_v2_13b/trt_engines/int8_weight_only/1-gpu/

Traceback (most recent call last):
File "/code/tensorrt_llm/examples/baichuan/build.py", line 477, in
build(0, args)
File "/code/tensorrt_llm/examples/baichuan/build.py", line 456, in build
serialize_engine(engine, os.path.join(args.output_dir, engine_name))
File "/code/tensorrt_llm/examples/baichuan/build.py", line 106, in serialize_engine
f.write(bytearray(engine))
MemoryError

and during the converting phase, i use the watch -n0.1 nvidia-smi and htop to observe both the gpu and cpu memory, both enough.could you please verify it, thx.

@77h2l

Hi @77h2l , we will see if we can reproduce the issue and provide you with a fix. Re-opening and making sure @byshiue is aware.

@77h2l

Hi @77h2l , we will see if we can reproduce the issue and provide you with a fix. Re-opening and making sure @byshiue is aware.

Thank you for your efficient response,just now I tried the int4 weight-only model convert and it works, but the int8 and fp16 failed, is there a least cpu memory required for model conversion? I noticed my max cpu mem was 56G, even if cpu offload it was enpugh to load the model weight.In addition, I repreduced the same issue as the #37 and I this this due to a bug related to the generation api about the eof charactor

BTW, baichuan-7b size model fp16 model converts also fails due to memory error, this is counterintuitive
I noticed that during the converting phase, the max gpu mem was 17540/23028 for an A10 machine, and later the gpu mem decreases down and return MemoryError. is there any offload trick not being used?

@77h2l

Hi, @77h2l, it looks you ask questions on baichuan-7b and baichuan-13b and it is a little confused. I am not sure which part you mention baichuan-7b and which part you want to mention baichuan-13b.

For baichuan-7b, can you share your scripts and the full error you encounter?

For baichuan-13b, it is expected OOM under FP16. Can you share your scripts and the full error you encounter under int8?

If possible, you can create another two bugs to track these two issues to prevent confusion.

@77h2l

Hi, @77h2l, it looks you ask questions on baichuan-7b and baichuan-13b and it is a little confused. I am not sure which part you mention baichuan-7b and which part you want to mention baichuan-13b.

For baichuan-7b, can you share your scripts and the full error you encounter?

For baichuan-13b, it is expected OOM under FP16. Can you share your scripts and the full error you encounter under int8?

If possible, you can create another two bugs to track these two issues to prevent confusion.

hi, @byshiue I have fixed both baichuan-7b and 13b model build phase to trt-llm. The only remained question is about the model accuracy after converted,as #37 mentioned both int8/int4 weight only model meets repeated charactor, I think that's due to config or parameter bug related to generation api.

another issue hope to reply is #73 seems lack of steps,thx.

@77h2l

Hi, @77h2l, it looks you ask questions on baichuan-7b and baichuan-13b and it is a little confused. I am not sure which part you mention baichuan-7b and which part you want to mention baichuan-13b.
For baichuan-7b, can you share your scripts and the full error you encounter?
For baichuan-13b, it is expected OOM under FP16. Can you share your scripts and the full error you encounter under int8?
If possible, you can create another two bugs to track these two issues to prevent confusion.

hi, @byshiue I have fixed both baichuan-7b and 13b model build phase to trt-llm. The only remained question is about the model accuracy after converted,as #37 mentioned both int8/int4 weight only model meets repeated charactor, I think that's due to config or parameter bug related to generation api.

Do you encounter accuracy issue on int8 weight only? Issue #37 only encountered problem on int4 weight only. Can you share your scripts to reproduce the issue?

@byshiue

@byshiue here is my build and run script:

python build.py --model_version v2_13b --model_dir ./Baichuan2-13B-Base/ --dtype float16 --use_gemm_plugin float16 --use_gpt_attention_plugin float16 --use_weight_only --output_dir ./tmp/baichuan_v2_13b/trt_engines/int8_weight_only/1-gpu/

python run.py --model_version v2_13b --max_output_len=500 --tokenizer_dir=./Baichuan2-13B-Base/ --engine_dir=./tmp/baichuan_v2_13b/trt_engines/int8_weight_only/1-gpu/ --input_text "世界上最高的山是"

and the output:

Input: "世界上最高的山是"
MSPT: 22.851788178655028 ...
Output: "珠穆朗玛峰,海拔8848米。
2、世界上海拔最高的山峰是珠穆朗玛峰,位于中国与尼泊尔边境上,它的主峰海拔高度8844.43米,是喜马拉雅山脉的主峰,同时也是世界上最高的山峰。
3、珠穆朗玛峰(Qomolangma),简称珠峰,又意译作圣母峰,尼泊尔称为萨加马塔峰,也叫“埃非勒斯峰”(Everest),位于中华人民共和国和尼泊尔交界的喜马拉雅山脉之
上,终年积雪。高度8844.43米,为世界第一高峰,中国最美的、令人震撼的十大名山之一。
4、珠穆朗玛峰山体呈巨型金字塔状,威武雄壮昂首天外,地形极端险峻,环境异常复杂。雪线高度:北坡为5800—6200米,南坡为5500—6100米。东北山脊、东南山脊和
西山山脊中间夹着三大陡壁(北壁、东壁和西南壁),在这些山脊和峭壁之间又分布着548条大陆型冰川,总面积达1457.07k㎡。
5、珠穆朗玛峰山体呈巨型金字塔状,威武雄壮昂首天外,地形极端险峻,环境异常复杂。雪线高度:北坡为5800—6200米,南坡为5500—6100米。东北山脊、东南山脊和
西山山脊中间夹着三大陡壁(北壁、东壁和西南壁),在这些山脊和峭壁之间又分布着548条大陆型冰川,总面积达1457.07k㎡。
6、珠穆朗玛峰山体呈巨型金字塔状,威武雄壮昂首天外,地形极端险峻,环境异常复杂。雪线高度:北坡为5800—6200米,南坡为5500—6100米。东北山脊、东南山脊和
西山山脊中间夹着三大陡壁(北壁、东壁和西南壁),在这些山脊和峭壁之间又分布着548条大陆型冰川,总面积达1457.0"

—— I this the later part of repeated token is a bit strange, it seems eos_token not work

It looks not a bug, but the model capability is little dropped.

You can print the output_ids and make sure it really does not generate eos_token.

@77h2l

@77h2l As we mentioned in #37, the accuracy drop on int4 is expected. It requires to use AWQ to keep the accuracy.

@byshiue

@byshiue hi actually I used the Base model to run the summarize task, I will take a try later to confirm my result, but anaway the int4 model acc drops a lot indeed.

I tested the Chat model of V1_13B, this is my data.

 Hugging Face (total latency: 154.34677147865295 sec)
 HF beam 0 result
   rouge1 : 10.597718234738085
   rouge2 : 2.4003784891871276
   rougeL : 7.611406867413062
   rougeLsum : 9.086245972389243

 TensorRT-LLM (total latency: 35.828752517700195 sec)
 TensorRT-LLM beam 0 result
   rouge1 : 25.385266398033444
   rouge2 : 7.640293733059095
   rougeL : 18.77161596008106
   rougeLsum : 21.086710024742416

This is a bit strange, but I observe the output and the output is normal. The only modification I made was when running HF, I added device_map='auto', torch_dtype=torch.float16 when loading the model, because I used 4090*2.

@byshiue

@byshiue hi actually I used the Base model to run the summarize task, I will take a try later to confirm my result, but anaway the int4 model acc drops a lot indeed.

I tested the Chat model of V1_13B, this is my data.
 Hugging Face (total latency: 154.34677147865295 sec)
 HF beam 0 result
   rouge1 : 10.597718234738085
   rouge2 : 2.4003784891871276
   rougeL : 7.611406867413062
   rougeLsum : 9.086245972389243

 TensorRT-LLM (total latency: 35.828752517700195 sec)
 TensorRT-LLM beam 0 result
   rouge1 : 25.385266398033444
   rouge2 : 7.640293733059095
   rougeL : 18.77161596008106
   rougeLsum : 21.086710024742416
This is a bit strange, but I observe the output and the output is normal. The only modification I made was when running HF, I added device_map='auto', torch_dtype=torch.float16 when loading the model, because I used 4090*2.

We don't verify HF's resuls under TP. So, we cannot gurantee its correctness on TP. You can add --log_level verbose to print results of summary to observe more example, but at least the score of trt llm looks good.