Repository navigation
Concretely which GeForce GPUs TensorRT-LLM supports. #146
Description
The installation says "The release supports GeForce 40-series GPUs."
Concretely which single NVidia GPUs are required (as a minimum requirement) and with their respective VRAM figures?
Only RTX 4090 RTX 4080 RTX 4070 Ti RTX 4070 RTX 4060 Ti RTX 4060 as stated in:
https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/
Are you planning to extend support for other more modest NVIDIA GPUs or via Google Colab?
In my case, I am able to run the text-generation-webui on local premises by tuning the execution of a Llama-2 7B LLM among a modest NVIDIA RTX Quadro 4GB VRAM and 64GB RAM laptop and achieving a throughput of up-to 5 Tokens/sec against a REST API.
Thanks.
Activity
jdemouth-nvidia commentedon Oct 27, 2023
Hi @cenguix ,
We do not have a concrete list of supported GPU boards.
TensorRT-LLM can be compiled for Volta, Turing, Ampere, Ada and Hopper GPU architectures. If your board has one of those architectures, it can be supported, irrespective of the fact that it's a GeForce or not. However, not all cards will be able to build all models or support all configurations. Some models require more GPU memory and will necessitate a lot of CPU RAM to be compiled. If you have a lower-end GPU.
I hope it helps.
Thanks,
Julien
cenguix commentedon Oct 27, 2023
Many thanks, Julien. My GPU is a Quadro T2000 4GB with Turing Architecture. If that is the case I will set up this weekend TensorRT-LLM and see how it goes.
Thanks again, Carlos