Skip to content

ModuleNotFoundError: No module named 'tensorrt_llm.bindings' #1194

New issue
New issue
Closed
Closed
ModuleNotFoundError: No module named 'tensorrt_llm.bindings'#1194
Assignees
MartinMarciniszyn
Labels
bugSomething isn't working

Description

@hellostronger
hellostronger
opened on Feb 29, 2024

System Info

a container from FROM nvidia/cuda:11.7.1-cudnn8-devel-ubuntu22.04

Who can help?

No response

Information

  • The official example scripts
  • My own modified scripts

Tasks

  • An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
  • My own task or dataset (give details below)

Reproduction

https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/executor.py line 11
import tensorrt_llm.bindings as tllm I do not understand tensorrt_llm.bindings
how can i fix the problem ModuleNotFoundError: No module named 'tensorrt_llm.bindings'

Expected behavior

ModuleNotFoundError: No module named 'tensorrt_llm.bindings' the problem not allert

actual behavior

ModuleNotFoundError: No module named 'tensorrt_llm.bindings'

additional notes

I con from annother resposity optimum-nvidia which use TensorRT-LLM, then i clone the main branch , then create a new python file just add
'''
from optimum.nvidia.pipelines import pipeline

then  it run and  the error  just like this
![image](https://github.com/NVIDIA/TensorRT-LLM/assets/48884346/0db237cc-d9a4-4cfc-b6d8-a14f951b2b94)

i  want to solve the proplem

Activity

  1. hellostronger
    added
    bugSomething isn't working
    on Feb 29, 2024
  2. byshiue
    assigned
    MartinMarciniszyn
    on Mar 5, 2024

  3. MartinMarciniszyn commented on Mar 7, 2024

    @MartinMarciniszyn
    MartinMarciniszyn
    on Mar 7, 2024
    Collaborator

    The bindings module will be found if you install the software following the instructions. Please provide the exact steps to reproduce the issue.

  4. ArlanCooper commented on Mar 12, 2024

    @ArlanCooper
    ArlanCooper
    on Mar 12, 2024

    how to solve it?

  5. ArlanCooper commented on Mar 12, 2024

    @ArlanCooper
    ArlanCooper
    on Mar 12, 2024

    i do the steps as instructions,as following:

    conda create -n tensorrt python==3.10.12
conda activate tensorrt
conda install mpi4py
pip install mpmath==1.3.0
pip install tensorrt_llm -U --extra-index-url https://pypi.nvidia.com
# Check installation
python -c "import tensorrt_llm"

    but it reports the errors as following:

    Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/powerop/work/rwq/TensorRT-LLM/tensorrt_llm/__init__.py", line 46, in <module>
    from .hlapi.llm import LLM, ModelConfig
  File "/home/powerop/work/rwq/TensorRT-LLM/tensorrt_llm/hlapi/__init__.py", line 1, in <module>
    from .llm import LLM, ModelConfig
  File "/home/powerop/work/rwq/TensorRT-LLM/tensorrt_llm/hlapi/llm.py", line 18, in <module>
    from ..executor import (GenerationExecutor, GenerationResult,
  File "/home/powerop/work/rwq/TensorRT-LLM/tensorrt_llm/executor.py", line 11, in <module>
    import tensorrt_llm.bindings as tllm
ModuleNotFoundError: No module named 'tensorrt_llm.bindings'

  6. MartinMarciniszyn commented on Mar 15, 2024

    @MartinMarciniszyn
    MartinMarciniszyn
    on Mar 15, 2024
    Collaborator

    We do not support installations in conda environments. This may work but it has not be validated by us. Hence, we cannot provide support for these kinds of installations. Please deploy with docker for full support of the functionality as explained in the installation instructions.

  7. MartinMarciniszyn
    closed this as completedon Mar 15, 2024

  8. Ednaordinary commented on Apr 27, 2024

    @Ednaordinary
    Ednaordinary
    on Apr 27, 2024

    Renaming .pyi files to .py in the bindings folder seemed to work in a python venv, not sure if it will break anything

    Actually! might be related to the next error I got:

    File "tensorrt_llm/runtime/model_runner_cpp.py", line 34, in <module>
    DataType.FLOAT: torch.float,
    ^^^^^^^^^^^^^^
AttributeError: type object 'DataType' has no attribute 'FLOAT'

  9. MarktHart commented on Apr 27, 2024

    @MarktHart
    MarktHart
    on Apr 27, 2024

    The bindings module will be found if you install the software following the instructions. Please provide the exact steps to reproduce the issue.

    For future reference:

    This is also a problem when following the steps when building a Docker image. You can solve it by setting ENV TORCH_CUDA_ARCH_LIST= .

  10. peterschmidt85 commented on Jun 23, 2024

    @peterschmidt85
    peterschmidt85
    on Jun 23, 2024

    I have the same issue even though I'm using the base CUDA 12.1 Docker image (without conda).

  11. MartinMarciniszyn commented on Jun 24, 2024

    @MartinMarciniszyn
    MartinMarciniszyn
    on Jun 24, 2024
    Collaborator

    @peterschmidt85 , please post the exact steps to reproduce the issue.

  12. peterschmidt85 commented on Jun 24, 2024

    @peterschmidt85
    peterschmidt85
    on Jun 24, 2024

    @peterschmidt85 , please post the exact steps to reproduce the issue.

    @MartinMarciniszyn,

    1. Run nvidia/cuda:12.1.0-devel-ubuntu22.04
    docker run --rm --runtime=nvidia --gpus all --entrypoint /bin/bash -it nvidia/cuda:12.1.0-devel-ubuntu22.04
    1. apt-get update && apt-get -y install python3.10 python3-pip openmpi-bin libopenmpi-dev
    2. pip3 install tensorrt_llm==0.9.0 -U --extra-index-url https://pypi.nvidia.com
    3. python3 -c "import tensorrt_llm"

    Fails with the problem that it is not compatible with NumPy 2.

    After pip install "numpy<2", I try it again, and it fails with

    ModuleNotFoundError: No module named 'tensorrt_llm.bindings'

    I was trying to install random versions of tensorrt and tensorrt_llm and once I got it working but later couldn't find how I did it.

  13. MartinMarciniszyn commented on Jun 24, 2024

    @MartinMarciniszyn
    MartinMarciniszyn
    on Jun 24, 2024
    Collaborator

    @peterschmidt85, please test this with the latest release 0.10.0.

  14. peterschmidt85 commented on Jun 24, 2024

    @peterschmidt85
    peterschmidt85
    on Jun 24, 2024

    5. python3 -c "import tensorrt_llm"

    Wow, 0.10.0 works. Thank you!

    Is there any reason why the older version stopped working? I supposed many pinned it and depend on it.

  15. MartinMarciniszyn commented on Jun 24, 2024

    @MartinMarciniszyn
    MartinMarciniszyn
    on Jun 24, 2024
    Collaborator

    I suspect this might be due to some Python dependencies that were not correctly pinned and evolved in the meanwhile. However, this is just a guess.

  16. peterschmidt85 commented on Jun 24, 2024

    @peterschmidt85
    peterschmidt85
    on Jun 24, 2024

    @MartinMarciniszyn Should I use nvcr.io/nvidia/tritonserver:24.05-trtllm-python-py3 with v0.10.0?
    According to https://github.com/triton-inference-server/triton_cli, they won't work together.

  17. MartinMarciniszyn commented on Jun 25, 2024

    @MartinMarciniszyn
    MartinMarciniszyn
    on Jun 25, 2024
    Collaborator

    Please see the Triton documentation. If it does not work out of the box, you might need to rebuild the backend for 0.10 from source.

  18. nhduong1203 commented on Feb 12, 2025

    @nhduong1203
    nhduong1203
    on Feb 12, 2025

    Try to not run python3 -c "import tensorrt_llm" in TensorRT-LLM dicrectory, because it's already have a folder name tensorrt_llm and will cause the error.

    Take me about hours to realize :(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions