Skip to content

executor failed running [/bin/bash -c bash ./install_tensorrt.sh && rm install_tensorrt.sh]: exit code: 2 #434

New issue
New issue
Closed
Closed
executor failed running [/bin/bash -c bash ./install_tensorrt.sh && rm install_tensorrt.sh]: exit code: 2#434
Assignees
byshiueShixiaowei02
Labels
triagedIssue has been triaged by maintainers

Description

@changer-zxy
changer-zxy
opened on Nov 20, 2023

make -C docker release_build
make: Entering directory '/data/yankai/TensorRT-LLM/docker'
Building docker image: tensorrt_llm/release:latest
DOCKER_BUILDKIT=1 docker build --pull
--progress auto
--build-arg BASE_IMAGE=nvcr.io/nvidia/pytorch
--build-arg BASE_TAG=23.08-py3
--build-arg BUILD_WHEEL_ARGS="--clean --trt_root /usr/local/tensorrt"
--build-arg TORCH_INSTALL_TYPE="skip"
--target release
--file Dockerfile.multi
--tag tensorrt_llm/release:latest
..
[+] Building 1026.3s (11/30)
=> [internal] load build definition from Dockerfile.multi 0.2s
=> => transferring dockerfile: 44B 0.0s
=> [internal] load .dockerignore 0.3s
=> => transferring context: 35B 0.0s
=> [internal] load metadata for nvcr.io/nvidia/pytorch:23.08-py3 3.0s
=> [internal] load build context 0.7s
=> => transferring context: 647.23kB 0.3s
=> [base 1/1] FROM nvcr.io/nvidia/pytorch:23.08-py3@sha256:12a39f22d6e3a3cfcb285a238b6219475181672ff41a557a75bdeeef6d630740 13.9s
=> => resolve nvcr.io/nvidia/pytorch:23.08-py3@sha256:12a39f22d6e3a3cfcb285a238b6219475181672ff41a557a75bdeeef6d630740 0.1s
=> [devel 1/10] COPY docker/common/install_base.sh install_base.sh 0.3s
=> [devel 2/10] RUN bash ./install_base.sh && rm install_base.sh 88.4s
=> [devel 3/10] COPY docker/common/install_cmake.sh install_cmake.sh 0.5s
=> [devel 4/10] RUN bash ./install_cmake.sh && rm install_cmake.sh 577.6s
=> [devel 5/10] COPY docker/common/install_tensorrt.sh install_tensorrt.sh 0.6s
=> ERROR [devel 6/10] RUN bash ./install_tensorrt.sh && rm install_tensorrt.sh 341.0s

[devel 6/10] RUN bash ./install_tensorrt.sh && rm install_tensorrt.sh:
#11 0.710 ++ grep -oP '(?<=^ID=).+' /etc/os-release
#11 0.711 ++ tr -d '"'
#11 0.714 + ID=ubuntu
#11 0.714 + case "$ID" in
#11 0.714 + install_ubuntu_requirements
#11 0.714 + CUDNN_VERSION=8
#11 0.714 + apt-get update
#11 1.336 Get:1 http://archive.ubuntu.com/ubuntu jammy InRelease [270 kB]
#11 1.401 Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
#11 3.027 Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
#11 3.058 Get:4 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1010 kB]
#11 3.404 Get:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB]
#11 3.774 Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages [17.5 MB]
#11 4.534 Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1203 kB]
#11 4.875 Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1404 kB]
#11 5.158 Get:9 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [44.0 kB]
#11 7.525 Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages [1792 kB]
#11 7.778 Get:11 http://archive.ubuntu.com/ubuntu jammy/restricted amd64 Packages [164 kB]
#11 7.783 Get:12 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 Packages [266 kB]
#11 7.792 Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1473 kB]
#11 8.758 Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1278 kB]
#11 8.775 Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [49.8 kB]
#11 8.776 Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1430 kB]
#11 9.105 Get:17 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [78.3 kB]
#11 9.111 Get:18 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [32.6 kB]
#11 9.205 Fetched 28.3 MB in 8s (3352 kB/s)
#11 9.205 Reading package lists...
#11 9.819 + apt-get install -y --no-install-recommends libcudnn8 libcudnn8-dev libnccl-dev
#11 9.874 Reading package lists...
#11 10.45 Building dependency tree...
#11 10.58 Reading state information...
#11 10.71 libcudnn8 is already the newest version (8.9.4.25-1+cuda12.2).
#11 10.71 libcudnn8-dev is already the newest version (8.9.4.25-1+cuda12.2).
#11 10.71 libnccl-dev is already the newest version (2.18.3-1+cuda12.2).
#11 10.71 0 upgraded, 0 newly installed, 0 to remove and 47 not upgraded.
#11 10.71 + apt-get clean
#11 10.72 + rm -rf /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy-backports_InRelease /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy-backports_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy-backports_universe_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy-updates_InRelease /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy-updates_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy-updates_multiverse_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy-updates_restricted_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy-updates_universe_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy_InRelease /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy_multiverse_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy_restricted_binary-amd64_Packages.lz4 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy_universe_binary-amd64_Packages.lz4 /var/lib/apt/lists/auxfiles /var/lib/apt/lists/lock /var/lib/apt/lists/partial /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_jammy-security_InRelease /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_jammy-security_main_binary-amd64_Packages.lz4 /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_jammy-security_multiverse_binary-amd64_Packages.lz4 /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_jammy-security_restricted_binary-amd64_Packages.lz4 /var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_jammy-security_universe_binary-amd64_Packages.lz4
#11 10.74 + install_tensorrt
#11 10.74 + TENSOR_RT_VERSION=9.1.0.4
#11 10.74 + CUDA_VERSION=12.2
#11 10.74 ++ python -c 'import sys; print(".".join(map(str, sys.version_info[0:2])))'
#11 10.78 + PY_VERSION=3.10
#11 10.78 ++ echo 310
#11 10.78 + PARSED_PY_VERSION=310
#11 10.78 + '[' -z '' ']'
#11 10.78 + ARCH=amd64
#11 10.78 + '[' -z amd64 ']'
#11 10.78 + '[' amd64 = arm64 ']'
#11 10.78 + '[' amd64 = amd64 ']'
#11 10.78 + ARCH=x86_64
#11 10.78 + '[' x86_64 = x86_64 ']'
#11 10.78 + DIR_NAME=x64-agnostic
#11 10.78 + '[' x86_64 = aarch64 ']'
#11 10.78 + OS=linux
#11 10.78 + RELEASE_URL_TRT=https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/secure/9.1.0/tars/tensorrt-9.1.0.4.linux.x86_64-gnu.cuda-12.2.tar.gz
#11 10.78 + wget --no-verbose https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/secure/9.1.0/tars/tensorrt-9.1.0.4.linux.x86_64-gnu.cuda-12.2.tar.gz -O /tmp/TensorRT.tar
#11 320.5 2023-11-20 03:34:10 URL:https://developer.download.nvidia.cn/403.html [252/252] -> "/tmp/TensorRT.tar" [1]
#11 320.5 + tar -xf /tmp/TensorRT.tar -C /usr/local/
#11 337.8
#11 337.8 gzip: stdin: unexpected end of file
#11 337.8 tar: Unexpected EOF in archive
#11 337.8 tar: Unexpected EOF in archive
#11 337.8 tar: Error is not recoverable: exiting now

executor failed running [/bin/bash -c bash ./install_tensorrt.sh && rm install_tensorrt.sh]: exit code: 2
Makefile:46: recipe for target 'release_build' failed
make: *** [release_build] Error 1

This error occurs when building docker, how to solve it?

Activity

  1. byshiue commented on Nov 20, 2023

    @byshiue
    byshiue
    on Nov 20, 2023
    Collaborator

    Can you try running last few lines locally?

    #11 10.78 + wget --no-verbose https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/secure/9.1.0/tars/tensorrt-9.1.0.4.linux.x86_64-gnu.cuda-12.2.tar.gz -O /tmp/TensorRT.tar
    #11 320.5 2023-11-20 03:34:10 URL:https://developer.download.nvidia.cn/403.html [252/252] -> "/tmp/TensorRT.tar" [1]
    #11 320.5 + tar -xf /tmp/TensorRT.tar -C /usr/local/

  2. byshiue
    self-assigned this
    on Nov 20, 2023
  3. byshiue
    added
    triagedIssue has been triaged by maintainers
    on Nov 20, 2023

  4. changer-zxy commented on Nov 20, 2023

    @changer-zxy
    changer-zxy
    on Nov 20, 2023
    Author

    Can you try running last few lines locally?

    #11 10.78 + wget --no-verbose https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/secure/9.1.0/tars/tensorrt-9.1.0.4.linux.x86_64-gnu.cuda-12.2.tar.gz -O /tmp/TensorRT.tar
    #11 320.5 2023-11-20 03:34:10 URL:https://developer.download.nvidia.cn/403.html [252/252] -> "/tmp/TensorRT.tar" [1]
    #11 320.5 + tar -xf /tmp/TensorRT.tar -C /usr/local/

    After I re-run it several times, this problem no longer occurs, but the following problem appears again...

    => ERROR [wheel 9/9] RUN python3 scripts/build_wheel.py --clean --trt_root /usr/local/tensorrt 475.1s

    [wheel 9/9] RUN python3 scripts/build_wheel.py --clean --trt_root /usr/local/tensorrt:
    #25 1.232 Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com, https://pypi.ngc.nvidia.com
    #25 2.703 Collecting build (from -r requirements.txt (line 1))
    #25 2.704 Obtaining dependency information for build from https://files.pythonhosted.org/packages/93/dd/b464b728b866aaa62785a609e0dd8c72201d62c5f7c53e7c20f4dceb085f/build-1.0.3-py3-none-any.whl.metadata
    #25 3.391 Downloading build-1.0.3-py3-none-any.whl.metadata (4.2 kB)
    #25 3.394 Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 2)) (2.1.0a0+29c30b1)
    #25 4.499 Collecting transformers==4.31.0 (from -r requirements.txt (line 3))
    #25 4.499 Obtaining dependency information for transformers==4.31.0 from https://files.pythonhosted.org/packages/21/02/ae8e595f45b6c8edee07913892b3b41f5f5f273962ad98851dc6a564bbb9/transformers-4.31.0-py3-none-any.whl.metadata
    #25 4.713 Downloading transformers-4.31.0-py3-none-any.whl.metadata (116 kB)
    #25 5.162 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 116.9/116.9 kB 268.3 kB/s eta 0:00:00
    #25 5.718 Collecting diffusers==0.15.0 (from -r requirements.txt (line 4))
    #25 5.943 Downloading diffusers-0.15.0-py3-none-any.whl (851 kB)
    #25 6.627 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 851.8/851.8 kB 1.3 MB/s eta 0:00:00
    #25 7.224 Collecting accelerate==0.20.3 (from -r requirements.txt (line 5))
    #25 7.225 Obtaining dependency information for accelerate==0.20.3 from https://files.pythonhosted.org/packages/10/d3/5382aa337d3e67214003a17b06bfc07cf0334356b4e8aaf3b12b0d38c83f/accelerate-0.20.3-py3-none-any.whl.metadata
    #25 7.443 Downloading accelerate-0.20.3-py3-none-any.whl.metadata (17 kB)
    #25 8.068 Collecting colored (from -r requirements.txt (line 6))
    #25 8.068 Obtaining dependency information for colored from https://files.pythonhosted.org/packages/6f/0d/a10351ef1a98e0b03d66887ec2d87c261f9a0fbff8f2bdb75614cc0a2850/colored-2.2.3-py3-none-any.whl.metadata
    #25 8.291 Downloading colored-2.2.3-py3-none-any.whl.metadata (3.6 kB)
    #25 8.293 Requirement already satisfied: polygraphy in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 7)) (0.48.1)
    #25 8.295 Requirement already satisfied: onnx>=1.12.0 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 8)) (1.14.0)
    #25 8.296 Requirement already satisfied: mpi4py in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 9)) (3.1.5)
    #25 8.298 Requirement already satisfied: tensorrt>=8.6.0 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 10)) (9.1.0.post12.dev4)
    #25 8.299 Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 11)) (1.22.2)
    #25 9.106 Collecting cuda-python==12.2.0 (from -r requirements.txt (line 12))
    #25 9.106 Obtaining dependency information for cuda-python==12.2.0 from https://files.pythonhosted.org/packages/2b/5c/ad3b6bee78f95134834dc2ce8805fc12ac70d7a9ba065e2d7c05263549dc/cuda_python-12.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
    #25 9.303 Downloading cuda_python-12.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (749 bytes)
    #25 9.904 Collecting sentencepiece>=0.1.99 (from -r requirements.txt (line 13))
    #25 10.14 Downloading sentencepiece-0.1.99-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
    #25 10.94 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 1.6 MB/s eta 0:00:00
    #25 10.95 Requirement already satisfied: wheel in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 14)) (0.41.1)
    #25 11.81 Collecting lark (from -r requirements.txt (line 15))
    #25 11.81 Obtaining dependency information for lark from https://files.pythonhosted.org/packages/99/ca/f3532a61dce7dd52fbd38737a12e16cdc7699697e23287eb7addfdd93e3f/lark-1.1.8-py3-none-any.whl.metadata
    #25 12.01 Downloading lark-1.1.8-py3-none-any.whl.metadata (1.9 kB)
    #25 12.38 Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers==4.31.0->-r requirements.txt (line 3)) (3.12.2)
    #25 13.04 Collecting huggingface-hub<1.0,>=0.14.1 (from transformers==4.31.0->-r requirements.txt (line 3))
    #25 13.04 Obtaining dependency information for huggingface-hub<1.0,>=0.14.1 from https://files.pythonhosted.org/packages/05/09/1945ca6ba3ad8ad6e2872ba682ce8d68c5e63c8e55458ed8ab4885709f1d/huggingface_hub-0.19.4-py3-none-any.whl.metadata
    #25 13.24 Downloading huggingface_hub-0.19.4-py3-none-any.whl.metadata (14 kB)
    #25 13.24 Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers==4.31.0->-r requirements.txt (line 3)) (23.1)
    #25 13.25 Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers==4.31.0->-r requirements.txt (line 3)) (6.0.1)
    #25 13.25 Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.31.0->-r requirements.txt (line 3)) (2023.6.3)
    #25 13.25 Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers==4.31.0->-r requirements.txt (line 3)) (2.31.0)
    #25 14.20 Collecting tokenizers!=0.11.3,<0.14,>=0.11.1 (from transformers==4.31.0->-r requirements.txt (line 3))
    #25 14.40 Downloading tokenizers-0.13.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)
    #25 18.31 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.8/7.8 MB 2.0 MB/s eta 0:00:00
    #25 19.37 Collecting safetensors>=0.3.1 (from transformers==4.31.0->-r requirements.txt (line 3))
    #25 19.37 Obtaining dependency information for safetensors>=0.3.1 from https://files.pythonhosted.org/packages/20/4e/878b080dbda92666233ec6f316a53969edcb58eab1aa399a64d0521cf953/safetensors-0.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
    #25 19.57 Downloading safetensors-0.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.8 kB)
    #25 19.58 Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers==4.31.0->-r requirements.txt (line 3)) (4.65.0)
    #25 19.66 Requirement already satisfied: Pillow in /usr/local/lib/python3.10/dist-packages (from diffusers==0.15.0->-r requirements.txt (line 4)) (9.2.0)
    #25 19.67 Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.10/dist-packages (from diffusers==0.15.0->-r requirements.txt (line 4)) (6.8.0)
    #25 19.70 Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate==0.20.3->-r requirements.txt (line 5)) (5.9.4)
    #25 19.70 Requirement already satisfied: cython in /usr/local/lib/python3.10/dist-packages (from cuda-python==12.2.0->-r requirements.txt (line 12)) (3.0.0)
    #25 20.18 Collecting pyproject_hooks (from build->-r requirements.txt (line 1))
    #25 20.41 Downloading pyproject_hooks-1.0.0-py3-none-any.whl (9.3 kB)
    #25 20.41 Requirement already satisfied: tomli>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from build->-r requirements.txt (line 1)) (2.0.1)
    #25 20.43 Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch->-r requirements.txt (line 2)) (4.7.1)
    #25 20.43 Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch->-r requirements.txt (line 2)) (1.12)
    #25 20.43 Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch->-r requirements.txt (line 2)) (2.6.3)
    #25 20.43 Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch->-r requirements.txt (line 2)) (3.1.2)
    #25 20.43 Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch->-r requirements.txt (line 2)) (2023.6.0)
    #25 20.44 Requirement already satisfied: protobuf>=3.20.2 in /usr/local/lib/python3.10/dist-packages (from onnx>=1.12.0->-r requirements.txt (line 8)) (4.21.12)
    #25 20.65 Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.10/dist-packages (from importlib-metadata->diffusers==0.15.0->-r requirements.txt (line 4)) (3.16.2)
    #25 20.66 Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch->-r requirements.txt (line 2)) (2.1.3)
    #25 20.70 Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.31.0->-r requirements.txt (line 3)) (3.2.0)
    #25 20.70 Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.31.0->-r requirements.txt (line 3)) (3.4)
    #25 20.70 Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.31.0->-r requirements.txt (line 3)) (1.26.16)
    #25 20.70 Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.31.0->-r requirements.txt (line 3)) (2023.7.22)
    #25 20.71 Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch->-r requirements.txt (line 2)) (1.3.0)
    #25 20.95 Downloading transformers-4.31.0-py3-none-any.whl (7.4 MB)
    #25 27.01 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.4/7.4 MB 1.2 MB/s eta 0:00:00
    #25 27.21 Downloading accelerate-0.20.3-py3-none-any.whl (227 kB)
    #25 27.37 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 227.6/227.6 kB 1.5 MB/s eta 0:00:00
    #25 27.58 Downloading cuda_python-12.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (20.1 MB)
    #25 57.51 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.1/20.1 MB 469.8 kB/s eta 0:00:00
    #25 57.71 Downloading build-1.0.3-py3-none-any.whl (18 kB)
    #25 57.94 Downloading colored-2.2.3-py3-none-any.whl (16 kB)
    #25 58.16 Downloading lark-1.1.8-py3-none-any.whl (111 kB)
    #25 58.42 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 111.6/111.6 kB 456.8 kB/s eta 0:00:00
    #25 58.62 Downloading huggingface_hub-0.19.4-py3-none-any.whl (311 kB)
    #25 59.35 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 311.7/311.7 kB 425.5 kB/s eta 0:00:00
    #25 59.55 Downloading safetensors-0.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
    #25 63.16 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 372.8 kB/s eta 0:00:00
    #25 64.94 Installing collected packages: tokenizers, sentencepiece, safetensors, pyproject_hooks, lark, cuda-python, colored, huggingface-hub, build, transformers, diffusers, accelerate
    #25 65.29 Attempting uninstall: cuda-python
    #25 65.29 Found existing installation: cuda-python 12.1.0rc5+1.g994d8d0
    #25 65.30 Uninstalling cuda-python-12.1.0rc5+1.g994d8d0:
    #25 65.35 Successfully uninstalled cuda-python-12.1.0rc5+1.g994d8d0
    #25 69.24 Successfully installed accelerate-0.20.3 build-1.0.3 colored-2.2.3 cuda-python-12.2.0 diffusers-0.15.0 huggingface-hub-0.19.4 lark-1.1.8 pyproject_hooks-1.0.0 safetensors-0.4.0 sentencepiece-0.1.99 tokenizers-0.13.3 transformers-4.31.0
    #25 69.24 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
    #25 70.60
    #25 70.60 [notice] A new release of pip is available: 23.2.1 -> 23.3.1
    #25 70.60 [notice] To update, run: python3 -m pip install --upgrade pip
    #25 71.35 -- The CXX compiler identification is GNU 11.4.0
    #25 71.36 -- Detecting CXX compiler ABI info
    #25 71.48 -- Detecting CXX compiler ABI info - done
    #25 71.49 -- Check for working CXX compiler: /usr/bin/c++ - skipped
    #25 71.49 -- Detecting CXX compile features
    #25 71.49 -- Detecting CXX compile features - done
    #25 71.49 -- NVTX is disabled
    #25 71.49 -- Importing batch manager
    #25 71.49 -- Building PyTorch
    #25 71.49 -- Building Google tests
    #25 71.49 -- Building benchmarks
    #25 71.49 -- Looking for a CUDA compiler
    #25 73.38 -- Looking for a CUDA compiler - /usr/local/cuda/bin/nvcc
    #25 73.38 -- CUDA compiler: /usr/local/cuda/bin/nvcc
    #25 73.41 -- GPU architectures: 70-real;80-real;86-real;89-real;90-real
    #25 74.39 -- The CUDA compiler identification is NVIDIA 12.2.128
    #25 74.40 -- Detecting CUDA compiler ABI info
    #25 76.54 -- Detecting CUDA compiler ABI info - done
    #25 76.58 -- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc - skipped
    #25 76.58 -- Detecting CUDA compile features
    #25 76.58 -- Detecting CUDA compile features - done
    #25 76.59 -- Found CUDAToolkit: /usr/local/cuda/include (found version "12.2.128")
    #25 76.61 -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
    #25 76.73 -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
    #25 76.73 -- Found Threads: TRUE
    #25 76.77 -- ========================= Importing and creating target nvinfer ==========================
    #25 76.77 -- Looking for library nvinfer
    #25 76.77 -- Library that was found /usr/local/tensorrt/targets/x86_64-linux-gnu/lib/libnvinfer.so
    #25 76.77 -- ==========================================================================================
    #25 76.77 -- ========================= Importing and creating target nvuffparser ==========================
    #25 76.77 -- Looking for library nvparsers
    #25 76.77 -- Library that was found nvparsers_LIB_PATH-NOTFOUND
    #25 76.77 -- ==========================================================================================
    #25 76.77 -- CUDAToolkit_VERSION 12.2 is greater or equal than 11.0, enable -DENABLE_BF16 flag
    #25 76.77 -- CUDAToolkit_VERSION 12.2 is greater or equal than 11.8, enable -DENABLE_FP8 flag
    #25 77.20 -- Found MPI_CXX: /opt/hpcx/ompi/lib/libmpi.so (found version "3.1")
    #25 77.20 -- Found MPI: TRUE (found version "3.1")
    #25 77.20 -- COMMON_HEADER_DIRS: /src/tensorrt_llm/cpp;/usr/local/cuda/include
    #25 77.20 -- TORCH_CUDA_ARCH_LIST: 7.0;8.0;8.6;8.9;9.0
    #25 77.20 CMake Warning at CMakeLists.txt:248 (message):
    #25 77.20 Ignoring environment variable TORCH_CUDA_ARCH_LIST=5.2 6.0 6.1 7.0 7.5 8.0
    #25 77.20 8.6 9.0+PTX
    #25 77.20
    #25 77.20
    #25 77.60 -- Found Python3: /usr/bin/python3.10 (found version "3.10.12") found components: Interpreter Development Development.Module Development.Embed
    #25 77.61 -- Found Python executable at /usr/bin/python3.10
    #25 77.61 -- Found Python libraries at /usr/lib/x86_64-linux-gnu
    #25 80.61 -- Found CUDA: /usr/local/cuda (found version "12.2")
    #25 80.61 -- Found CUDAToolkit: /usr/local/cuda/include (found version "12.2.128")
    #25 80.69 -- Caffe2: CUDA detected: 12.2
    #25 80.69 -- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
    #25 80.69 -- Caffe2: CUDA toolkit directory: /usr/local/cuda
    #25 80.89 -- Caffe2: Header version is: 12.2
    #25 81.05 -- /usr/local/cuda/lib64/libnvrtc.so shorthash is eaa826f0
    #25 81.05 -- USE_CUDNN is set to 0. Compiling without cuDNN support
    #25 81.06 -- Added CUDA NVCC flags for: -gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_89,code=sm_89;-gencode;arch=compute_90,code=sm_90
    #25 81.09 CMake Warning at /usr/local/lib/python3.10/dist-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
    #25 81.09 static library kineto_LIBRARY-NOTFOUND not found.
    #25 81.09 Call Stack (most recent call first):
    #25 81.09 /usr/local/lib/python3.10/dist-packages/torch/share/cmake/Torch/TorchConfig.cmake:127 (append_torchlib_if_found)
    #25 81.09 CMakeLists.txt:281 (find_package)
    #25 81.09
    #25 81.09
    #25 81.09 -- Found Torch: /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch.so
    #25 81.09 -- TORCH_CXX_FLAGS: -D_GLIBCXX_USE_CXX11_ABI=1
    #25 81.09 -- Building for TensorRT version: 9.1.0, library version: 9
    #25 81.11 -- Using MPI_CXX_INCLUDE_DIRS: /opt/hpcx/ompi/include;/opt/hpcx/ompi/include/openmpi;/opt/hpcx/ompi/include/openmpi/opal/mca/hwloc/hwloc201/hwloc/include;/opt/hpcx/ompi/include/openmpi/opal/mca/event/libevent2022/libevent;/opt/hpcx/ompi/include/openmpi/opal/mca/event/libevent2022/libevent/include
    #25 81.11 -- Using MPI_CXX_LIBRARIES: /opt/hpcx/ompi/lib/libmpi.so
    #25 81.13 -- CMAKE_SYSTEM_PROCESSOR: x86_64
    #25 82.62 -- USE_CXX11_ABI: True
    #25 474.9 [ 11%] Creating directories for 'googletest-populate'
    #25 474.9 [ 22%] Performing download step (git clone) for 'googletest-populate'
    #25 474.9 Cloning into 'googletest-src'...
    #25 474.9 fatal: unable to access 'https://github.com/google/googletest.git/': Failed to connect to github.com port 443 after 130010 ms: Connection timed out
    #25 474.9 Cloning into 'googletest-src'...
    #25 474.9 fatal: unable to access 'https://github.com/google/googletest.git/': Failed to connect to github.com port 443 after 131050 ms: Connection timed out
    #25 474.9 Cloning into 'googletest-src'...
    #25 474.9 fatal: unable to access 'https://github.com/google/googletest.git/': Failed to connect to github.com port 443 after 131050 ms: Connection timed out
    #25 474.9 -- Had to git clone more than once: 3 times.
    #25 474.9 CMake Error at googletest-subbuild/googletest-populate-prefix/tmp/googletest-populate-gitclone.cmake:39 (message):
    #25 474.9 Failed to clone repository: 'https://github.com/google/googletest.git'
    #25 474.9
    #25 474.9
    #25 474.9 gmake[2]: *** [CMakeFiles/googletest-populate.dir/build.make:102: googletest-populate-prefix/src/googletest-populate-stamp/googletest-populate-download] Error 1
    #25 474.9 gmake[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/googletest-populate.dir/all] Error 2
    #25 474.9 gmake: *** [Makefile:91: all] Error 2
    #25 474.9
    #25 474.9 CMake Error at /usr/local/cmake-3.24.4-linux-x86_64/share/cmake-3.24/Modules/FetchContent.cmake:1604 (message):
    #25 474.9 Build step for googletest failed: 2
    #25 474.9 Call Stack (most recent call first):
    #25 474.9 /usr/local/cmake-3.24.4-linux-x86_64/share/cmake-3.24/Modules/FetchContent.cmake:1744:EVAL:2 (__FetchContent_directPopulate)
    #25 474.9 /usr/local/cmake-3.24.4-linux-x86_64/share/cmake-3.24/Modules/FetchContent.cmake:1744 (cmake_language)
    #25 474.9 /usr/local/cmake-3.24.4-linux-x86_64/share/cmake-3.24/Modules/FetchContent.cmake:1958 (FetchContent_Populate)
    #25 474.9 tests/CMakeLists.txt:21 (FetchContent_MakeAvailable)
    #25 474.9
    #25 474.9
    #25 474.9 -- Configuring incomplete, errors occurred!
    #25 474.9 See also "/src/tensorrt_llm/cpp/build/CMakeFiles/CMakeOutput.log".
    #25 474.9 See also "/src/tensorrt_llm/cpp/build/CMakeFiles/CMakeError.log".
    #25 475.0 Traceback (most recent call last):
    #25 475.0 File "/src/tensorrt_llm/scripts/build_wheel.py", line 248, in
    #25 475.0 main(**vars(args))
    #25 475.0 File "/src/tensorrt_llm/scripts/build_wheel.py", line 149, in main
    #25 475.0 build_run(
    #25 475.0 File "/usr/lib/python3.10/subprocess.py", line 526, in run
    #25 475.0 raise CalledProcessError(retcode, process.args,
    #25 475.0 subprocess.CalledProcessError: Command 'cmake -DCMAKE_BUILD_TYPE="Release" -DBUILD_PYT="ON" -DTRT_LIB_DIR=/usr/local/tensorrt/targets/x86_64-linux-gnu/lib -DTRT_INCLUDE_DIR=/usr/local/tensorrt/include -S "/src/tensorrt_llm/cpp"' returned non-zero exit status 1.

    executor failed running [/bin/bash -c python3 scripts/build_wheel.py ${BUILD_WHEEL_ARGS}]: exit code: 1
    Makefile:46: recipe for target 'release_build' failed
    make: *** [release_build] Error 1
    make: Leaving directory '/data/yankai/TensorRT-LLM/docker'

  5. changer-zxy
    closed this as not plannedon Nov 20, 2023
  6. changer-zxy
    reopened this on Nov 20, 2023

  7. Shixiaowei02 commented on Nov 20, 2023

    @Shixiaowei02
    Shixiaowei02
    on Nov 20, 2023
    Collaborator

    Please confirm that you can access github.com and other related websites normally. Thanks!

  8. Shixiaowei02
    added
    build
    on Nov 20, 2023
  9. Shixiaowei02
    self-assigned this
    on Nov 20, 2023
  10. changer-zxy
    closed this as completedon Nov 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

triagedIssue has been triaged by maintainers

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions