@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 9c82a1bec3
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 03:00 UTC

[Doc] Update installation doc (#3746)

[Doc] Update installation doc for build from source and explain the dependency on torch/cuda version (#3746) Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>

youkaichao committed Mar 30, 2024 at 16:34 UTC 9c82a1bec3a177ff2d611c092c19d25cabd90bb0
1 file changed +21 -12
docs/source/getting_started/installation.rst
+21 -12
@@ -19,7 +19,7 @@ You can install vLLM using pip:
19
20 .. code-block:: console
21
22 - $ # (Optional) Create a new conda environment.
22 + $ # (Recommended) Create a new conda environment.
23 $ conda create -n myenv python=3.9 -y
24 $ conda activate myenv
25
@@ -28,24 +28,19 @@ You can install vLLM using pip:
28
29 .. note::
30
31 - As of now, vLLM's binaries are compiled on CUDA 12.1 by default.
32 - However, you can install vLLM with CUDA 11.8 by running:
31 + As of now, vLLM's binaries are compiled with CUDA 12.1 and public PyTorch release versions by default.
32 + We also provide vLLM binaries compiled with CUDA 11.8 and public PyTorch release versions:
33
34 .. code-block:: console
35
36 $ # Install vLLM with CUDA 11.8.
37 - $ export VLLM_VERSION=0.2.4
37 + $ export VLLM_VERSION=0.4.0
38 $ export PYTHON_VERSION=39
39 - $ pip install https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cu118-cp${PYTHON_VERSION}-cp${PYTHON_VERSION}-manylinux1_x86_64.whl
39 + $ pip install https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cu118-cp${PYTHON_VERSION}-cp${PYTHON_VERSION}-manylinux1_x86_64.whl --extra-index-url https://download.pytorch.org/whl/cu118
40
41 - $ # Re-install PyTorch with CUDA 11.8.
42 - $ pip uninstall torch -y
43 - $ pip install torch --upgrade --index-url https://download.pytorch.org/whl/cu118
44 -
45 - $ # Re-install xFormers with CUDA 11.8.
46 - $ pip uninstall xformers -y
47 - $ pip install --upgrade xformers --index-url https://download.pytorch.org/whl/cu118
41 + In order to be performant, vLLM has to compile many cuda kernels. The compilation unfortunately introduces binary incompatibility with other CUDA versions and PyTorch versions, even for the same PyTorch version with different building configurations.
42
43 + Therefore, it is recommended to install vLLM with a **fresh new** conda environment. If either you have a different CUDA version or you want to use an existing PyTorch installation, you need to build vLLM from source. See below for instructions.
44
45 .. _build_from_source:
46
@@ -77,6 +72,20 @@ You can also build and install vLLM from source:
72 $ # Use `--ipc=host` to make sure the shared memory is large enough.
73 $ docker run --gpus all -it --rm --ipc=host nvcr.io/nvidia/pytorch:23.10-py3
74
75 + If you don't want to use docker, it is recommended to have a full installation of CUDA Toolkit. You can download and install it from `the official website <https://developer.nvidia.com/cuda-toolkit-archive>`_. After installation, set the environment variable `CUDA_HOME` to the installation path of CUDA Toolkit, and make sure that the `nvcc` compiler is in your `PATH`, e.g.:
76 +
77 + .. code-block:: console
78 +
79 + $ export CUDA_HOME=/usr/local/cuda
80 + $ export PATH="${CUDA_HOME}/bin:$PATH"
81 +
82 + Here is a sanity check to verify that the CUDA Toolkit is correctly installed:
83 +
84 + .. code-block:: console
85 +
86 + $ nvcc --version # verify that nvcc is in your PATH
87 + $ ${CUDA_HOME}/bin/nvcc --version # verify that nvcc is in your CUDA_HOME
88 +
89 .. note::
90 If you are developing the C++ backend of vLLM, consider building vLLM with
91