@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 06e9ebebd5
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 06:45 UTC

Add instructions to install vLLM+cu118 (#1717)

Woosuk Kwon committed Nov 18, 2023 at 23:48 UTC 06e9ebebd51c3db779dedec5556251c8ecc3a00a
1 file changed +19 -5
docs/source/getting_started/installation.rst
+19 -5
@@ -3,14 +3,14 @@
3 Installation
4 ============
5
6 -vLLM is a Python library that also contains pre-compiled C++ and CUDA (11.8) binaries.
6 +vLLM is a Python library that also contains pre-compiled C++ and CUDA (12.1) binaries.
7
8 Requirements
9 ------------
10
11 * OS: Linux
12 * Python: 3.8 -- 3.11
13 -* GPU: compute capability 7.0 or higher (e.g., V100, T4, RTX20xx, A100, L4, etc.)
13 +* GPU: compute capability 7.0 or higher (e.g., V100, T4, RTX20xx, A100, L4, H100, etc.)
14
15 Install with pip
16 ----------------
@@ -23,9 +23,24 @@ You can install vLLM using pip:
23 $ conda create -n myenv python=3.8 -y
24 $ conda activate myenv
25
26 - $ # Install vLLM.
26 + $ # Install vLLM with CUDA 12.1.
27 $ pip install vllm
28
29 +.. note::
30 +
31 + As of now, vLLM's binaries are compiled on CUDA 12.1 by default.
32 + However, you can install vLLM with CUDA 11.8 by running:
33 +
34 + .. code-block:: console
35 +
36 + $ # Install vLLM with CUDA 11.8.
37 + $ # Replace `cp310` with your Python version (e.g., `cp38`, `cp39`, `cp311`).
38 + $ pip install https://github.com/vllm-project/vllm/releases/download/v0.2.2/vllm-0.2.2+cu118-cp310-cp310-manylinux1_x86_64.whl
39 +
40 + $ # Re-install PyTorch with CUDA 11.8.
41 + $ pip uninstall torch -y
42 + $ pip install torch --upgrade --index-url https://download.pytorch.org/whl/cu118
43 +
44
45 .. _build_from_source:
46
@@ -45,6 +60,5 @@ You can also build and install vLLM from source:
60
61 .. code-block:: console
62
48 - $ # Pull the Docker image with CUDA 11.8.
63 $ # Use `--ipc=host` to make sure the shared memory is large enough.
50 - $ docker run --gpus all -it --rm --ipc=host nvcr.io/nvidia/pytorch:22.12-py3
64 + $ docker run --gpus all -it --rm --ipc=host nvcr.io/nvidia/pytorch:23.10-py3