@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / b9cecc2635
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 08:00 UTC

[Docs] Update installation page (#1005)

Woosuk Kwon committed Sep 10, 2023 at 14:23 UTC b9cecc26359794af863b3484a3464108b7d5ee5f
1 file changed +11 -19
docs/source/getting_started/installation.rst
+11 -19
@@ -3,31 +3,15 @@
3 Installation
4 ============
5
6 -vLLM is a Python library that also contains some C++ and CUDA code.
7 -This additional code requires compilation on the user's machine.
6 +vLLM is a Python library that also contains pre-compiled C++ and CUDA (11.8) binaries.
7
8 Requirements
9 ------------
10
11 * OS: Linux
13 -* Python: 3.8 or higher
14 -* CUDA: 11.0 -- 11.8
12 +* Python: 3.8 -- 3.11
13 * GPU: compute capability 7.0 or higher (e.g., V100, T4, RTX20xx, A100, L4, etc.)
14
17 -.. note::
18 - As of now, vLLM does not support CUDA 12.
19 - If you are using Hopper or Lovelace GPUs, please use CUDA 11.8 instead of CUDA 12.
20 -
21 -.. tip::
22 - If you have trouble installing vLLM, we recommend using the NVIDIA PyTorch Docker image.
23 -
24 - .. code-block:: console
25 -
26 - $ # Pull the Docker image with CUDA 11.8.
27 - $ docker run --gpus all -it --rm --shm-size=8g nvcr.io/nvidia/pytorch:22.12-py3
28 -
29 - Inside the Docker container, please execute :code:`pip uninstall torch` before installing vLLM.
30 -
15 Install with pip
16 ----------------
17
@@ -40,7 +24,7 @@ You can install vLLM using pip:
24 $ conda activate myenv
25
26 $ # Install vLLM.
43 - $ pip install vllm # This may take 5-10 minutes.
27 + $ pip install vllm
28
29
30 .. _build_from_source:
@@ -55,3 +39,11 @@ You can also build and install vLLM from source:
39 $ git clone https://github.com/vllm-project/vllm.git
40 $ cd vllm
41 $ pip install -e . # This may take 5-10 minutes.
42 +
43 +.. tip::
44 + If you have trouble building vLLM, we recommend using the NVIDIA PyTorch Docker image.
45 +
46 + .. code-block:: console
47 +
48 + $ # Pull the Docker image with CUDA 11.8.
49 + $ docker run --gpus all -it --rm --shm-size=8g nvcr.io/nvidia/pytorch:22.12-py3