A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 08:00 UTC
[Docs] Update installation page (#1005)
Woosuk Kwon committed
Sep 10, 2023 at 14:23 UTC
b9cecc26359794af863b3484a3464108b7d5ee5f
1 file changed
+11
-19
docs/source/getting_started/installation.rst
+11
-19
@@ -3,31 +3,15 @@
3
Installation
4
============
5
6
-vLLM is a Python library that also contains some C++ and CUDA code.
7
-This additional code requires compilation on the user's machine.
6
+vLLM is a Python library that also contains pre-compiled C++ and CUDA (11.8) binaries.
7
8
Requirements
9
------------
10
11
* OS: Linux
13
-* Python: 3.8 or higher
14
-* CUDA: 11.0 -- 11.8
12
+* Python: 3.8 -- 3.11
13
* GPU: compute capability 7.0 or higher (e.g., V100, T4, RTX20xx, A100, L4, etc.)
14
17
-.. note::
18
- As of now, vLLM does not support CUDA 12.
19
- If you are using Hopper or Lovelace GPUs, please use CUDA 11.8 instead of CUDA 12.
20
-
21
-.. tip::
22
- If you have trouble installing vLLM, we recommend using the NVIDIA PyTorch Docker image.
23
-
24
- .. code-block:: console
25
-
26
- $ # Pull the Docker image with CUDA 11.8.
27
- $ docker run --gpus all -it --rm --shm-size=8g nvcr.io/nvidia/pytorch:22.12-py3
28
-
29
- Inside the Docker container, please execute :code:`pip uninstall torch` before installing vLLM.
30
-
15
Install with pip
16
----------------
17
@@ -40,7 +24,7 @@ You can install vLLM using pip:
24
$ conda activate myenv
25
26
$ # Install vLLM.
43
- $ pip install vllm # This may take 5-10 minutes.
27
+ $ pip install vllm
28
29
30
.. _build_from_source:
@@ -55,3 +39,11 @@ You can also build and install vLLM from source:
39
$ git clone https://github.com/vllm-project/vllm.git
40
$ cd vllm
41
$ pip install -e . # This may take 5-10 minutes.
42
+
43
+.. tip::
44
+ If you have trouble building vLLM, we recommend using the NVIDIA PyTorch Docker image.
45
+
46
+ .. code-block:: console
47
+
48
+ $ # Pull the Docker image with CUDA 11.8.
49
+ $ docker run --gpus all -it --rm --shm-size=8g nvcr.io/nvidia/pytorch:22.12-py3