A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 09:15 UTC
Add a doc for installation (#128)
Woosuk Kwon committed
May 27, 2023 at 01:13 UTC
56b7f0efa4d83865afc2da38b40b2f337d778dda
1 file changed
+41
-1
docs/source/getting_started/installation.rst
+41
-1
@@ -1,10 +1,50 @@
1
Installation
2
============
3
4
+CacheFlow is a Python library that includes some C++ and CUDA code.
5
+CacheFlow can run on systems that meet the following requirements:
6
+
7
+* OS: Linux
8
+* Python: 3.8 or higher
9
+* CUDA: 11.0 -- 11.8
10
+* GPU: compute capability 7.0 or higher (e.g., V100, T4, RTX20xx, A100, etc.)
11
+
12
+.. note::
13
+ As of now, CacheFlow does not support CUDA 12.
14
+ If you are using Hopper or Lovelace GPUs, please use CUDA 11.8.
15
+
16
+.. tip::
17
+ If you have trouble installing CacheFlow, we recommend using the NVIDIA PyTorch Docker image.
18
+
19
+ .. code-block:: console
20
+
21
+ $ docker run --gpus all -it --rm --shm-size=8g nvcr.io/nvidia/pytorch:22.12-py3
22
+
23
+Install with pip
24
+----------------
25
+
26
+You can install CacheFlow using pip:
27
+
28
+.. code-block:: console
29
+
30
+ $ # (Optional) Create a new conda environment.
31
+ $ conda create -n myenv python=3.8 -y
32
+ $ conda activate myenv
33
+
34
+ $ # Install CacheFlow.
35
+ $ pip install cacheflow
36
+
37
+
38
+.. _build_from_source:
39
+
40
Build from source
41
-----------------
42
43
+You can also build and install CacheFlow from source.
44
+
45
.. code-block:: console
46
47
+ $ git clone https://github.com/WoosukKwon/cacheflow.git
48
+ $ cd cacheflow
49
$ pip install -r requirements.txt
10
- $ pip install -e . # This may take several minutes.
50
+ $ pip install -e . # This may take 5-10 minutes.