@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 56b7f0efa4
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 09:15 UTC

Add a doc for installation (#128)

Woosuk Kwon committed May 27, 2023 at 01:13 UTC 56b7f0efa4d83865afc2da38b40b2f337d778dda
1 file changed +41 -1
docs/source/getting_started/installation.rst
+41 -1
@@ -1,10 +1,50 @@
1 Installation
2 ============
3
4 +CacheFlow is a Python library that includes some C++ and CUDA code.
5 +CacheFlow can run on systems that meet the following requirements:
6 +
7 +* OS: Linux
8 +* Python: 3.8 or higher
9 +* CUDA: 11.0 -- 11.8
10 +* GPU: compute capability 7.0 or higher (e.g., V100, T4, RTX20xx, A100, etc.)
11 +
12 +.. note::
13 + As of now, CacheFlow does not support CUDA 12.
14 + If you are using Hopper or Lovelace GPUs, please use CUDA 11.8.
15 +
16 +.. tip::
17 + If you have trouble installing CacheFlow, we recommend using the NVIDIA PyTorch Docker image.
18 +
19 + .. code-block:: console
20 +
21 + $ docker run --gpus all -it --rm --shm-size=8g nvcr.io/nvidia/pytorch:22.12-py3
22 +
23 +Install with pip
24 +----------------
25 +
26 +You can install CacheFlow using pip:
27 +
28 +.. code-block:: console
29 +
30 + $ # (Optional) Create a new conda environment.
31 + $ conda create -n myenv python=3.8 -y
32 + $ conda activate myenv
33 +
34 + $ # Install CacheFlow.
35 + $ pip install cacheflow
36 +
37 +
38 +.. _build_from_source:
39 +
40 Build from source
41 -----------------
42
43 +You can also build and install CacheFlow from source.
44 +
45 .. code-block:: console
46
47 + $ git clone https://github.com/WoosukKwon/cacheflow.git
48 + $ cd cacheflow
49 $ pip install -r requirements.txt
10 - $ pip install -e . # This may take several minutes.
50 + $ pip install -e . # This may take 5-10 minutes.