A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 08:00 UTC
Write README and front page of doc (#147)
Woosuk Kwon committed
Jun 18, 2023 at 03:19 UTC
dcda03b4cb2c757682abe0faf14a6379dcde1cb2
9 files changed
+65
-60
README.md
+39
-51
@@ -1,66 +1,54 @@
1
-# vLLM
1
+# vLLM: Easy, Fast, and Cheap LLM Serving for Everyone
2
3
-## Build from source
3
+| [**Documentation**](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/) | [**Blog**]() |
4
5
-```bash
6
-pip install -r requirements.txt
7
-pip install -e . # This may take several minutes.
8
-```
5
+vLLM is a fast and easy-to-use library for LLM inference and serving.
6
10
-## Test simple server
7
+## Latest News 🔥
8
12
-```bash
13
-# Single-GPU inference.
14
-python examples/simple_server.py # --model <your_model>
9
+- [2023/06] We officially released vLLM! vLLM has powered [LMSYS Vicuna and Chatbot Arena](https://chat.lmsys.org) since mid April. Check out our [blog post]().
10
16
-# Multi-GPU inference (e.g., 2 GPUs).
17
-ray start --head
18
-python examples/simple_server.py -tp 2 # --model <your_model>
19
-```
11
+## Getting Started
12
21
-The detailed arguments for `simple_server.py` can be found by:
22
-```bash
23
-python examples/simple_server.py --help
24
-```
13
+Visit our [documentation](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/) to get started.
14
+- [Installation](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/getting_started/installation.html): `pip install vllm`
15
+- [Quickstart](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/getting_started/quickstart.html)
16
+- [Supported Models](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/models/supported_models.html)
17
26
-## FastAPI server
18
+## Key Features
19
28
-To start the server:
29
-```bash
30
-ray start --head
31
-python -m vllm.entrypoints.fastapi_server # --model <your_model>
32
-```
20
+vLLM comes with many powerful features that include:
21
34
-To test the server:
35
-```bash
36
-python test_cli_client.py
37
-```
22
+- State-of-the-art performance in serving throughput
23
+- Efficient management of attention key and value memory with **PagedAttention**
24
+- Seamless integration with popular HuggingFace models
25
+- Dynamic batching of incoming requests
26
+- Optimized CUDA kernels
27
+- High-throughput serving with various decoding algorithms, including *parallel sampling* and *beam search*
28
+- Tensor parallelism support for distributed inference
29
+- Streaming outputs
30
+- OpenAI-compatible API server
31
39
-## Gradio web server
32
+## Performance
33
41
-Install the following additional dependencies:
42
-```bash
43
-pip install gradio
44
-```
34
+vLLM outperforms HuggingFace Transformers (HF) by up to 24x and Text Generation Inference (TGI) by up to 3.5x, in terms of throughput.
35
+For details, check out our [blog post]().
36
46
-Start the server:
47
-```bash
48
-python -m vllm.http_frontend.fastapi_frontend
49
-# At another terminal
50
-python -m vllm.http_frontend.gradio_webserver
51
-```
37
+<p align="center">
38
+ <img src="./assets/figures/perf_a10g_n1.png" width="45%">
39
+ <img src="./assets/figures/perf_a100_n1.png" width="45%">
40
+ <br>
41
+ <em> Serving throughput when each request asks for 1 output completion. </em>
42
+</p>
43
53
-## Load LLaMA weights
44
+<p align="center">
45
+ <img src="./assets/figures/perf_a10g_n3.png" width="45%">
46
+ <img src="./assets/figures/perf_a100_n3.png" width="45%">
47
+ <br>
48
+ <em> Serving throughput when each request asks for 3 output completions. </em>
49
+</p>
50
55
-Since LLaMA weight is not fully public, we cannot directly download the LLaMA weights from huggingface. Therefore, you need to follow the following process to load the LLaMA weights.
51
+## Contributing
52
57
-1. Converting LLaMA weights to huggingface format with [this script](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py).
58
- ```bash
59
- python src/transformers/models/llama/convert_llama_weights_to_hf.py \
60
- --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path/llama-7b
61
- ```
62
-2. For all the commands above, specify the model with `--model /output/path/llama-7b` to load the model. For example:
63
- ```bash
64
- python simple_server.py --model /output/path/llama-7b
65
- python -m vllm.http_frontend.fastapi_frontend --model /output/path/llama-7b
66
- ```
53
+We welcome and value any contributions and collaborations.
54
+Please check out [CONTRIBUTING.md](./CONTRIBUTING.md) for how to get involved.
assets/figures/perf_a100_n1.png
Binary files /dev/null and b/assets/figures/perf_a100_n1.png differ
assets/figures/perf_a100_n3.png
Binary files /dev/null and b/assets/figures/perf_a100_n3.png differ
assets/figures/perf_a10g_n1.png
Binary files /dev/null and b/assets/figures/perf_a10g_n1.png differ
assets/figures/perf_a10g_n3.png
Binary files /dev/null and b/assets/figures/perf_a10g_n3.png differ
docs/source/getting_started/installation.rst
+8
-5
@@ -3,17 +3,20 @@
3
Installation
4
============
5
6
-vLLM is a Python library that includes some C++ and CUDA code.
7
-vLLM can run on systems that meet the following requirements:
6
+vLLM is a Python library that also contains some C++ and CUDA code.
7
+This additional code requires compilation on the user's machine.
8
+
9
+Requirements
10
+------------
11
12
* OS: Linux
13
* Python: 3.8 or higher
14
* CUDA: 11.0 -- 11.8
12
-* GPU: compute capability 7.0 or higher (e.g., V100, T4, RTX20xx, A100, etc.)
15
+* GPU: compute capability 7.0 or higher (e.g., V100, T4, RTX20xx, A100, L4, etc.)
16
17
.. note::
18
As of now, vLLM does not support CUDA 12.
16
- If you are using Hopper or Lovelace GPUs, please use CUDA 11.8.
19
+ If you are using Hopper or Lovelace GPUs, please use CUDA 11.8 instead of CUDA 12.
20
21
.. tip::
22
If you have trouble installing vLLM, we recommend using the NVIDIA PyTorch Docker image.
@@ -45,7 +48,7 @@ You can install vLLM using pip:
48
Build from source
49
-----------------
50
48
-You can also build and install vLLM from source.
51
+You can also build and install vLLM from source:
52
53
.. code-block:: console
54
docs/source/index.rst
+15
-1
@@ -1,7 +1,21 @@
1
Welcome to vLLM!
2
================
3
4
-vLLM is a high-throughput and memory-efficient inference and serving engine for large language models (LLM).
4
+**vLLM** is a fast and easy-to-use library for LLM inference and serving.
5
+Its core features include:
6
+
7
+- State-of-the-art performance in serving throughput
8
+- Efficient management of attention key and value memory with **PagedAttention**
9
+- Seamless integration with popular HuggingFace models
10
+- Dynamic batching of incoming requests
11
+- Optimized CUDA kernels
12
+- High-throughput serving with various decoding algorithms, including *parallel sampling* and *beam search*
13
+- Tensor parallelism support for distributed inference
14
+- Streaming outputs
15
+- OpenAI-compatible API server
16
+
17
+For more information, please refer to our `blog post <>`_.
18
+
19
20
Documentation
21
-------------
docs/source/models/supported_models.rst
+2
-2
@@ -3,7 +3,7 @@
3
Supported Models
4
================
5
6
-vLLM supports a variety of generative Transformer models in `HuggingFace Transformers <https://github.com/huggingface/transformers>`_.
6
+vLLM supports a variety of generative Transformer models in `HuggingFace Transformers <https://huggingface.co/models>`_.
7
The following is the list of model architectures that are currently supported by vLLM.
8
Alongside each architecture, we include some popular models that use it.
9
@@ -18,7 +18,7 @@ Alongside each architecture, we include some popular models that use it.
18
* - :code:`GPTNeoXForCausalLM`
19
- GPT-NeoX, Pythia, OpenAssistant, Dolly V2, StableLM
20
* - :code:`LlamaForCausalLM`
21
- - LLaMA, Vicuna, Alpaca, Koala
21
+ - LLaMA, Vicuna, Alpaca, Koala, Guanaco
22
* - :code:`OPTForCausalLM`
23
- OPT, OPT-IML
24
setup.py
+1
-1
@@ -165,7 +165,7 @@ setuptools.setup(
165
"Topic :: Scientific/Engineering :: Artificial Intelligence",
166
],
167
packages=setuptools.find_packages(
168
- exclude=("benchmarks", "csrc", "docs", "examples", "tests")),
168
+ exclude=("assets", "benchmarks", "csrc", "docs", "examples", "tests")),
169
python_requires=">=3.8",
170
install_requires=get_requirements(),
171
ext_modules=ext_modules,