@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / dcda03b4cb
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 08:00 UTC

Write README and front page of doc (#147)

Woosuk Kwon committed Jun 18, 2023 at 03:19 UTC dcda03b4cb2c757682abe0faf14a6379dcde1cb2
9 files changed +65 -60
README.md
+39 -51
@@ -1,66 +1,54 @@
1 -# vLLM
1 +# vLLM: Easy, Fast, and Cheap LLM Serving for Everyone
2
3 -## Build from source
3 +| [**Documentation**](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/) | [**Blog**]() |
4
5 -```bash
6 -pip install -r requirements.txt
7 -pip install -e . # This may take several minutes.
8 -```
5 +vLLM is a fast and easy-to-use library for LLM inference and serving.
6
10 -## Test simple server
7 +## Latest News 🔥
8
12 -```bash
13 -# Single-GPU inference.
14 -python examples/simple_server.py # --model <your_model>
9 +- [2023/06] We officially released vLLM! vLLM has powered [LMSYS Vicuna and Chatbot Arena](https://chat.lmsys.org) since mid April. Check out our [blog post]().
10
16 -# Multi-GPU inference (e.g., 2 GPUs).
17 -ray start --head
18 -python examples/simple_server.py -tp 2 # --model <your_model>
19 -```
11 +## Getting Started
12
21 -The detailed arguments for `simple_server.py` can be found by:
22 -```bash
23 -python examples/simple_server.py --help
24 -```
13 +Visit our [documentation](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/) to get started.
14 +- [Installation](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/getting_started/installation.html): `pip install vllm`
15 +- [Quickstart](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/getting_started/quickstart.html)
16 +- [Supported Models](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/models/supported_models.html)
17
26 -## FastAPI server
18 +## Key Features
19
28 -To start the server:
29 -```bash
30 -ray start --head
31 -python -m vllm.entrypoints.fastapi_server # --model <your_model>
32 -```
20 +vLLM comes with many powerful features that include:
21
34 -To test the server:
35 -```bash
36 -python test_cli_client.py
37 -```
22 +- State-of-the-art performance in serving throughput
23 +- Efficient management of attention key and value memory with **PagedAttention**
24 +- Seamless integration with popular HuggingFace models
25 +- Dynamic batching of incoming requests
26 +- Optimized CUDA kernels
27 +- High-throughput serving with various decoding algorithms, including *parallel sampling* and *beam search*
28 +- Tensor parallelism support for distributed inference
29 +- Streaming outputs
30 +- OpenAI-compatible API server
31
39 -## Gradio web server
32 +## Performance
33
41 -Install the following additional dependencies:
42 -```bash
43 -pip install gradio
44 -```
34 +vLLM outperforms HuggingFace Transformers (HF) by up to 24x and Text Generation Inference (TGI) by up to 3.5x, in terms of throughput.
35 +For details, check out our [blog post]().
36
46 -Start the server:
47 -```bash
48 -python -m vllm.http_frontend.fastapi_frontend
49 -# At another terminal
50 -python -m vllm.http_frontend.gradio_webserver
51 -```
37 +<p align="center">
38 + <img src="./assets/figures/perf_a10g_n1.png" width="45%">
39 + <img src="./assets/figures/perf_a100_n1.png" width="45%">
40 + <br>
41 + <em> Serving throughput when each request asks for 1 output completion. </em>
42 +</p>
43
53 -## Load LLaMA weights
44 +<p align="center">
45 + <img src="./assets/figures/perf_a10g_n3.png" width="45%">
46 + <img src="./assets/figures/perf_a100_n3.png" width="45%">
47 + <br>
48 + <em> Serving throughput when each request asks for 3 output completions. </em>
49 +</p>
50
55 -Since LLaMA weight is not fully public, we cannot directly download the LLaMA weights from huggingface. Therefore, you need to follow the following process to load the LLaMA weights.
51 +## Contributing
52
57 -1. Converting LLaMA weights to huggingface format with [this script](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py).
58 - ```bash
59 - python src/transformers/models/llama/convert_llama_weights_to_hf.py \
60 - --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path/llama-7b
61 - ```
62 -2. For all the commands above, specify the model with `--model /output/path/llama-7b` to load the model. For example:
63 - ```bash
64 - python simple_server.py --model /output/path/llama-7b
65 - python -m vllm.http_frontend.fastapi_frontend --model /output/path/llama-7b
66 - ```
53 +We welcome and value any contributions and collaborations.
54 +Please check out [CONTRIBUTING.md](./CONTRIBUTING.md) for how to get involved.
assets/figures/perf_a100_n1.png
Binary files /dev/null and b/assets/figures/perf_a100_n1.png differ
assets/figures/perf_a100_n3.png
Binary files /dev/null and b/assets/figures/perf_a100_n3.png differ
assets/figures/perf_a10g_n1.png
Binary files /dev/null and b/assets/figures/perf_a10g_n1.png differ
assets/figures/perf_a10g_n3.png
Binary files /dev/null and b/assets/figures/perf_a10g_n3.png differ
docs/source/getting_started/installation.rst
+8 -5
@@ -3,17 +3,20 @@
3 Installation
4 ============
5
6 -vLLM is a Python library that includes some C++ and CUDA code.
7 -vLLM can run on systems that meet the following requirements:
6 +vLLM is a Python library that also contains some C++ and CUDA code.
7 +This additional code requires compilation on the user's machine.
8 +
9 +Requirements
10 +------------
11
12 * OS: Linux
13 * Python: 3.8 or higher
14 * CUDA: 11.0 -- 11.8
12 -* GPU: compute capability 7.0 or higher (e.g., V100, T4, RTX20xx, A100, etc.)
15 +* GPU: compute capability 7.0 or higher (e.g., V100, T4, RTX20xx, A100, L4, etc.)
16
17 .. note::
18 As of now, vLLM does not support CUDA 12.
16 - If you are using Hopper or Lovelace GPUs, please use CUDA 11.8.
19 + If you are using Hopper or Lovelace GPUs, please use CUDA 11.8 instead of CUDA 12.
20
21 .. tip::
22 If you have trouble installing vLLM, we recommend using the NVIDIA PyTorch Docker image.
@@ -45,7 +48,7 @@ You can install vLLM using pip:
48 Build from source
49 -----------------
50
48 -You can also build and install vLLM from source.
51 +You can also build and install vLLM from source:
52
53 .. code-block:: console
54
docs/source/index.rst
+15 -1
@@ -1,7 +1,21 @@
1 Welcome to vLLM!
2 ================
3
4 -vLLM is a high-throughput and memory-efficient inference and serving engine for large language models (LLM).
4 +**vLLM** is a fast and easy-to-use library for LLM inference and serving.
5 +Its core features include:
6 +
7 +- State-of-the-art performance in serving throughput
8 +- Efficient management of attention key and value memory with **PagedAttention**
9 +- Seamless integration with popular HuggingFace models
10 +- Dynamic batching of incoming requests
11 +- Optimized CUDA kernels
12 +- High-throughput serving with various decoding algorithms, including *parallel sampling* and *beam search*
13 +- Tensor parallelism support for distributed inference
14 +- Streaming outputs
15 +- OpenAI-compatible API server
16 +
17 +For more information, please refer to our `blog post <>`_.
18 +
19
20 Documentation
21 -------------
docs/source/models/supported_models.rst
+2 -2
@@ -3,7 +3,7 @@
3 Supported Models
4 ================
5
6 -vLLM supports a variety of generative Transformer models in `HuggingFace Transformers <https://github.com/huggingface/transformers>`_.
6 +vLLM supports a variety of generative Transformer models in `HuggingFace Transformers <https://huggingface.co/models>`_.
7 The following is the list of model architectures that are currently supported by vLLM.
8 Alongside each architecture, we include some popular models that use it.
9
@@ -18,7 +18,7 @@ Alongside each architecture, we include some popular models that use it.
18 * - :code:`GPTNeoXForCausalLM`
19 - GPT-NeoX, Pythia, OpenAssistant, Dolly V2, StableLM
20 * - :code:`LlamaForCausalLM`
21 - - LLaMA, Vicuna, Alpaca, Koala
21 + - LLaMA, Vicuna, Alpaca, Koala, Guanaco
22 * - :code:`OPTForCausalLM`
23 - OPT, OPT-IML
24
setup.py
+1 -1
@@ -165,7 +165,7 @@ setuptools.setup(
165 "Topic :: Scientific/Engineering :: Artificial Intelligence",
166 ],
167 packages=setuptools.find_packages(
168 - exclude=("benchmarks", "csrc", "docs", "examples", "tests")),
168 + exclude=("assets", "benchmarks", "csrc", "docs", "examples", "tests")),
169 python_requires=">=3.8",
170 install_requires=get_requirements(),
171 ext_modules=ext_modules,