A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 08:00 UTC
Add logo and polish readme (#156)
Zhuohan Li committed
Jun 19, 2023 at 16:31 UTC
a255885f83b8d302fa684d4c972b2c69c44d47eb
14 files changed
+258
-42
.gitignore
+167
-10
@@ -1,15 +1,172 @@
1
-**/*.pyc
2
-**/__pycache__/
3
-*.egg-info/
4
-*.eggs/
1
+# Byte-compiled / optimized / DLL files
2
+__pycache__/
3
+*.py[cod]
4
+*$py.class
5
+
6
+# C extensions
7
*.so
6
-*.log
7
-*.csv
8
+
9
+# Distribution / packaging
10
+.Python
11
build/
9
-docs/build/
12
+develop-eggs/
13
dist/
14
+downloads/
15
+eggs/
16
+.eggs/
17
+lib/
18
+lib64/
19
+parts/
20
+sdist/
21
+var/
22
+wheels/
23
+share/python-wheels/
24
+*.egg-info/
25
+.installed.cfg
26
+*.egg
27
+MANIFEST
28
12
-*.pkl
13
-*.png
14
-**/log.txt
29
+# PyInstaller
30
+# Usually these files are written by a python script from a template
31
+# before PyInstaller builds the exe, so as to inject date/other infos into it.
32
+*.manifest
33
+*.spec
34
+
35
+# Installer logs
36
+pip-log.txt
37
+pip-delete-this-directory.txt
38
+
39
+# Unit test / coverage reports
40
+htmlcov/
41
+.tox/
42
+.nox/
43
+.coverage
44
+.coverage.*
45
+.cache
46
+nosetests.xml
47
+coverage.xml
48
+*.cover
49
+*.py,cover
50
+.hypothesis/
51
+.pytest_cache/
52
+cover/
53
+
54
+# Translations
55
+*.mo
56
+*.pot
57
+
58
+# Django stuff:
59
+*.log
60
+local_settings.py
61
+db.sqlite3
62
+db.sqlite3-journal
63
+
64
+# Flask stuff:
65
+instance/
66
+.webassets-cache
67
+
68
+# Scrapy stuff:
69
+.scrapy
70
+
71
+# Sphinx documentation
72
+docs/_build/
73
+
74
+# PyBuilder
75
+.pybuilder/
76
+target/
77
+
78
+# Jupyter Notebook
79
+.ipynb_checkpoints
80
+
81
+# IPython
82
+profile_default/
83
+ipython_config.py
84
+
85
+# pyenv
86
+# For a library or package, you might want to ignore these files since the code is
87
+# intended to run in multiple environments; otherwise, check them in:
88
+# .python-version
89
+
90
+# pipenv
91
+# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
92
+# However, in case of collaboration, if having platform-specific dependencies or dependencies
93
+# having no cross-platform support, pipenv may install dependencies that don't work, or not
94
+# install all needed dependencies.
95
+#Pipfile.lock
96
+
97
+# poetry
98
+# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
99
+# This is especially recommended for binary packages to ensure reproducibility, and is more
100
+# commonly ignored for libraries.
101
+# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
102
+#poetry.lock
103
+
104
+# pdm
105
+# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
106
+#pdm.lock
107
+# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
108
+# in version control.
109
+# https://pdm.fming.dev/#use-with-ide
110
+.pdm.toml
111
+
112
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
113
+__pypackages__/
114
+
115
+# Celery stuff
116
+celerybeat-schedule
117
+celerybeat.pid
118
+
119
+# SageMath parsed files
120
+*.sage.py
121
+
122
+# Environments
123
+.env
124
+.venv
125
+env/
126
+venv/
127
+ENV/
128
+env.bak/
129
+venv.bak/
130
+
131
+# Spyder project settings
132
+.spyderproject
133
+.spyproject
134
+
135
+# Rope project settings
136
+.ropeproject
137
+
138
+# mkdocs documentation
139
+/site
140
+
141
+# mypy
142
+.mypy_cache/
143
+.dmypy.json
144
+dmypy.json
145
+
146
+# Pyre type checker
147
+.pyre/
148
+
149
+# pytype static type analyzer
150
+.pytype/
151
+
152
+# Cython debug symbols
153
+cython_debug/
154
+
155
+# PyCharm
156
+# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
157
+# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
158
+# and can be added to the global gitignore or merged into this file. For a more nuclear
159
+# option (not recommended) you can uncomment the following to ignore the entire idea folder.
160
+.idea/
161
+
162
+# VSCode
163
.vscode/
164
+
165
+# DS Store
166
+.DS_Store
167
+
168
+# Results
169
+*.csv
170
+
171
+# Python pickle files
172
+*.pkl
README.md
+54
-20
@@ -1,50 +1,84 @@
1
-# vLLM: Easy, Fast, and Cheap LLM Serving for Everyone
1
+<p align="center">
2
+ <picture>
3
+ <source media="(prefers-color-scheme: dark)" srcset="./docs/source/assets/logos/vllm-logo-text-dark.png">
4
+ <img alt="vLLM" src="./docs/source/assets/logos/vllm-logo-text-light.png" width=55%>
5
+ </picture>
6
+</p>
7
3
-| [**Documentation**](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/) | [**Blog**]() |
8
+<h3 align="center">
9
+Easy, fast, and cheap LLM serving for everyone
10
+</h3>
11
5
-vLLM is a fast and easy-to-use library for LLM inference and serving.
12
+<p align="center">
13
+| <a href="https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/"><b>Documentation</b></a> | <a href=""><b>Blog</b></a> |
14
7
-## Latest News 🔥
15
+</p>
16
9
-- [2023/06] We officially released vLLM! vLLM has powered [LMSYS Vicuna and Chatbot Arena](https://chat.lmsys.org) since mid April. Check out our [blog post]().
17
+---
18
11
-## Getting Started
19
+*Latest News* 🔥
20
13
-Visit our [documentation](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/) to get started.
14
-- [Installation](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/getting_started/installation.html): `pip install vllm`
15
-- [Quickstart](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/getting_started/quickstart.html)
16
-- [Supported Models](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/models/supported_models.html)
21
+- [2023/06] We officially released vLLM! vLLM has powered [LMSYS Vicuna and Chatbot Arena](https://chat.lmsys.org) since mid April. Check out our [blog post]().
22
+
23
+---
24
18
-## Key Features
25
+vLLM is a fast and easy to use library for LLM inference and serving.
26
20
-vLLM comes with many powerful features that include:
27
+vLLM is fast with:
28
22
-- State-of-the-art performance in serving throughput
29
+- State-of-the-art serving throughput
30
- Efficient management of attention key and value memory with **PagedAttention**
24
-- Seamless integration with popular HuggingFace models
31
- Dynamic batching of incoming requests
32
- Optimized CUDA kernels
27
-- High-throughput serving with various decoding algorithms, including *parallel sampling* and *beam search*
33
+
34
+vLLM is flexible and easy to use with:
35
+
36
+- Seamless integration with popular HuggingFace models
37
+- High-throughput serving with various decoding algorithms, including *parallel sampling*, *beam search*, and more
38
- Tensor parallelism support for distributed inference
39
- Streaming outputs
40
- OpenAI-compatible API server
41
42
+Install vLLM with pip or [from source](https://llm-serving-cacheflow.readthedocs-hosted.com/en/latest/getting_started/installation.html#build-from-source):
43
+
44
+```bash
45
+pip install vllm
46
+```
47
+
48
+## Getting Started
49
+
50
+Visit our [documentation](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/) to get started.
51
+- [Installation](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/getting_started/installation.html)
52
+- [Quickstart](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/getting_started/quickstart.html)
53
+- [Supported Models](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/models/supported_models.html)
54
+
55
## Performance
56
57
vLLM outperforms HuggingFace Transformers (HF) by up to 24x and Text Generation Inference (TGI) by up to 3.5x, in terms of throughput.
58
For details, check out our [blog post]().
59
60
<p align="center">
38
- <img src="./assets/figures/perf_a10g_n1.png" width="45%">
39
- <img src="./assets/figures/perf_a100_n1.png" width="45%">
61
+ <picture>
62
+ <source media="(prefers-color-scheme: dark)" srcset="./docs/source/assets/figures/perf_a10g_n1_dark.png">
63
+ <img src="./docs/source/assets/figures/perf_a10g_n1_light.png" width="45%">
64
+ </picture>
65
+ <picture>
66
+ <source media="(prefers-color-scheme: dark)" srcset="./docs/source/assets/figures/perf_a100_n1_dark.png">
67
+ <img src="./docs/source/assets/figures/perf_a100_n1_light.png" width="45%">
68
+ </picture>
69
<br>
70
<em> Serving throughput when each request asks for 1 output completion. </em>
71
</p>
72
73
<p align="center">
45
- <img src="./assets/figures/perf_a10g_n3.png" width="45%">
46
- <img src="./assets/figures/perf_a100_n3.png" width="45%">
47
- <br>
74
+ <picture>
75
+ <source media="(prefers-color-scheme: dark)" srcset="./docs/source/assets/figures/perf_a10g_n3_dark.png">
76
+ <img src="./docs/source/assets/figures/perf_a10g_n3_light.png" width="45%">
77
+ </picture>
78
+ <picture>
79
+ <source media="(prefers-color-scheme: dark)" srcset="./docs/source/assets/figures/perf_a100_n3_dark.png">
80
+ <img src="./docs/source/assets/figures/perf_a100_n3_light.png" width="45%">
81
+ </picture> <br>
82
<em> Serving throughput when each request asks for 3 output completions. </em>
83
</p>
84
docs/source/assets/figures/perf_a100_n1_dark.png
renamed
docs/source/assets/figures/perf_a100_n1_light.png
renamed
docs/source/assets/figures/perf_a100_n3_dark.png
renamed
docs/source/assets/figures/perf_a100_n3_light.png
renamed
docs/source/assets/figures/perf_a10g_n1_dark.png
renamed
docs/source/assets/figures/perf_a10g_n1_light.png
renamed
docs/source/assets/figures/perf_a10g_n3_dark.png
renamed
docs/source/assets/figures/perf_a10g_n3_light.png
renamed
docs/source/assets/logos/vllm-logo-only-light.png
Binary files /dev/null and b/docs/source/assets/logos/vllm-logo-only-light.png differ
docs/source/assets/logos/vllm-logo-text-dark.png
Binary files /dev/null and b/docs/source/assets/logos/vllm-logo-text-dark.png differ
docs/source/assets/logos/vllm-logo-text-light.png
Binary files /dev/null and b/docs/source/assets/logos/vllm-logo-text-light.png differ
docs/source/index.rst
+37
-12
@@ -1,18 +1,43 @@
1
Welcome to vLLM!
2
================
3
4
-**vLLM** is a fast and easy-to-use library for LLM inference and serving.
5
-Its core features include:
6
-
7
-- State-of-the-art performance in serving throughput
8
-- Efficient management of attention key and value memory with **PagedAttention**
9
-- Seamless integration with popular HuggingFace models
10
-- Dynamic batching of incoming requests
11
-- Optimized CUDA kernels
12
-- High-throughput serving with various decoding algorithms, including *parallel sampling* and *beam search*
13
-- Tensor parallelism support for distributed inference
14
-- Streaming outputs
15
-- OpenAI-compatible API server
4
+.. figure:: ./assets/logos/vllm-logo-text-light.png
5
+ :width: 60%
6
+ :align: center
7
+ :alt: vLLM
8
+ :class: no-scaled-link
9
+
10
+.. raw:: html
11
+
12
+ <p style="text-align:center">
13
+ <strong>Easy, fast, and cheap LLM serving for everyone
14
+ </strong>
15
+ </p>
16
+
17
+ <p style="text-align:center">
18
+ <a class="github-button" href="https://github.com/WoosukKwon/vllm" data-show-count="true" data-size="large" aria-label="Star skypilot-org/skypilot on GitHub">Star</a>
19
+ <a class="github-button" href="https://github.com/WoosukKwon/vllm/subscription" data-icon="octicon-eye" data-size="large" aria-label="Watch skypilot-org/skypilot on GitHub">Watch</a>
20
+ <a class="github-button" href="https://github.com/WoosukKwon/vllm/fork" data-icon="octicon-repo-forked" data-size="large" aria-label="Fork skypilot-org/skypilot on GitHub">Fork</a>
21
+ </p>
22
+
23
+
24
+
25
+vLLM is a fast and easy to use library for LLM inference and serving.
26
+
27
+vLLM is fast with:
28
+
29
+* State-of-the-art serving throughput
30
+* Efficient management of attention key and value memory with **PagedAttention**
31
+* Dynamic batching of incoming requests
32
+* Optimized CUDA kernels
33
+
34
+vLLM is flexible and easy to use with:
35
+
36
+* Seamless integration with popular HuggingFace models
37
+* High-throughput serving with various decoding algorithms, including *parallel sampling*, *beam search*, and more
38
+* Tensor parallelism support for distributed inference
39
+* Streaming outputs
40
+* OpenAI-compatible API server
41
42
For more information, please refer to our `blog post <>`_.
43