A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 08:00 UTC
[Doc] Documentation for distributed inference (#261)
Zhuohan Li committed
Jun 26, 2023 at 11:34 UTC
2cf1a333b63a303fd4b65dd499f2e9b606e6525a
4 files changed
+54
-3
.gitignore
+3
@@ -170,3 +170,6 @@ cython_debug/
170
171
# Python pickle files
172
*.pkl
173
+
174
+# Sphinx documentation
175
+_build/
README.md
+1
-1
@@ -28,7 +28,7 @@ vLLM is fast with:
28
29
- State-of-the-art serving throughput
30
- Efficient management of attention key and value memory with **PagedAttention**
31
-- Dynamic batching of incoming requests
31
+- Continuous batching of incoming requests
32
- Optimized CUDA kernels
33
34
vLLM is flexible and easy to use with:
docs/source/index.rst
+12
-2
@@ -29,7 +29,7 @@ vLLM is fast with:
29
30
* State-of-the-art serving throughput
31
* Efficient management of attention key and value memory with **PagedAttention**
32
-* Dynamic batching of incoming requests
32
+* Continuous batching of incoming requests
33
* Optimized CUDA kernels
34
35
vLLM is flexible and easy to use with:
@@ -40,7 +40,11 @@ vLLM is flexible and easy to use with:
40
* Streaming outputs
41
* OpenAI-compatible API server
42
43
-For more information, please refer to our `blog post <https://vllm.ai>`_.
43
+For more information, check out the following:
44
+
45
+* `vLLM announcing blog post <https://vllm.ai>`_ (intro to PagedAttention)
46
+* `How continuous batching enables 23x throughput in LLM inference while reducing p50 latency <https://www.anyscale.com/blog/continuous-batching-llm-inference>`_ by Cade Daniel et al.
47
+
48
49
50
Documentation
@@ -53,6 +57,12 @@ Documentation
57
getting_started/installation
58
getting_started/quickstart
59
60
+.. toctree::
61
+ :maxdepth: 1
62
+ :caption: Serving
63
+
64
+ serving/distributed_serving
65
+
66
.. toctree::
67
:maxdepth: 1
68
:caption: Models
docs/source/serving/distributed_serving.rst
new
+38
@@ -0,0 +1,38 @@
1
+.. _distributed_serving:
2
+
3
+Distributed Inference and Serving
4
+=================================
5
+
6
+vLLM supports distributed tensor-parallel inference and serving. Currently, we support `Megatron-LM's tensor parallel algorithm <https://arxiv.org/pdf/1909.08053.pdf>`_. We manage the distributed runtime with `Ray <https://github.com/ray-project/ray>`_. To run distributed inference, install Ray with:
7
+
8
+.. code-block:: console
9
+
10
+ $ pip install ray
11
+
12
+To run multi-GPU inference with the :code:`LLM` class, set the :code:`tensor_parallel_size` argument to the number of GPUs you want to use. For example, to run inference on 4 GPUs:
13
+
14
+.. code-block:: python
15
+
16
+ from vllm import LLM
17
+ llm = LLM("facebook/opt-13b", tensor_parallel_size=4)
18
+ output = llm.generate("San Franciso is a")
19
+
20
+To run multi-GPU serving, pass in the :code:`--tensor-parallel-size` argument when starting the server. For example, to run API server on 4 GPUs:
21
+
22
+.. code-block:: console
23
+
24
+ $ python -m vllm.entrypoints.api_server \
25
+ $ --model facebook/opt-13b \
26
+ $ --tensor-parallel-size 4
27
+
28
+To scale vLLM beyond a single machine, start a `Ray runtime <https://docs.ray.io/en/latest/ray-core/starting-ray.html>`_ via CLI before running vLLM:
29
+
30
+.. code-block:: console
31
+
32
+ $ # On head node
33
+ $ ray start --head
34
+
35
+ $ # On worker nodes
36
+ $ ray start --address=<ray-head-address>
37
+
38
+After that, you can run inference and serving on multiple machines by launching the vLLM process on the head node by setting :code:`tensor_parallel_size` to the number of GPUs to be the total number of GPUs across all machines.
\ No newline at end of file