@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 2cf1a333b6
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 08:00 UTC

[Doc] Documentation for distributed inference (#261)

Zhuohan Li committed Jun 26, 2023 at 11:34 UTC 2cf1a333b63a303fd4b65dd499f2e9b606e6525a
4 files changed +54 -3
.gitignore
+3
@@ -170,3 +170,6 @@ cython_debug/
170
171 # Python pickle files
172 *.pkl
173 +
174 +# Sphinx documentation
175 +_build/
README.md
+1 -1
@@ -28,7 +28,7 @@ vLLM is fast with:
28
29 - State-of-the-art serving throughput
30 - Efficient management of attention key and value memory with **PagedAttention**
31 -- Dynamic batching of incoming requests
31 +- Continuous batching of incoming requests
32 - Optimized CUDA kernels
33
34 vLLM is flexible and easy to use with:
docs/source/index.rst
+12 -2
@@ -29,7 +29,7 @@ vLLM is fast with:
29
30 * State-of-the-art serving throughput
31 * Efficient management of attention key and value memory with **PagedAttention**
32 -* Dynamic batching of incoming requests
32 +* Continuous batching of incoming requests
33 * Optimized CUDA kernels
34
35 vLLM is flexible and easy to use with:
@@ -40,7 +40,11 @@ vLLM is flexible and easy to use with:
40 * Streaming outputs
41 * OpenAI-compatible API server
42
43 -For more information, please refer to our `blog post <https://vllm.ai>`_.
43 +For more information, check out the following:
44 +
45 +* `vLLM announcing blog post <https://vllm.ai>`_ (intro to PagedAttention)
46 +* `How continuous batching enables 23x throughput in LLM inference while reducing p50 latency <https://www.anyscale.com/blog/continuous-batching-llm-inference>`_ by Cade Daniel et al.
47 +
48
49
50 Documentation
@@ -53,6 +57,12 @@ Documentation
57 getting_started/installation
58 getting_started/quickstart
59
60 +.. toctree::
61 + :maxdepth: 1
62 + :caption: Serving
63 +
64 + serving/distributed_serving
65 +
66 .. toctree::
67 :maxdepth: 1
68 :caption: Models
docs/source/serving/distributed_serving.rst new
+38
@@ -0,0 +1,38 @@
1 +.. _distributed_serving:
2 +
3 +Distributed Inference and Serving
4 +=================================
5 +
6 +vLLM supports distributed tensor-parallel inference and serving. Currently, we support `Megatron-LM's tensor parallel algorithm <https://arxiv.org/pdf/1909.08053.pdf>`_. We manage the distributed runtime with `Ray <https://github.com/ray-project/ray>`_. To run distributed inference, install Ray with:
7 +
8 +.. code-block:: console
9 +
10 + $ pip install ray
11 +
12 +To run multi-GPU inference with the :code:`LLM` class, set the :code:`tensor_parallel_size` argument to the number of GPUs you want to use. For example, to run inference on 4 GPUs:
13 +
14 +.. code-block:: python
15 +
16 + from vllm import LLM
17 + llm = LLM("facebook/opt-13b", tensor_parallel_size=4)
18 + output = llm.generate("San Franciso is a")
19 +
20 +To run multi-GPU serving, pass in the :code:`--tensor-parallel-size` argument when starting the server. For example, to run API server on 4 GPUs:
21 +
22 +.. code-block:: console
23 +
24 + $ python -m vllm.entrypoints.api_server \
25 + $ --model facebook/opt-13b \
26 + $ --tensor-parallel-size 4
27 +
28 +To scale vLLM beyond a single machine, start a `Ray runtime <https://docs.ray.io/en/latest/ray-core/starting-ray.html>`_ via CLI before running vLLM:
29 +
30 +.. code-block:: console
31 +
32 + $ # On head node
33 + $ ray start --head
34 +
35 + $ # On worker nodes
36 + $ ray start --address=<ray-head-address>
37 +
38 +After that, you can run inference and serving on multiple machines by launching the vLLM process on the head node by setting :code:`tensor_parallel_size` to the number of GPUs to be the total number of GPUs across all machines.
\ No newline at end of file