@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 5390d6664f
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 07:00 UTC

[Doc] Add the start of an arch overview page (#10368)

Russell Bryant committed Nov 19, 2024 at 04:52 UTC 5390d6664f65d84f37a5fb524e967b01baad9100
10 files changed +334 -78
.github/workflows/png-lint.yml new
+37
@@ -0,0 +1,37 @@
1 +name: Lint PNG exports from excalidraw
2 +on:
3 + push:
4 + branches:
5 + - "main"
6 + paths:
7 + - '*.excalidraw.png'
8 + - '.github/workflows/png-lint.yml'
9 + pull_request:
10 + branches:
11 + - "main"
12 + paths:
13 + - '*.excalidraw.png'
14 + - '.github/workflows/png-lint.yml'
15 +
16 +env:
17 + LC_ALL: en_US.UTF-8
18 +
19 +defaults:
20 + run:
21 + shell: bash
22 +
23 +permissions:
24 + contents: read
25 +
26 +jobs:
27 + actionlint:
28 + runs-on: ubuntu-latest
29 + steps:
30 + - name: "Checkout"
31 + uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
32 + with:
33 + fetch-depth: 0
34 +
35 + - name: "Run png-lint.sh to check excalidraw exported images"
36 + run: |
37 + tools/png-lint.sh
docs/source/assets/design/arch_overview/entrypoints.excalidraw.png
Binary files /dev/null and b/docs/source/assets/design/arch_overview/entrypoints.excalidraw.png differ
docs/source/assets/design/arch_overview/llm_engine.excalidraw.png
Binary files /dev/null and b/docs/source/assets/design/arch_overview/llm_engine.excalidraw.png differ
docs/source/design/arch_overview.rst new
+274
@@ -0,0 +1,274 @@
1 +.. _arch_overview:
2 +
3 +Architecture Overview
4 +======================
5 +
6 +This document provides an overview of the vLLM architecture.
7 +
8 +.. contents:: Table of Contents
9 + :local:
10 + :depth: 2
11 +
12 +Entrypoints
13 +-----------
14 +
15 +vLLM provides a number of entrypoints for interacting with the system. The
16 +following diagram shows the relationship between them.
17 +
18 +.. image:: /assets/design/arch_overview/entrypoints.excalidraw.png
19 + :alt: Entrypoints Diagram
20 +
21 +LLM Class
22 +^^^^^^^^^
23 +
24 +The LLM class provides the primary Python interface for doing offline inference,
25 +which is interacting with a model without using a separate model inference
26 +server.
27 +
28 +Here is a sample of `LLM` class usage:
29 +
30 +.. code-block:: python
31 +
32 + from vllm import LLM, SamplingParams
33 +
34 + # Define a list of input prompts
35 + prompts = [
36 + "Hello, my name is",
37 + "The capital of France is",
38 + "The largest ocean is",
39 + ]
40 +
41 + # Define sampling parameters
42 + sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
43 +
44 + # Initialize the LLM engine with the OPT-125M model
45 + llm = LLM(model="Qwen/Qwen2.5-1.5B-Instruct")
46 +
47 + # Generate outputs for the input prompts
48 + outputs = llm.generate(prompts, sampling_params)
49 +
50 + # Print the generated outputs
51 + for output in outputs:
52 + prompt = output.prompt
53 + generated_text = output.outputs[0].text
54 + print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
55 +
56 +More API details can be found in the :doc:`Offline Inference
57 +</dev/offline_inference/offline_index>` section of the API docs.
58 +
59 +The code for the `LLM` class can be found in `vllm/entrypoints/llm.py
60 +<https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/llm.py>`_.
61 +
62 +OpenAI-compatible API server
63 +^^^^^^^^^^^^^^^^^^^^^^^^^^^^
64 +
65 +The second primary interface to vLLM is via its OpenAI-compatible API server.
66 +This server can be started using the `vllm serve` command.
67 +
68 +.. code-block:: bash
69 +
70 + vllm serve <model>
71 +
72 +The code for the `vllm` CLI can be found in `vllm/scripts.py
73 +<https://github.com/vllm-project/vllm/blob/main/vllm/scripts.py>`_.
74 +
75 +Sometimes you may see the API server entrypoint used directly instead of via the
76 +`vllm` CLI command. For example:
77 +
78 +.. code-block:: bash
79 +
80 + python -m vllm.entrypoints.openai.api_server --model <model>
81 +
82 +That code can be found in `vllm/entrypoints/openai/api_server.py
83 +<https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/openai/api_server.py>`_.
84 +
85 +More details on the API server can be found in the :doc:`OpenAI Compatible
86 +Server </serving/openai_compatible_server>` document.
87 +
88 +LLM Engine
89 +----------
90 +
91 +The `LLMEngine` and `AsyncLLMEngine` classes are central to the functioning of
92 +the vLLM system, handling model inference and asynchronous request processing.
93 +
94 +.. image:: /assets/design/arch_overview/llm_engine.excalidraw.png
95 + :alt: LLMEngine Diagram
96 +
97 +LLMEngine
98 +^^^^^^^^^
99 +
100 +The `LLMEngine` class is the core component of the vLLM engine. It is
101 +responsible for receiving requests from clients and generating outputs from the
102 +model. The `LLMEngine` includes input processing, model execution (possibly
103 +distributed across multiple hosts and/or GPUs), scheduling, and output
104 +processing.
105 +
106 +- **Input Processing**: Handles tokenization of input text using the specified
107 + tokenizer.
108 +
109 +- **Scheduling**: Chooses which requests are processed in each step.
110 +
111 +- **Model Execution**: Manages the execution of the language model, including
112 + distributed execution across multiple GPUs.
113 +
114 +- **Output Processing**: Processes the outputs generated by the model, decoding the
115 + token IDs from a language model into human-readable text.
116 +
117 +The code for `LLMEngine` can be found in `vllm/engine/llm_engine.py`_.
118 +
119 +.. _vllm/engine/llm_engine.py: https://github.com/vllm-project/vllm/tree/main/vllm/engine/llm_engine.py
120 +
121 +AsyncLLMEngine
122 +^^^^^^^^^^^^^^
123 +
124 +The `AsyncLLMEngine` class is an asynchronous wrapper for the `LLMEngine` class.
125 +It uses `asyncio` to create a background loop that continuously processes
126 +incoming requests. The `AsyncLLMEngine` is designed for online serving, where it
127 +can handle multiple concurrent requests and stream outputs to clients.
128 +
129 +The OpenAI-compatible API server uses the `AsyncLLMEngine`. There is also a demo
130 +API server that serves as a simpler example in
131 +`vllm/entrypoints/api_server.py`_.
132 +
133 +.. _vllm/entrypoints/api_server.py: https://github.com/vllm-project/vllm/tree/main/vllm/entrypoints/api_server.py
134 +
135 +The code for `AsyncLLMEngine` can be found in `vllm/engine/async_llm_engine.py`_.
136 +
137 +.. _vllm/engine/async_llm_engine.py: https://github.com/vllm-project/vllm/tree/main/vllm/engine/async_llm_engine.py
138 +
139 +Worker
140 +------
141 +
142 +A worker is a process that runs the model inference. vLLM follows the common
143 +practice of using one process to control one accelerator device, such as GPUs.
144 +For example, if we use tensor parallelism of size 2 and pipeline parallelism of
145 +size 2, we will have 4 workers in total. Workers are identified by their
146 +``rank`` and ``local_rank``. ``rank`` is used for global orchestration, while
147 +``local_rank`` is mainly used for assigning the accelerator device and accessing
148 +local resources such as the file system and shared memory.
149 +
150 +Model Runner
151 +------------
152 +
153 +Every worker has one model runner object, responsible for loading and running
154 +the model. Much of the model execution logic resides here, such as preparing
155 +input tensors and capturing cudagraphs.
156 +
157 +Model
158 +-----
159 +
160 +Every model runner object has one model object, which is the actual
161 +``torch.nn.Module`` instance. See :ref:`huggingface_integration` for how various
162 +configurations affect the class we ultimately get.
163 +
164 +Class Hierarchy
165 +---------------
166 +
167 +The following figure shows the class hierarchy of vLLM:
168 +
169 + .. figure:: /assets/design/hierarchy.png
170 + :alt: query
171 + :width: 100%
172 + :align: center
173 +
174 +There are several important design choices behind this class hierarchy:
175 +
176 +1. **Extensibility**: All classes in the hierarchy accept a configuration object
177 +containing all the necessary information. The `VllmConfig
178 +<https://github.com/vllm-project/vllm/blob/d1c6799b8870e513bf4f2305cbf6cda9fc3d773b/vllm/config.py#L2036>`__
179 +class is the main configuration object that is passed around. The class
180 +hierarchy is quite deep, and every class needs to read the configuration it is
181 +interested in. By encapsulating all configurations in one object, we can easily
182 +pass the configuration object around and access the configuration we need.
183 +Suppose we want to add a new feature (this is often the case given how fast the
184 +field of LLM inference is evolving) that only touches the model runner. We will
185 +have to add a new configuration option in the `VllmConfig` class. Since we pass
186 +the whole config object around, we only need to add the configuration option to
187 +the `VllmConfig` class, and the model runner can access it directly. We don't
188 +need to change the constructor of the engine, worker, or model class to pass the
189 +new configuration option.
190 +
191 +2. **Uniformity**: The model runner needs a unified interface to create and
192 +initialize the model. vLLM supports more than 50 types of popular open-source
193 +models. Each model has its own initialization logic. If the constructor
194 +signature varies with models, the model runner does not know how to call the
195 +constructor accordingly, without complicated and error-prone inspection logic.
196 +By making the constructor of the model class uniform, the model runner can
197 +easily create and initialize the model without knowing the specific model type.
198 +This is also useful for composing models. Vision-language models often consist
199 +of a vision model and a language model. By making the constructor uniform, we
200 +can easily create a vision model and a language model and compose them into a
201 +vision-language model.
202 +
203 +.. note::
204 +
205 + To support this change, all vLLM models' signatures have been updated to:
206 +
207 + .. code-block:: python
208 +
209 + def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
210 +
211 + To avoid accidentally passing incorrect arguments, the constructor is now keyword-only. This ensures that the constructor will raise an error if old configurations are passed. vLLM developers have already made this change for all models within vLLM. For out-of-tree registered models, developers need to update their models, for example by adding shim code to adapt the old constructor signature to the new one:
212 +
213 + .. code-block:: python
214 +
215 + class MyOldModel(nn.Module):
216 + def __init__(
217 + self,
218 + config,
219 + cache_config: Optional[CacheConfig] = None,
220 + quant_config: Optional[QuantizationConfig] = None,
221 + lora_config: Optional[LoRAConfig] = None,
222 + prefix: str = "",
223 + ) -> None:
224 + ...
225 +
226 + from vllm.config import VllmConfig
227 + class MyNewModel(MyOldModel):
228 + def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
229 + config = vllm_config.model_config.hf_config
230 + cache_config = vllm_config.cache_config
231 + quant_config = vllm_config.quant_config
232 + lora_config = vllm_config.lora_config
233 + super().__init__(config, cache_config, quant_config, lora_config, prefix)
234 +
235 + if __version__ >= "0.6.4":
236 + MyModel = MyNewModel
237 + else:
238 + MyModel = MyOldModel
239 +
240 + This way, the model can work with both old and new versions of vLLM.
241 +
242 +3. **Sharding and Quantization at Initialization**: Certain features require
243 +changing the model weights. For example, tensor parallelism needs to shard the
244 +model weights, and quantization needs to quantize the model weights. There are
245 +two possible ways to implement this feature. One way is to change the model
246 +weights after the model is initialized. The other way is to change the model
247 +weights during the model initialization. vLLM chooses the latter. The first
248 +approach is not scalable to large models. Suppose we want to run a 405B model
249 +(with roughly 810GB weights) with 16 H100 80GB GPUs. Ideally, every GPU should
250 +only load 50GB weights. If we change the model weights after the model is
251 +initialized, we need to load the full 810GB weights to every GPU and then shard
252 +the weights, leading to a huge memory overhead. Instead, if we shard the weights
253 +during the model initialization, every layer will only create a shard of the
254 +weights it needs, leading to a much smaller memory overhead. The same idea
255 +applies to quantization. Note that we also add an additional argument ``prefix``
256 +to the model's constructor so that the model can initialize itself differently
257 +based on the prefix. This is useful for non-uniform quantization, where
258 +different parts of the model are quantized differently. The ``prefix`` is
259 +usually an empty string for the top-level model and a string like ``"vision"``
260 +or ``"language"`` for the sub-models. In general, it matches the name of the
261 +module's state dict in the checkpoint file.
262 +
263 +One disadvantage of this design is that it is hard to write unit tests for
264 +individual components in vLLM because every component needs to be initialized by
265 +a complete config object. We solve this problem by providing a default
266 +initialization function that creates a default config object with all fields set
267 +to ``None``. If the component we want to test only cares about a few fields in
268 +the config object, we can create a default config object and set the fields we
269 +care about. This way, we can test the component in isolation. Note that many
270 +tests in vLLM are end-to-end tests that test the whole system, so this is not a
271 +big problem.
272 +
273 +In summary, the complete config object ``VllmConfig`` can be treated as an
274 +engine-level global state that is shared among all vLLM classes.
docs/source/design/class_hierarchy.rst deleted
-74
@@ -1,74 +0,0 @@
1 -.. _class_hierarchy:
2 -
3 -vLLM's Class Hierarchy
4 -=======================
5 -
6 -This document describes the class hierarchy of vLLM. We will explain the relationships between the core classes, their responsibilities, and the design choices behind them to make vLLM more modular and extensible.
7 -
8 -1. **Entrypoints**: vLLM has two entrypoints: `command line usage <https://github.com/vllm-project/vllm/blob/d1c6799b8870e513bf4f2305cbf6cda9fc3d773b/vllm/entrypoints/api_server.py#L138>`__ with ``vllm serve`` for launching an OpenAI-API compatible server, and `library-style usage <https://github.com/vllm-project/vllm/blob/d1c6799b8870e513bf4f2305cbf6cda9fc3d773b/vllm/entrypoints/llm.py#L38>`__ with the ``vllm.LLM`` class for running inference in a Python script. These are user-facing entrypoints that end-users interact with. Under the hood, both create an engine object to handle model inference.
9 -
10 -2. **Engine**: Each vLLM instance contains one engine object, orchestrating and serving as the control plane for model inference. Depending on the configuration, the engine can create multiple workers to handle the inference workload.
11 -
12 -3. **Worker**: A worker is a process that runs the model inference. vLLM follows the common practice of using one process to control one accelerator device, such as GPUs. For example, if we use tensor parallelism of size 2 and pipeline parallelism of size 2, we will have 4 workers in total. Workers are identified by their ``rank`` and ``local_rank``. ``rank`` is used for global orchestration, while ``local_rank`` is mainly used for assigning the accelerator device and accessing local resources such as the file system and shared memory.
13 -
14 -4. **Model Runner**: Every worker has one model runner object, responsible for loading and running the model. Much of the model execution logic resides here, such as preparing input tensors and capturing cudagraphs.
15 -
16 -5. **Model**: Every model runner object has one model object, which is the actual ``torch.nn.Module`` instance. See :ref:`huggingface_integration` for how various configurations affect the class we ultimately get.
17 -
18 -The following figure shows the class hierarchy of vLLM:
19 -
20 - .. figure:: ../assets/design/hierarchy.png
21 - :alt: query
22 - :width: 100%
23 - :align: center
24 -
25 -There are several important design choices behind this class hierarchy:
26 -
27 -1. **Extensibility**: All classes in the hierarchy accept a configuration object containing all the necessary information. The `VllmConfig <https://github.com/vllm-project/vllm/blob/d1c6799b8870e513bf4f2305cbf6cda9fc3d773b/vllm/config.py#L2036>`__ class is the main configuration object that is passed around. The class hierarchy is quite deep, and every class needs to read the configuration it is interested in. By encapsulating all configurations in one object, we can easily pass the configuration object around and access the configuration we need. Suppose we want to add a new feature (this is often the case given how fast the field of LLM inference is evolving) that only touches the model runner. We will have to add a new configuration option in the `VllmConfig` class. Since we pass the whole config object around, we only need to add the configuration option to the `VllmConfig` class, and the model runner can access it directly. We don't need to change the constructor of the engine, worker, or model class to pass the new configuration option.
28 -
29 -2. **Uniformity**: The model runner needs a unified interface to create and initialize the model. vLLM supports more than 50 types of popular open-source models. Each model has its own initialization logic. If the constructor signature varies with models, the model runner does not know how to call the constructor accordingly, without complicated and error-prone inspection logic. By making the constructor of the model class uniform, the model runner can easily create and initialize the model without knowing the specific model type. This is also useful for composing models. Vision-language models often consist of a vision model and a language model. By making the constructor uniform, we can easily create a vision model and a language model and compose them into a vision-language model.
30 -
31 -.. note::
32 -
33 - To support this change, all vLLM models' signatures have been updated to:
34 -
35 - .. code-block:: python
36 -
37 - def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
38 -
39 - To avoid accidentally passing incorrect arguments, the constructor is now keyword-only. This ensures that the constructor will raise an error if old configurations are passed. vLLM developers have already made this change for all models within vLLM. For out-of-tree registered models, developers need to update their models, for example by adding shim code to adapt the old constructor signature to the new one:
40 -
41 - .. code-block:: python
42 -
43 - class MyOldModel(nn.Module):
44 - def __init__(
45 - self,
46 - config,
47 - cache_config: Optional[CacheConfig] = None,
48 - quant_config: Optional[QuantizationConfig] = None,
49 - lora_config: Optional[LoRAConfig] = None,
50 - prefix: str = "",
51 - ) -> None:
52 - ...
53 -
54 - from vllm.config import VllmConfig
55 - class MyNewModel(MyOldModel):
56 - def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
57 - config = vllm_config.model_config.hf_config
58 - cache_config = vllm_config.cache_config
59 - quant_config = vllm_config.quant_config
60 - lora_config = vllm_config.lora_config
61 - super().__init__(config, cache_config, quant_config, lora_config, prefix)
62 -
63 - if __version__ >= "0.6.4":
64 - MyModel = MyNewModel
65 - else:
66 - MyModel = MyOldModel
67 -
68 - This way, the model can work with both old and new versions of vLLM.
69 -
70 -3. **Sharding and Quantization at Initialization**: Certain features require changing the model weights. For example, tensor parallelism needs to shard the model weights, and quantization needs to quantize the model weights. There are two possible ways to implement this feature. One way is to change the model weights after the model is initialized. The other way is to change the model weights during the model initialization. vLLM chooses the latter. The first approach is not scalable to large models. Suppose we want to run a 405B model (with roughly 810GB weights) with 16 H100 80GB GPUs. Ideally, every GPU should only load 50GB weights. If we change the model weights after the model is initialized, we need to load the full 810GB weights to every GPU and then shard the weights, leading to a huge memory overhead. Instead, if we shard the weights during the model initialization, every layer will only create a shard of the weights it needs, leading to a much smaller memory overhead. The same idea applies to quantization. Note that we also add an additional argument ``prefix`` to the model's constructor so that the model can initialize itself differently based on the prefix. This is useful for non-uniform quantization, where different parts of the model are quantized differently. The ``prefix`` is usually an empty string for the top-level model and a string like ``"vision"`` or ``"language"`` for the sub-models. In general, it matches the name of the module's state dict in the checkpoint file.
71 -
72 -One disadvantage of this design is that it is hard to write unit tests for individual components in vLLM because every component needs to be initialized by a complete config object. We solve this problem by providing a default initialization function that creates a default config object with all fields set to ``None``. If the component we want to test only cares about a few fields in the config object, we can create a default config object and set the fields we care about. This way, we can test the component in isolation. Note that many tests in vLLM are end-to-end tests that test the whole system, so this is not a big problem.
73 -
74 -In summary, the complete config object ``VllmConfig`` can be treated as an engine-level global state that is shared among all vLLM classes.
docs/source/design/plugin_system.rst
+2 -2
@@ -8,7 +8,7 @@ The community frequently requests the ability to extend vLLM with custom feature
8 How Plugins Work in vLLM
9 ------------------------
10
11 -Plugins are user-registered code that vLLM executes. Given vLLM's architecture (see :ref:`class_hierarchy`), multiple processes may be involved, especially when using distributed inference with various parallelism techniques. To enable plugins successfully, every process created by vLLM needs to load the plugin. This is done by the `load_general_plugins <https://github.com/vllm-project/vllm/blob/c76ac49d266e27aa3fea84ef2df1f813d24c91c7/vllm/plugins/__init__.py#L16>`__ function in the ``vllm.plugins`` module. This function is called for every process created by vLLM before it starts any work.
11 +Plugins are user-registered code that vLLM executes. Given vLLM's architecture (see :ref:`arch_overview`), multiple processes may be involved, especially when using distributed inference with various parallelism techniques. To enable plugins successfully, every process created by vLLM needs to load the plugin. This is done by the `load_general_plugins <https://github.com/vllm-project/vllm/blob/c76ac49d266e27aa3fea84ef2df1f813d24c91c7/vllm/plugins/__init__.py#L16>`__ function in the ``vllm.plugins`` module. This function is called for every process created by vLLM before it starts any work.
12
13 How vLLM Discovers Plugins
14 --------------------------
@@ -59,4 +59,4 @@ Guidelines for Writing Plugins
59 Compatibility Guarantee
60 -----------------------
61
62 -vLLM guarantees the interface of documented plugins, such as ``ModelRegistry.register_model``, will always be available for plugins to register models. However, it is the responsibility of plugin developers to ensure their plugins are compatible with the version of vLLM they are targeting. For example, ``"vllm_add_dummy_model.my_llava:MyLlava"`` should be compatible with the version of vLLM that the plugin targets. The interface for the model may change during vLLM's development.
\ No newline at end of file
62 +vLLM guarantees the interface of documented plugins, such as ``ModelRegistry.register_model``, will always be available for plugins to register models. However, it is the responsibility of plugin developers to ensure their plugins are compatible with the version of vLLM they are targeting. For example, ``"vllm_add_dummy_model.my_llava:MyLlava"`` should be compatible with the version of vLLM that the plugin targets. The interface for the model may change during vLLM's development.
docs/source/index.rst
+1 -1
@@ -157,7 +157,7 @@ Documentation
157 :maxdepth: 2
158 :caption: Design
159
160 - design/class_hierarchy
160 + design/arch_overview
161 design/huggingface_integration
162 design/plugin_system
163 design/input_processing/model_inputs_index
format.sh
+4
@@ -299,6 +299,10 @@ echo 'vLLM shellcheck:'
299 tools/shellcheck.sh
300 echo 'vLLM shellcheck: Done'
301
302 +echo 'excalidraw png check:'
303 +tools/png-lint.sh
304 +echo 'excalidraw png check: Done'
305 +
306 if ! git diff --quiet &>/dev/null; then
307 echo
308 echo "🔍🔍There are files changed by the format checker or by you that are not added and committed:"
tools/png-lint.sh new
+15
@@ -0,0 +1,15 @@
1 +#!/bin/bash
2 +
3 +# Ensure that *.excalidraw.png files have the excalidraw metadata
4 +# embedded in them. This ensures they can be loaded back into
5 +# the tool and edited in the future.
6 +
7 +find . -iname '*.excalidraw.png' | while read -r file; do
8 + if git check-ignore -q "$file"; then
9 + continue
10 + fi
11 + if ! grep -q "excalidraw+json" "$file"; then
12 + echo "$file was not exported from excalidraw with 'Embed Scene' enabled."
13 + exit 1
14 + fi
15 +done
vllm/engine/arg_utils.py
+1 -1
@@ -793,7 +793,7 @@ class EngineArgs:
793 type=str,
794 default=[],
795 help="The pattern(s) to ignore when loading the model."
796 - "Default to 'original/**/*' to avoid repeated loading of llama's "
796 + "Default to `original/**/*` to avoid repeated loading of llama's "
797 "checkpoints.")
798 parser.add_argument(
799 '--preemption-mode',