@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 300 lines 14.4 KB
Rendered Raw
1 # Architecture Overview
2
3 This document provides an overview of the vLLM architecture.
4
5 [TOC]
6
7 ## Entrypoints
8
9 vLLM provides a number of entrypoints for interacting with the system. The
10 following diagram shows the relationship between them.
11
12 ![Entrypoints Diagram](../assets/design/arch_overview/entrypoints.excalidraw.png)
13
14 ### LLM Class
15
16 The LLM class provides the primary Python interface for doing offline inference,
17 which is interacting with a model without using a separate model inference
18 server.
19
20 Here is a sample of `LLM` class usage:
21
22 ??? code
23
24 ```python
25 from vllm import LLM, SamplingParams
26
27 # Define a list of input prompts
28 prompts = [
29 "Hello, my name is",
30 "The capital of France is",
31 "The largest ocean is",
32 ]
33
34 # Define sampling parameters
35 sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
36
37 # Initialize the LLM engine with the OPT-125M model
38 llm = LLM(model="facebook/opt-125m")
39
40 # Generate outputs for the input prompts
41 outputs = llm.generate(prompts, sampling_params)
42
43 # Print the generated outputs
44 for output in outputs:
45 prompt = output.prompt
46 generated_text = output.outputs[0].text
47 print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
48 ```
49
50 More API details can be found in the [Offline Inference](../api/README.md#offline-inference) section of the API docs.
51
52 The code for the `LLM` class can be found in [vllm/entrypoints/llm.py](../../vllm/entrypoints/llm.py).
53
54 ### Online Serving
55
56 The second primary interface to vLLM is via its online server.
57 This server can be started using the `vllm serve` command.
58
59 ```bash
60 vllm serve <model>
61 ```
62
63 The code for the `vllm` CLI can be found in [vllm/entrypoints/cli/main.py](../../vllm/entrypoints/cli/main.py).
64
65 More details on the API server can be found in the [Online Serving](../serving/online_serving/README.md) document.
66
67 ## V1 Process Architecture
68
69 vLLM V1 uses a multi-process architecture to separate concerns and maximize throughput. Understanding this architecture is important for properly sizing CPU resources in your deployment. The key processes are:
70
71 ### API Server Process
72
73 The API server process handles HTTP requests (e.g., the OpenAI-compatible API), performs input processing (tokenization, multi-modal data loading), and streams results back to clients. It communicates with the engine core process(es) via ZMQ sockets.
74
75 By default, there is **1 API server process**, but when data parallelism is used, the API server count automatically scales to match the data parallel size. This can also be manually configured with the `--api-server-count` flag. Each API server connects to **all** engine cores via ZMQ in a many-to-many topology, enabling any API server to route requests to any engine core. Each API server process uses multiple CPU threads for media loading (controlled by `VLLM_MEDIA_LOADING_THREAD_COUNT`, default 8).
76
77 The code can be found in [vllm/entrypoints/launchers/api_server](../../vllm/entrypoints/launchers/api_server) and [vllm/v1/utils.py](../../vllm/v1/utils.py).
78
79 ### Engine Core Process
80
81 The engine core process runs the scheduler, manages KV cache, and coordinates model execution across GPU workers. It runs a busy loop that continuously schedules requests and dispatches work to the GPU workers.
82
83 There is **1 engine core process per data parallel rank**. For example, with `--data-parallel-size 4`, there are 4 engine core processes.
84
85 The code can be found in [vllm/v1/engine/core.py](../../vllm/v1/engine/core.py) and [vllm/v1/engine/utils.py](../../vllm/v1/engine/utils.py).
86
87 ### GPU Worker Processes
88
89 Each GPU is managed by a dedicated worker process. The worker process loads model weights, executes forward passes, and manages GPU memory. Workers communicate with the engine core process that owns them.
90
91 There is **1 worker process per GPU**. The total number of GPU worker processes equals `tensor_parallel_size x pipeline_parallel_size` per engine core.
92
93 The code can be found in [vllm/v1/executor/multiproc_executor.py](../../vllm/v1/executor/multiproc_executor.py) and [vllm/v1/worker/gpu_worker.py](../../vllm/v1/worker/gpu_worker.py).
94
95 ### DP Coordinator Process (conditional)
96
97 When using data parallelism (`--data-parallel-size > 1`), an additional coordinator process manages load balancing across DP ranks and coordinates synchronized forward passes for MoE models.
98
99 There is **1 DP coordinator process** (only when data parallelism is enabled).
100
101 The code can be found in [vllm/v1/engine/coordinator.py](../../vllm/v1/engine/coordinator.py).
102
103 ### Process Count Summary
104
105 For a deployment with `N` GPUs, `TP` tensor parallel size, `DP` data parallel size, and `A` API server count:
106
107 | Process Type | Count | Notes |
108 | - | - | - |
109 | API Server | `A` (default `DP`) | Handles HTTP requests and input processing |
110 | Engine Core | `DP` (default 1) | Scheduler and KV cache management |
111 | GPU Worker | `N` (= `DP x PP x TP`) | One per GPU, executes model forward passes |
112 | DP Coordinator | 1 if `DP > 1`, else 0 | Load balancing across DP ranks |
113 | **Total** | **`A + DP + N` (+ 1 if DP > 1)** | |
114
115 For example, a typical single-node deployment with 4 GPUs (`vllm serve -tp=4`) has:
116
117 - 1 API server + 1 engine core + 4 GPU workers = **6 processes**
118
119 <figure markdown="1">
120 ![V1 Process Architecture - TP=4](../assets/design/arch_overview/v1_process_architecture_tp4.png)
121 </figure>
122
123 A data parallel deployment with 8 GPUs (`vllm serve -tp=2 -dp=4`) has:
124
125 - 4 API servers + 4 engine cores + 8 GPU workers + 1 DP coordinator = **17 processes**
126
127 <figure markdown="1">
128 ![V1 Process Architecture - TP=2, DP=4](../assets/design/arch_overview/v1_process_architecture_tp2_dp4.png)
129 </figure>
130
131 For CPU resource sizing recommendations, see
132 [CPU Resources for GPU Deployments](../configuration/optimization.md#cpu-resources-for-gpu-deployments).
133
134 ## LLM Engine
135
136 The `LLMEngine` and `AsyncLLMEngine` classes are central to the functioning of
137 the vLLM system, handling model inference and asynchronous request processing.
138
139 ![LLMEngine Diagram](../assets/design/arch_overview/llm_engine.excalidraw.png)
140
141 ### LLMEngine
142
143 The `LLMEngine` class is the core component of the vLLM engine. It is
144 responsible for receiving requests from clients and generating outputs from the
145 model. The `LLMEngine` includes input processing, model execution (possibly
146 distributed across multiple hosts and/or GPUs), scheduling, and output
147 processing.
148
149 - **Input Processing**: Handles tokenization of input text using the specified
150 tokenizer.
151 - **Scheduling**: Chooses which requests are processed in each step.
152 - **Model Execution**: Manages the execution of the language model, including
153 distributed execution across multiple GPUs.
154 - **Output Processing**: Processes the outputs generated by the model, decoding the
155 token IDs from a language model into human-readable text.
156
157 The code for `LLMEngine` can be found in [vllm/engine/llm_engine.py](../../vllm/engine/llm_engine.py).
158
159 ### AsyncLLMEngine
160
161 The `AsyncLLMEngine` class is an asynchronous wrapper for the `LLMEngine` class.
162 It uses `asyncio` to create a background loop that continuously processes
163 incoming requests. The `AsyncLLMEngine` is designed for online serving, where it
164 can handle multiple concurrent requests and stream outputs to clients.
165
166 The OpenAI-compatible API server uses the `AsyncLLMEngine`. There is also a demo
167 API server that serves as a simpler example in [examples/applications/api_server/server.py](../../examples/applications/api_server/server.py).
168
169 The code for `AsyncLLMEngine` can be found in [vllm/engine/async_llm_engine.py](../../vllm/engine/async_llm_engine.py).
170
171 ## Worker
172
173 A worker is a process that runs the model inference. vLLM follows the common
174 practice of using one process to control one accelerator device, such as GPUs.
175 For example, if we use tensor parallelism of size 2 and pipeline parallelism of
176 size 2, we will have 4 workers in total. Workers are identified by their
177 `rank` and `local_rank`. `rank` is used for global orchestration, while
178 `local_rank` is mainly used for assigning the accelerator device and accessing
179 local resources such as the file system and shared memory.
180
181 ## Model Runner
182
183 Every worker has one model runner object, responsible for loading and running
184 the model. Much of the model execution logic resides here, such as preparing
185 input tensors and capturing cudagraphs.
186
187 ## Model
188
189 Every model runner object has one model object, which is the actual
190 `torch.nn.Module` instance. See [huggingface_integration](huggingface_integration.md) for how various
191 configurations affect the class we ultimately get.
192
193 ## Class Hierarchy
194
195 The following figure shows the class hierarchy of vLLM:
196
197 ![Class Hierarchy](../assets/design/hierarchy.png)
198
199 There are several important design choices behind this class hierarchy:
200
201 1\. **Extensibility**: All classes in the hierarchy accept a configuration object
202 containing all the necessary information. The [VllmConfig](https://github.com/vllm-project/vllm/blob/d1c6799b8870e513bf4f2305cbf6cda9fc3d773b/vllm/config.py#L2036)
203 class is the main configuration object that is passed around. The class
204 hierarchy is quite deep, and every class needs to read the configuration it is
205 interested in. By encapsulating all configurations in one object, we can easily
206 pass the configuration object around and access the configuration we need.
207 Suppose we want to add a new feature (this is often the case given how fast the
208 field of LLM inference is evolving) that only touches the model runner. We will
209 have to add a new configuration option in the `VllmConfig` class. Since we pass
210 the whole config object around, we only need to add the configuration option to
211 the `VllmConfig` class, and the model runner can access it directly. We don't
212 need to change the constructor of the engine, worker, or model class to pass the
213 new configuration option.
214
215 2\. **Uniformity**: The model runner needs a unified interface to create and
216 initialize the model. vLLM supports more than 50 types of popular open-source
217 models. Each model has its own initialization logic. If the constructor
218 signature varies with models, the model runner does not know how to call the
219 constructor accordingly, without complicated and error-prone inspection logic.
220 By making the constructor of the model class uniform, the model runner can
221 easily create and initialize the model without knowing the specific model type.
222 This is also useful for composing models. Vision-language models often consist
223 of a vision model and a language model. By making the constructor uniform, we
224 can easily create a vision model and a language model and compose them into a
225 vision-language model.
226
227 !!! note
228 To support this change, all vLLM models' signatures have been updated to:
229
230 ```python
231 def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
232 ```
233
234 To avoid accidentally passing incorrect arguments, the constructor is now keyword-only. This ensures that the constructor will raise an error if old configurations are passed. vLLM developers have already made this change for all models within vLLM. For out-of-tree registered models, developers need to update their models, for example by adding shim code to adapt the old constructor signature to the new one:
235
236 ??? code
237
238 ```python
239 class MyOldModel(nn.Module):
240 def __init__(
241 self,
242 config,
243 cache_config: Optional[CacheConfig] = None,
244 quant_config: Optional[QuantizationConfig] = None,
245 lora_config: Optional[LoRAConfig] = None,
246 prefix: str = "",
247 ) -> None:
248 ...
249
250 from vllm.config import VllmConfig
251 class MyNewModel(MyOldModel):
252 def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
253 config = vllm_config.model_config.hf_config
254 cache_config = vllm_config.cache_config
255 quant_config = vllm_config.quant_config
256 lora_config = vllm_config.lora_config
257 super().__init__(config, cache_config, quant_config, lora_config, prefix)
258
259 from packaging import version
260 if version.parse(__version__) >= version.parse("0.6.4"):
261 MyModel = MyNewModel
262 else:
263 MyModel = MyOldModel
264 ```
265
266 This way, the model can work with both old and new versions of vLLM.
267
268 3\. **Sharding and Quantization at Initialization**: Certain features require
269 changing the model weights. For example, tensor parallelism needs to shard the
270 model weights, and quantization needs to quantize the model weights. There are
271 two possible ways to implement this feature. One way is to change the model
272 weights after the model is initialized. The other way is to change the model
273 weights during the model initialization. vLLM chooses the latter. The first
274 approach is not scalable to large models. Suppose we want to run a 405B model
275 (with roughly 810GB weights) with 16 H100 80GB GPUs. Ideally, every GPU should
276 only load 50GB weights. If we change the model weights after the model is
277 initialized, we need to load the full 810GB weights to every GPU and then shard
278 the weights, leading to a huge memory overhead. Instead, if we shard the weights
279 during the model initialization, every layer will only create a shard of the
280 weights it needs, leading to a much smaller memory overhead. The same idea
281 applies to quantization. Note that we also add an additional argument `prefix`
282 to the model's constructor so that the model can initialize itself differently
283 based on the prefix. This is useful for non-uniform quantization, where
284 different parts of the model are quantized differently. The `prefix` is
285 usually an empty string for the top-level model and a string like `"vision"`
286 or `"language"` for the sub-models. In general, it matches the name of the
287 module's state dict in the checkpoint file.
288
289 One disadvantage of this design is that it is hard to write unit tests for
290 individual components in vLLM because every component needs to be initialized by
291 a complete config object. We solve this problem by providing a default
292 initialization function that creates a default config object with all fields set
293 to `None`. If the component we want to test only cares about a few fields in
294 the config object, we can create a default config object and set the fields we
295 care about. This way, we can test the component in isolation. Note that many
296 tests in vLLM are end-to-end tests that test the whole system, so this is not a
297 big problem.
298
299 In summary, the complete config object `VllmConfig` can be treated as an
300 engine-level global state that is shared among all vLLM classes.