A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 8, 17:45 UTC
[Docs] Add sections on process architecture and minimum CPU resources (#33940)
It seems users can be confused about vLLM's performance when running with very small amounts of CPU cores available. We are missing a clear overview of what vLLM's process architecture is, so I added this along with some diagrams in arch_overview.md, and included a section on CPU resource recommendations in optimization.md Signed-off-by: mgoin <mgoin64@gmail.com>
Michael Goin committed
Feb 6, 2026 at 10:26 UTC
c39ee9ee2ba1e004610f671212b62f2572311da0
4 files changed
+113
docs/assets/design/arch_overview/v1_process_architecture_tp2_dp4.png
Binary files /dev/null and b/docs/assets/design/arch_overview/v1_process_architecture_tp2_dp4.png differ
docs/assets/design/arch_overview/v1_process_architecture_tp4.png
Binary files /dev/null and b/docs/assets/design/arch_overview/v1_process_architecture_tp4.png differ
docs/configuration/optimization.md
+46
@@ -291,6 +291,52 @@ Based on the configuration, the content of the multi-modal caches on `P0` and `P
291
K: Stores the hashes of multi-modal items
292
V: Stores the processed tensor data of multi-modal items
293
294
+## CPU Resources for GPU Deployments
295
+
296
+vLLM V1 uses a multi-process architecture (see [V1 Process Architecture](../design/arch_overview.md#v1-process-architecture)) where each process requires CPU resources. Underprovisioning CPU cores is a common source of performance degradation, especially in virtualized environments.
297
+
298
+### Minimum CPU Requirements
299
+
300
+For a deployment with `N` GPUs, there are at minimum:
301
+
302
+- **1 API server process** -- handles HTTP requests, tokenization, and input processing
303
+- **1 engine core process** -- runs the scheduler and coordinates GPU workers
304
+- **N GPU worker processes** -- one per GPU, executes model forward passes
305
+
306
+This means there are always at least **`2 + N` processes** competing for CPU time.
307
+
308
+!!! warning
309
+ Using fewer physical CPU cores than processes will cause contention and significantly degrade throughput and latency. The engine core process runs a busy loop and is particularly sensitive to CPU starvation.
310
+
311
+The minimum is `2 + N` physical cores (1 for the API server, 1 for the engine core, and 1 per GPU worker). In practice, allocating more cores improves performance because the OS, PyTorch background threads, and other system processes also need CPU time.
312
+
313
+!!! important
314
+ Please note we are referring to **physical CPU cores** here. If your system has hyperthreading enabled, then 1 vCPU = 1 hyperthread = 1/2 physical CPU core, so you need `2 x (2 + N)` minimum vCPUs.
315
+
316
+### Data Parallel and Multi-API Server Deployments
317
+
318
+When using data parallelism or multiple API servers, the CPU requirements increase:
319
+
320
+```console
321
+Minimum physical cores = A + DP + N + (1 if DP > 1 else 0)
322
+```
323
+
324
+where `A` is the API server count (defaults to `DP`), `DP` is the data parallel size, and `N` is the total number of GPUs. For example, with `DP=4, TP=2` on 8 GPUs:
325
+
326
+```console
327
+4 API servers + 4 engine cores + 8 GPU workers + 1 DP coordinator = 17 processes
328
+```
329
+
330
+### Performance Impact
331
+
332
+CPU underprovisioning particularly impacts:
333
+
334
+- **Input processing throughput** -- tokenization, chat template rendering, and multi-modal data loading all run on CPU
335
+- **Scheduling latency** -- the engine core scheduler runs on CPU and directly affects how quickly new tokens are dispatched to the GPU workers
336
+- **Output processing** -- detokenization, networking, and especially streaming token responses use CPU cycles
337
+
338
+If you observe that GPU utilization is lower than expected, CPU contention may be the bottleneck. Increasing the number of available CPU cores and even the clock speed can significantly improve end-to-end performance.
339
+
340
## Attention Backend Selection
341
342
vLLM supports multiple attention backends optimized for different hardware and use cases. The backend is automatically selected based on your GPU architecture, model type, and configuration, but you can also manually specify one for optimal performance.
docs/design/arch_overview.md
+67
@@ -78,6 +78,73 @@ That code can be found in [vllm/entrypoints/openai/api_server.py](../../vllm/ent
78
79
More details on the API server can be found in the [OpenAI-Compatible Server](../serving/openai_compatible_server.md) document.
80
81
+## V1 Process Architecture
82
+
83
+vLLM V1 uses a multi-process architecture to separate concerns and maximize throughput. Understanding this architecture is important for properly sizing CPU resources in your deployment. The key processes are:
84
+
85
+### API Server Process
86
+
87
+The API server process handles HTTP requests (e.g., the OpenAI-compatible API), performs input processing (tokenization, multi-modal data loading), and streams results back to clients. It communicates with the engine core process(es) via ZMQ sockets.
88
+
89
+By default, there is **1 API server process**, but when data parallelism is used, the API server count automatically scales to match the data parallel size. This can also be manually configured with the `--api-server-count` flag. Each API server connects to **all** engine cores via ZMQ in a many-to-many topology, enabling any API server to route requests to any engine core. Each API server process uses multiple CPU threads for media loading (controlled by `VLLM_MEDIA_LOADING_THREAD_COUNT`, default 8).
90
+
91
+The code can be found in [vllm/entrypoints/openai/api_server.py](../../vllm/entrypoints/openai/api_server.py) and [vllm/v1/utils.py](../../vllm/v1/utils.py).
92
+
93
+### Engine Core Process
94
+
95
+The engine core process runs the scheduler, manages KV cache, and coordinates model execution across GPU workers. It runs a busy loop that continuously schedules requests and dispatches work to the GPU workers.
96
+
97
+There is **1 engine core process per data parallel rank**. For example, with `--data-parallel-size 4`, there are 4 engine core processes.
98
+
99
+The code can be found in [vllm/v1/engine/core.py](../../vllm/v1/engine/core.py) and [vllm/v1/engine/utils.py](../../vllm/v1/engine/utils.py).
100
+
101
+### GPU Worker Processes
102
+
103
+Each GPU is managed by a dedicated worker process. The worker process loads model weights, executes forward passes, and manages GPU memory. Workers communicate with the engine core process that owns them.
104
+
105
+There is **1 worker process per GPU**. The total number of GPU worker processes equals `tensor_parallel_size x pipeline_parallel_size` per engine core.
106
+
107
+The code can be found in [vllm/v1/executor/multiproc_executor.py](../../vllm/v1/executor/multiproc_executor.py) and [vllm/v1/worker/gpu_worker.py](../../vllm/v1/worker/gpu_worker.py).
108
+
109
+### DP Coordinator Process (conditional)
110
+
111
+When using data parallelism (`--data-parallel-size > 1`), an additional coordinator process manages load balancing across DP ranks and coordinates synchronized forward passes for MoE models.
112
+
113
+There is **1 DP coordinator process** (only when data parallelism is enabled).
114
+
115
+The code can be found in [vllm/v1/engine/coordinator.py](../../vllm/v1/engine/coordinator.py).
116
+
117
+### Process Count Summary
118
+
119
+For a deployment with `N` GPUs, `TP` tensor parallel size, `DP` data parallel size, and `A` API server count:
120
+
121
+| Process Type | Count | Notes |
122
+|---|---|---|
123
+| API Server | `A` (default `DP`) | Handles HTTP requests and input processing |
124
+| Engine Core | `DP` (default 1) | Scheduler and KV cache management |
125
+| GPU Worker | `N` (= `DP x TP`) | One per GPU, executes model forward passes |
126
+| DP Coordinator | 1 if `DP > 1`, else 0 | Load balancing across DP ranks |
127
+| **Total** | **`A + DP + N` (+ 1 if DP > 1)** | |
128
+
129
+For example, a typical single-node deployment with 4 GPUs (`vllm serve -tp=4`) has:
130
+
131
+- 1 API server + 1 engine core + 4 GPU workers = **6 processes**
132
+
133
+<figure markdown="1">
134
+
135
+</figure>
136
+
137
+A data parallel deployment with 8 GPUs (`vllm serve -tp=2 -dp=4`) has:
138
+
139
+- 4 API servers + 4 engine cores + 8 GPU workers + 1 DP coordinator = **17 processes**
140
+
141
+<figure markdown="1">
142
+
143
+</figure>
144
+
145
+For CPU resource sizing recommendations, see
146
+[CPU Resources for GPU Deployments](../configuration/optimization.md#cpu-resources-for-gpu-deployments).
147
+
148
## LLM Engine
149
150
The `LLMEngine` and `AsyncLLMEngine` classes are central to the functioning of