A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 7, 06:30 UTC
[Docs] Update design/multiprocessing.md (#30677)
Signed-off-by: windsonsea <haifeng.yao@daocloud.io>
Michael Yao committed
Mar 5, 2026 at 01:58 UTC
fd3bfe74c972bccc3c7c45cb3be44cb4c3a26090
1 file changed
+13
-17
docs/design/multiprocessing.md
+13
-17
@@ -12,9 +12,8 @@ page for information on known issues and how to solve them.
12
13
The use of Python multiprocessing in vLLM is complicated by:
14
15
-- The use of vLLM as a library and the inability to control the code using vLLM
16
-- Varying levels of incompatibilities between multiprocessing methods and vLLM
17
- dependencies
15
+- using vLLM as a library, which limits control over its internal code;
16
+- incompatibilities between certain multiprocessing methods and vLLM dependencies.
17
18
This document describes how vLLM deals with these challenges.
19
@@ -22,11 +21,9 @@ This document describes how vLLM deals with these challenges.
21
22
[Python multiprocessing methods](https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods) include:
23
25
-- `spawn` - spawn a new Python process. The default on Windows and macOS.
26
-
24
+- `spawn` - Spawn a new Python process. The default on Windows and macOS.
25
- `fork` - Use `os.fork()` to fork the Python interpreter. The default on
26
Linux for Python versions prior to 3.14.
29
-
27
- `forkserver` - Spawn a server process that will fork a new process on request.
28
The default on Linux for Python version 3.14 and newer.
29
@@ -36,8 +33,8 @@ This document describes how vLLM deals with these challenges.
33
threads. If you are under macOS, using `fork` may cause the process to crash.
34
35
`spawn` is more compatible with dependencies, but can be problematic when vLLM
39
-is used as a library. If the consuming code does not use a `__main__` guard (`if
40
-__name__ == "__main__":`), the code will be inadvertently re-executed when vLLM
36
+is used as a library. If the consuming code does not use a `__main__` guard
37
+(`if __name__ == "__main__":`), the code will be inadvertently re-executed when vLLM
38
spawns a new process. This can lead to infinite recursion, among other problems.
39
40
`forkserver` will spawn a new server process that will fork new processes on
@@ -57,8 +54,7 @@ Multiple vLLM dependencies indicate either a preference or requirement for using
54
- <https://pytorch.org/docs/stable/multiprocessing.html#sharing-cuda-tensors>
55
- <https://docs.habana.ai/en/latest/PyTorch/Getting_Started_with_PyTorch_and_Gaudi/Getting_Started_with_PyTorch.html?highlight=multiprocessing#torch-multiprocessing-for-dataloaders>
56
60
-It is perhaps more accurate to say that there are known problems with using
61
-`fork` after initializing these dependencies.
57
+Known issues exist when using `fork` after initializing these dependencies.
58
59
## Current State (v0)
60
@@ -66,8 +62,8 @@ The environment variable `VLLM_WORKER_MULTIPROC_METHOD` can be used to control w
62
63
- <https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/envs.py#L339-L342>
64
69
-When we know we own the process because the `vllm` command was used, we use
70
-`spawn` because it's the most widely compatible.
65
+If the main process is controlled via the `vllm` command,
66
+`spawn` is used because it's the most widely compatible.
67
68
- <https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/scripts.py#L123-L140>
69
@@ -104,8 +100,8 @@ dependencies and code using vLLM as a library.
100
### Changes Made in v1
101
102
There is not an easy solution with Python's `multiprocessing` that will work
107
-everywhere. As a first step, we can get v1 into a state where it does "best
108
-effort" choice of multiprocessing method to maximize compatibility.
103
+everywhere. As a first step, we can get v1 into a state where it does
104
+"best effort" choice of multiprocessing method to maximize compatibility.
105
106
- Default to `fork`.
107
- Use `spawn` when we know we control the main process (`vllm` was executed).
@@ -154,8 +150,8 @@ RuntimeError:
150
### Detect if a `__main__` guard is present
151
152
It has been suggested that we could behave better if we could detect whether
157
-code using vLLM as a library has a `__main__` guard in place. This [post on
158
-stackoverflow](https://stackoverflow.com/questions/77220442/multiprocessing-pool-in-a-python-class-without-name-main-guard)
153
+code using vLLM as a library has a `__main__` guard in place. This
154
+[post on Stack Overflow](https://stackoverflow.com/questions/77220442/multiprocessing-pool-in-a-python-class-without-name-main-guard)
155
was from a library author facing the same question.
156
157
It is possible to detect whether we are in the original, `__main__` process, or
@@ -192,4 +188,4 @@ that works around these challenges.
188
2. We can explore other libraries that may better suit our needs. Examples to
189
consider:
190
195
-- <https://github.com/joblib/loky>
191
+ - <https://github.com/joblib/loky>