@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 191 lines 8.13 KB
Rendered Raw
1 # Python Multiprocessing
2
3 ## Debugging
4
5 Please see the [Troubleshooting](../usage/troubleshooting.md#python-multiprocessing)
6 page for information on known issues and how to solve them.
7
8 ## Introduction
9
10 !!! important
11 The source code references are to the state of the code at the time of writing in December 2024.
12
13 The use of Python multiprocessing in vLLM is complicated by:
14
15 - using vLLM as a library, which limits control over its internal code;
16 - incompatibilities between certain multiprocessing methods and vLLM dependencies.
17
18 This document describes how vLLM deals with these challenges.
19
20 ## Multiprocessing Methods
21
22 [Python multiprocessing methods](https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods) include:
23
24 - `spawn` - Spawn a new Python process. The default on Windows and macOS.
25 - `fork` - Use `os.fork()` to fork the Python interpreter. The default on
26 Linux for Python versions prior to 3.14.
27 - `forkserver` - Spawn a server process that will fork a new process on request.
28 The default on Linux for Python version 3.14 and newer.
29
30 ### Tradeoffs
31
32 `fork` is the fastest method, but is incompatible with dependencies that use
33 threads. If you are under macOS, using `fork` may cause the process to crash.
34
35 `spawn` is more compatible with dependencies, but can be problematic when vLLM
36 is used as a library. If the consuming code does not use a `__main__` guard
37 (`if __name__ == "__main__":`), the code will be inadvertently re-executed when vLLM
38 spawns a new process. This can lead to infinite recursion, among other problems.
39
40 `forkserver` will spawn a new server process that will fork new processes on
41 demand. This unfortunately has the same problem as `spawn` when vLLM is used as
42 a library. The server process is created as a spawned new process, which will
43 re-execute code not protected by a `__main__` guard.
44
45 For both `spawn` and `forkserver`, the process must not depend on inheriting any
46 global state as would be the case with `fork`.
47
48 ## Compatibility with Dependencies
49
50 Multiple vLLM dependencies indicate either a preference or requirement for using
51 `spawn`:
52
53 - <https://pytorch.org/docs/stable/notes/multiprocessing.html#cuda-in-multiprocessing>
54 - <https://pytorch.org/docs/stable/multiprocessing.html#sharing-cuda-tensors>
55 - <https://docs.habana.ai/en/latest/PyTorch/Getting_Started_with_PyTorch_and_Gaudi/Getting_Started_with_PyTorch.html?highlight=multiprocessing#torch-multiprocessing-for-dataloaders>
56
57 Known issues exist when using `fork` after initializing these dependencies.
58
59 ## Current State (v0)
60
61 The environment variable `VLLM_WORKER_MULTIPROC_METHOD` can be used to control which method is used by vLLM. The current default is `fork`.
62
63 - <https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/envs.py#L339-L342>
64
65 If the main process is controlled via the `vllm` command,
66 `spawn` is used because it's the most widely compatible.
67
68 - <https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/scripts.py#L123-L140>
69
70 The `multiproc_xpu_executor` forces the use of `spawn`.
71
72 - <https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/executor/multiproc_xpu_executor.py#L14-L18>
73
74 There are other miscellaneous places hard-coding the use of `spawn`:
75
76 - <https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/distributed/device_communicators/all_reduce_utils.py#L135>
77 - <https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/entrypoints/openai/api_server.py#L184>
78
79 Related PRs:
80
81 - <https://github.com/vllm-project/vllm/pull/8823>
82
83 ## Prior State in v1
84
85 There was an environment variable to control whether multiprocessing is used in
86 the v1 engine core, `VLLM_ENABLE_V1_MULTIPROCESSING`. This defaulted to off.
87
88 - <https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/envs.py#L452-L454>
89
90 When it was enabled, the v1 `LLMEngine` would create a new process to run the
91 engine core.
92
93 - <https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/v1/engine/llm_engine.py#L93-L95>
94 - <https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/v1/engine/llm_engine.py#L70-L77>
95 - <https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/v1/engine/core_client.py#L44-L45>
96
97 It was off by default for all the reasons mentioned above - compatibility with
98 dependencies and code using vLLM as a library.
99
100 ### Changes Made in v1
101
102 There is not an easy solution with Python's `multiprocessing` that will work
103 everywhere. As a first step, we can get v1 into a state where it does
104 "best effort" choice of multiprocessing method to maximize compatibility.
105
106 - Default to `fork`.
107 - Use `spawn` when we know we control the main process (`vllm` was executed).
108 - If we detect `cuda` was previously initialized, force `spawn` and emit a
109 warning. We know `fork` will break, so this is the best we can do.
110
111 The case that is known to still break in this scenario is code using vLLM as a
112 library that initializes `cuda` before calling vLLM. The warning we emit should
113 instruct users to either add a `__main__` guard or to disable multiprocessing.
114
115 If that known-failure case occurs, the user will see two messages that explain
116 what is happening. First, a log message from vLLM:
117
118 ```console
119 WARNING 12-11 14:50:37 multiproc_worker_utils.py:281] CUDA was previously
120 initialized. We must use the `spawn` multiprocessing start method. Setting
121 VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See
122 https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing
123 for more information.
124 ```
125
126 Second, Python itself will raise an exception with a nice explanation:
127
128 ```console
129 RuntimeError:
130 An attempt has been made to start a new process before the
131 current process has finished its bootstrapping phase.
132
133 This probably means that you are not using fork to start your
134 child processes and you have forgotten to use the proper idiom
135 in the main module:
136
137 if __name__ == '__main__':
138 freeze_support()
139 ...
140
141 The "freeze_support()" line can be omitted if the program
142 is not going to be frozen to produce an executable.
143
144 To fix this issue, refer to the "Safe importing of main module"
145 section in https://docs.python.org/3/library/multiprocessing.html
146 ```
147
148 ## Alternatives Considered
149
150 ### Detect if a `__main__` guard is present
151
152 It has been suggested that we could behave better if we could detect whether
153 code using vLLM as a library has a `__main__` guard in place. This
154 [post on Stack Overflow](https://stackoverflow.com/questions/77220442/multiprocessing-pool-in-a-python-class-without-name-main-guard)
155 was from a library author facing the same question.
156
157 It is possible to detect whether we are in the original, `__main__` process, or
158 a subsequent spawned process. However, it does not appear to be straight forward
159 to detect whether a `__main__` guard is present in the code.
160
161 This option has been discarded as impractical.
162
163 ### Use `forkserver`
164
165 At first it appears that `forkserver` is a nice solution to the problem.
166 However, the way it works presents the same challenges that `spawn` does when
167 vLLM is used as a library.
168
169 ### Force `spawn` all the time
170
171 One way to clean this up is to just force the use of `spawn` all the time and
172 document that the use of a `__main__` guard is required when using vLLM as a
173 library. This would unfortunately break existing code and make vLLM harder to
174 use, violating the desire to make the `LLM` class as easy as possible to use.
175
176 Instead of pushing this on our users, we will retain the complexity to do our
177 best to make things work.
178
179 ## Future Work
180
181 We may want to consider a different worker management approach in the future
182 that works around these challenges.
183
184 1. We could implement something `forkserver`-like, but have the process manager
185 be something we initially launch by running our own subprocess and a custom
186 entrypoint for worker management (launch a `vllm-manager` process).
187
188 2. We can explore other libraries that may better suit our needs. Examples to
189 consider:
190
191 - <https://github.com/joblib/loky>