A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 04:30 UTC
[Core] V1: Use multiprocessing by default (#11074)
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Russell Bryant committed
Dec 13, 2024 at 19:27 UTC
4863e5fba51b8e1a5012e2a7582aece0ca575b89
10 files changed
+299
-17
docs/source/design/multiprocessing.md
new
+195
@@ -0,0 +1,195 @@
1
+# Python Multiprocessing
2
+
3
+## Debugging
4
+
5
+Please see the [Debugging
6
+Tips](https://docs.vllm.ai/en/latest/getting_started/debugging.html#python-multiprocessing)
7
+page for information on known issues and how to solve them.
8
+
9
+## Introduction
10
+
11
+*Note that source code references are to the state of the code at the time of writing in December, 2024.*
12
+
13
+The use of Python multiprocessing in vLLM is complicated by:
14
+
15
+- The use of vLLM as a library and the inability to control the code using vLLM
16
+- Varying levels of incompatibilities between multiprocessing methods and vLLM
17
+ dependencies
18
+
19
+This document describes how vLLM deals with these challenges.
20
+
21
+## Multiprocessing Methods
22
+
23
+[Python multiprocessing methods](https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods) include:
24
+
25
+- `spawn` - spawn a new Python process. This will be the default as of Python
26
+ 3.14.
27
+
28
+- `fork` - Use `os.fork()` to fork the Python interpreter. This is the default
29
+ in Python versions prior to 3.14.
30
+
31
+- `forkserver` - Spawn a server process that will fork a new process on request.
32
+
33
+### Tradeoffs
34
+
35
+`fork` is the fastest method, but is incompatible with dependencies that use
36
+threads.
37
+
38
+`spawn` is more compatible with dependencies, but can be problematic when vLLM
39
+is used as a library. If the consuming code does not use a `__main__` guard (`if
40
+__name__ == "__main__":`), the code will be inadvertently re-executed when vLLM
41
+spawns a new process. This can lead to infinite recursion, among other problems.
42
+
43
+`forkserver` will spawn a new server process that will fork new processes on
44
+demand. This unfortunately has the same problem as `spawn` when vLLM is used as
45
+a library. The server process is created as a spawned new process, which will
46
+re-execute code not protected by a `__main__` guard.
47
+
48
+For both `spawn` and `forkserver`, the process must not depend on inheriting any
49
+global state as would be the case with `fork`.
50
+
51
+## Compatibility with Dependencies
52
+
53
+Multiple vLLM dependencies indicate either a preference or requirement for using
54
+`spawn`:
55
+
56
+- <https://pytorch.org/docs/stable/notes/multiprocessing.html#cuda-in-multiprocessing>
57
+- <https://pytorch.org/docs/stable/multiprocessing.html#sharing-cuda-tensors>
58
+- <https://docs.habana.ai/en/latest/PyTorch/Getting_Started_with_PyTorch_and_Gaudi/Getting_Started_with_PyTorch.html?highlight=multiprocessing#torch-multiprocessing-for-dataloaders>
59
+
60
+It is perhaps more accurate to say that there are known problems with using
61
+`fork` after initializing these dependencies.
62
+
63
+## Current State (v0)
64
+
65
+The environment variable `VLLM_WORKER_MULTIPROC_METHOD` can be used to control which method is used by vLLM. The current default is `fork`.
66
+
67
+- <https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/envs.py#L339-L342>
68
+
69
+When we know we own the process because the `vllm` command was used, we use
70
+`spawn` because it's the most widely compatible.
71
+
72
+- <https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/scripts.py#L123-L140>
73
+
74
+The `multiproc_xpu_executor` forces the use of `spawn`.
75
+
76
+- <https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/executor/multiproc_xpu_executor.py#L14-L18>
77
+
78
+There are other miscellaneous places hard-coding the use of `spawn`:
79
+
80
+- <https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/distributed/device_communicators/custom_all_reduce_utils.py#L135>
81
+- <https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/entrypoints/openai/api_server.py#L184>
82
+
83
+Related PRs:
84
+
85
+- <https://github.com/vllm-project/vllm/pull/8823>
86
+
87
+## Prior State in v1
88
+
89
+There was an environment variable to control whether multiprocessing is used in
90
+the v1 engine core, `VLLM_ENABLE_V1_MULTIPROCESSING`. This defaulted to off.
91
+
92
+- <https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/envs.py#L452-L454>
93
+
94
+When it was enabled, the v1 `LLMEngine` would create a new process to run the
95
+engine core.
96
+
97
+- <https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/v1/engine/llm_engine.py#L93-L95>
98
+- <https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/v1/engine/llm_engine.py#L70-L77>
99
+- https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/v1/engine/core_client.py#L44-L45
100
+
101
+It was off by default for all the reasons mentioned above - compatibility with
102
+dependencies and code using vLLM as a library.
103
+
104
+### Changes Made in v1
105
+
106
+There is not an easy solution with Python's `multiprocessing` that will work
107
+everywhere. As a first step, we can get v1 into a state where it does "best
108
+effort" choice of multiprocessing method to maximize compatibility.
109
+
110
+- Default to `fork`.
111
+- Use `spawn` when we know we control the main process (`vllm` was executed).
112
+- If we detect `cuda` was previously initialized, force `spawn` and emit a
113
+ warning. We know `fork` will break, so this is the best we can do.
114
+
115
+The case that is known to still break in this scenario is code using vLLM as a
116
+library that initializes `cuda` before calling vLLM. The warning we emit should
117
+instruct users to either add a `__main__` guard or to disable multiprocessing.
118
+
119
+If that known-failure case occurs, the user will see two messages that explain
120
+what is happening. First, a log message from vLLM:
121
+
122
+```
123
+ WARNING 12-11 14:50:37 multiproc_worker_utils.py:281] CUDA was previously
124
+ initialized. We must use the `spawn` multiprocessing start method. Setting
125
+ VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See
126
+ https://docs.vllm.ai/en/latest/getting_started/debugging.html#python-multiprocessing
127
+ for more information.
128
+```
129
+
130
+Second, Python itself will raise an exception with a nice explanation:
131
+
132
+```
133
+RuntimeError:
134
+ An attempt has been made to start a new process before the
135
+ current process has finished its bootstrapping phase.
136
+
137
+ This probably means that you are not using fork to start your
138
+ child processes and you have forgotten to use the proper idiom
139
+ in the main module:
140
+
141
+ if __name__ == '__main__':
142
+ freeze_support()
143
+ ...
144
+
145
+ The "freeze_support()" line can be omitted if the program
146
+ is not going to be frozen to produce an executable.
147
+
148
+ To fix this issue, refer to the "Safe importing of main module"
149
+ section in https://docs.python.org/3/library/multiprocessing.html
150
+```
151
+
152
+## Alternatives Considered
153
+
154
+### Detect if a `__main__` guard is present
155
+
156
+It has been suggested that we could behave better if we could detect whether
157
+code using vLLM as a library has a `__main__` guard in place. This [post on
158
+stackoverflow](https://stackoverflow.com/questions/77220442/multiprocessing-pool-in-a-python-class-without-name-main-guard)
159
+was from a library author facing the same question.
160
+
161
+It is possible to detect whether we are in the original, `__main__` process, or
162
+a subsequent spawned process. However, it does not appear to be straight forward
163
+to detect whether a `__main__` guard is present in the code.
164
+
165
+This option has been discarded as impractical.
166
+
167
+### Use `forkserver`
168
+
169
+At first it appears that `forkserver` is a nice solution to the problem.
170
+However, the way it works presents the same challenges that `spawn` does when
171
+vLLM is used as a library.
172
+
173
+### Force `spawn` all the time
174
+
175
+One way to clean this up is to just force the use of `spawn` all the time and
176
+document that the use of a `__main__` guard is required when using vLLM as a
177
+library. This would unfortunately break existing code and make vLLM harder to
178
+use, violating the desire to make the `LLM` class as easy as possible to use.
179
+
180
+Instead of pushing this on our users, we will retain the complexity to do our
181
+best to make things work.
182
+
183
+## Future Work
184
+
185
+We may want to consider a different worker management approach in the future
186
+that works around these challenges.
187
+
188
+1. We could implement something `forkserver`-like, but have the process manager
189
+ be something we initially launch by running our own subprocess and a custom
190
+ entrypoint for worker management (launch a `vllm-manager` process).
191
+
192
+2. We can explore other libraries that may better suit our needs. Examples to
193
+ consider:
194
+
195
+- <https://github.com/joblib/loky>
docs/source/getting_started/debugging.rst
+56
@@ -136,6 +136,62 @@ If the test script hangs or crashes, usually it means the hardware/drivers are b
136
137
Adjust ``--nproc-per-node``, ``--nnodes``, and ``--node-rank`` according to your setup, being sure to execute different commands (with different ``--node-rank``) on different nodes.
138
139
+Python multiprocessing
140
+----------------------
141
+
142
+`RuntimeError` Exception
143
+^^^^^^^^^^^^^^^^^^^^^^^^
144
+
145
+If you have seen a warning in your logs like this:
146
+
147
+.. code-block:: console
148
+
149
+ WARNING 12-11 14:50:37 multiproc_worker_utils.py:281] CUDA was previously
150
+ initialized. We must use the `spawn` multiprocessing start method. Setting
151
+ VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See
152
+ https://docs.vllm.ai/en/latest/getting_started/debugging.html#python-multiprocessing
153
+ for more information.
154
+
155
+or an error from Python that looks like this:
156
+
157
+.. code-block:: console
158
+
159
+ RuntimeError:
160
+ An attempt has been made to start a new process before the
161
+ current process has finished its bootstrapping phase.
162
+
163
+ This probably means that you are not using fork to start your
164
+ child processes and you have forgotten to use the proper idiom
165
+ in the main module:
166
+
167
+ if __name__ == '__main__':
168
+ freeze_support()
169
+ ...
170
+
171
+ The "freeze_support()" line can be omitted if the program
172
+ is not going to be frozen to produce an executable.
173
+
174
+ To fix this issue, refer to the "Safe importing of main module"
175
+ section in https://docs.python.org/3/library/multiprocessing.html
176
+
177
+then you must update your Python code to guard usage of ``vllm`` behind a ``if
178
+__name__ == '__main__':`` block. For example, instead of this:
179
+
180
+.. code-block:: python
181
+
182
+ import vllm
183
+
184
+ llm = vllm.LLM(...)
185
+
186
+try this instead:
187
+
188
+.. code-block:: python
189
+
190
+ if __name__ == '__main__':
191
+ import vllm
192
+
193
+ llm = vllm.LLM(...)
194
+
195
Known Issues
196
----------------------------------------
197
- In ``v0.5.2``, ``v0.5.3``, and ``v0.5.3.post1``, there is a bug caused by `zmq <https://github.com/zeromq/pyzmq/issues/2000>`_ , which can occasionally cause vLLM to hang depending on the machine configuration. The solution is to upgrade to the latest version of ``vllm`` to include the `fix <https://github.com/vllm-project/vllm/pull/6759>`_.
docs/source/index.rst
+1
@@ -173,6 +173,7 @@ Documentation
173
design/input_processing/model_inputs_index
174
design/kernel/paged_attention
175
design/multimodal/multimodal_index
176
+ design/multiprocessing
177
178
.. For Developers: contributing to the vLLM project
179
vllm/entrypoints/llm.py
+4
@@ -232,6 +232,10 @@ class LLM:
232
233
self.request_counter = Counter()
234
235
+ def __del__(self):
236
+ if self.llm_engine and hasattr(self.llm_engine, "shutdown"):
237
+ self.llm_engine.shutdown()
238
+
239
@staticmethod
240
def get_engine_class() -> Type[LLMEngine]:
241
if envs.VLLM_USE_V1:
vllm/envs.py
+2
-2
@@ -69,7 +69,7 @@ if TYPE_CHECKING:
69
VLLM_SKIP_P2P_CHECK: bool = False
70
VLLM_DISABLED_KERNELS: List[str] = []
71
VLLM_USE_V1: bool = False
72
- VLLM_ENABLE_V1_MULTIPROCESSING: bool = False
72
+ VLLM_ENABLE_V1_MULTIPROCESSING: bool = True
73
VLLM_LOG_BATCHSIZE_INTERVAL: float = -1
74
75
@@ -460,7 +460,7 @@ environment_variables: Dict[str, Callable[[], Any]] = {
460
461
# If set, enable multiprocessing in LLM for the V1 code path.
462
"VLLM_ENABLE_V1_MULTIPROCESSING":
463
- lambda: bool(int(os.getenv("VLLM_ENABLE_V1_MULTIPROCESSING", "0"))),
463
+ lambda: bool(int(os.getenv("VLLM_ENABLE_V1_MULTIPROCESSING", "1"))),
464
"VLLM_LOG_BATCHSIZE_INTERVAL":
465
lambda: float(os.getenv("VLLM_LOG_BATCHSIZE_INTERVAL", "-1")),
466
}
vllm/executor/multiproc_worker_utils.py
+14
-6
@@ -274,7 +274,20 @@ def _add_prefix(file: TextIO, worker_name: str, pid: int) -> None:
274
file.write = write_with_prefix # type: ignore[method-assign]
275
276
277
+def _check_multiproc_method():
278
+ if (cuda_is_initialized()
279
+ and os.environ.get("VLLM_WORKER_MULTIPROC_METHOD") != "spawn"):
280
+ logger.warning("CUDA was previously initialized. We must use "
281
+ "the `spawn` multiprocessing start method. Setting "
282
+ "VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. "
283
+ "See https://docs.vllm.ai/en/latest/getting_started/"
284
+ "debugging.html#python-multiprocessing "
285
+ "for more information.")
286
+ os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"
287
+
288
+
289
def get_mp_context():
290
+ _check_multiproc_method()
291
mp_method = envs.VLLM_WORKER_MULTIPROC_METHOD
292
return multiprocessing.get_context(mp_method)
293
@@ -284,12 +297,7 @@ def set_multiprocessing_worker_envs(parallel_config):
297
in a multiprocessing environment. This should be called by the parent
298
process before worker processes are created"""
299
287
- if (cuda_is_initialized()
288
- and os.environ.get("VLLM_WORKER_MULTIPROC_METHOD") != "spawn"):
289
- logger.warning("CUDA was previously initialized. We must use "
290
- "the `spawn` multiprocessing start method. Setting "
291
- "VLLM_WORKER_MULTIPROC_METHOD to 'spawn'.")
292
- os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"
300
+ _check_multiproc_method()
301
302
# Configure thread parallelism if OMP_NUM_THREADS isn't set
303
#
vllm/v1/engine/core.py
+2
-6
@@ -1,4 +1,3 @@
1
-import multiprocessing
1
import pickle
2
import queue
3
import signal
@@ -13,6 +12,7 @@ import zmq.asyncio
12
from msgspec import msgpack
13
14
from vllm.config import CacheConfig, VllmConfig
15
+from vllm.executor.multiproc_worker_utils import get_mp_context
16
from vllm.logger import init_logger
17
from vllm.usage.usage_lib import UsageContext
18
from vllm.v1.core.scheduler import Scheduler
@@ -210,11 +210,7 @@ class EngineCoreProc(EngineCore):
210
output_path: str,
211
ready_path: str,
212
) -> EngineCoreProcHandle:
213
- # The current process might have CUDA context,
214
- # so we need to spawn a new process.
215
- # NOTE(rob): this is a problem for using EngineCoreProc w/
216
- # LLM, since we need a if __name__ == "__main__" guard.
217
- context = multiprocessing.get_context("spawn")
213
+ context = get_mp_context()
214
215
process_kwargs = {
216
"input_path": input_path,
vllm/v1/engine/core_client.py
+9
-2
@@ -159,10 +159,16 @@ class MPClient(EngineCoreClient):
159
atexit.register(self.shutdown)
160
161
def shutdown(self):
162
+ # During final garbage collection in process shutdown, atexit may be
163
+ # None.
164
+ if atexit:
165
+ # in case shutdown gets called via __del__ first
166
+ atexit.unregister(self.shutdown)
167
+
168
# Shut down the zmq context.
169
self.ctx.destroy(linger=0)
170
165
- if hasattr(self, "proc_handle"):
171
+ if hasattr(self, "proc_handle") and self.proc_handle:
172
# Shutdown the process if needed.
173
if self.proc_handle.proc.is_alive():
174
self.proc_handle.proc.terminate()
@@ -178,8 +184,9 @@ class MPClient(EngineCoreClient):
184
]
185
for ipc_socket in ipc_sockets:
186
socket_file = ipc_socket.replace("ipc://", "")
181
- if os.path.exists(socket_file):
187
+ if os and os.path.exists(socket_file):
188
os.remove(socket_file)
189
+ self.proc_handle = None
190
191
def __del__(self):
192
self.shutdown()
vllm/v1/engine/llm_engine.py
+7
@@ -196,3 +196,10 @@ class LLMEngine:
196
f"found type: {type(tokenizer_group)}")
197
198
return tokenizer_group
199
+
200
+ def __del__(self):
201
+ self.shutdown()
202
+
203
+ def shutdown(self):
204
+ if engine_core := getattr(self, "engine_core", None):
205
+ engine_core.shutdown()
vllm/v1/executor/multiproc_executor.py
+9
-1
@@ -163,6 +163,10 @@ class MultiprocExecutor:
163
termination and kill signals if needed."""
164
165
def wait_for_termination(procs, timeout):
166
+ if not time:
167
+ # If we are in late stage shutdown, the interpreter may replace
168
+ # `time` with `None`.
169
+ return all(not proc.is_alive() for proc in procs)
170
start_time = time.time()
171
while time.time() - start_time < timeout:
172
if all(not proc.is_alive() for proc in procs):
@@ -187,10 +191,14 @@ class MultiprocExecutor:
191
for w in self.workers:
192
# Remove the zmq ipc socket file
193
socket_path = w.ready_path.replace("ipc://", "")
190
- if os.path.exists(socket_path):
194
+ if os and os.path.exists(socket_path):
195
os.remove(socket_path)
196
197
def shutdown(self):
198
+ if atexit:
199
+ # in case shutdown was called explicitly, we don't need to call it
200
+ # again
201
+ atexit.unregister(self.shutdown)
202
"""Properly shut down the executor and its workers"""
203
if (hasattr(self, 'workers') and self.workers is not None):
204
for w in self.workers: #TODO: not sure if needed