@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 4863e5fba5
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 04:30 UTC

[Core] V1: Use multiprocessing by default (#11074)

Signed-off-by: Russell Bryant <rbryant@redhat.com>

Russell Bryant committed Dec 13, 2024 at 19:27 UTC 4863e5fba51b8e1a5012e2a7582aece0ca575b89
10 files changed +299 -17
docs/source/design/multiprocessing.md new
+195
@@ -0,0 +1,195 @@
1 +# Python Multiprocessing
2 +
3 +## Debugging
4 +
5 +Please see the [Debugging
6 +Tips](https://docs.vllm.ai/en/latest/getting_started/debugging.html#python-multiprocessing)
7 +page for information on known issues and how to solve them.
8 +
9 +## Introduction
10 +
11 +*Note that source code references are to the state of the code at the time of writing in December, 2024.*
12 +
13 +The use of Python multiprocessing in vLLM is complicated by:
14 +
15 +- The use of vLLM as a library and the inability to control the code using vLLM
16 +- Varying levels of incompatibilities between multiprocessing methods and vLLM
17 + dependencies
18 +
19 +This document describes how vLLM deals with these challenges.
20 +
21 +## Multiprocessing Methods
22 +
23 +[Python multiprocessing methods](https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods) include:
24 +
25 +- `spawn` - spawn a new Python process. This will be the default as of Python
26 + 3.14.
27 +
28 +- `fork` - Use `os.fork()` to fork the Python interpreter. This is the default
29 + in Python versions prior to 3.14.
30 +
31 +- `forkserver` - Spawn a server process that will fork a new process on request.
32 +
33 +### Tradeoffs
34 +
35 +`fork` is the fastest method, but is incompatible with dependencies that use
36 +threads.
37 +
38 +`spawn` is more compatible with dependencies, but can be problematic when vLLM
39 +is used as a library. If the consuming code does not use a `__main__` guard (`if
40 +__name__ == "__main__":`), the code will be inadvertently re-executed when vLLM
41 +spawns a new process. This can lead to infinite recursion, among other problems.
42 +
43 +`forkserver` will spawn a new server process that will fork new processes on
44 +demand. This unfortunately has the same problem as `spawn` when vLLM is used as
45 +a library. The server process is created as a spawned new process, which will
46 +re-execute code not protected by a `__main__` guard.
47 +
48 +For both `spawn` and `forkserver`, the process must not depend on inheriting any
49 +global state as would be the case with `fork`.
50 +
51 +## Compatibility with Dependencies
52 +
53 +Multiple vLLM dependencies indicate either a preference or requirement for using
54 +`spawn`:
55 +
56 +- <https://pytorch.org/docs/stable/notes/multiprocessing.html#cuda-in-multiprocessing>
57 +- <https://pytorch.org/docs/stable/multiprocessing.html#sharing-cuda-tensors>
58 +- <https://docs.habana.ai/en/latest/PyTorch/Getting_Started_with_PyTorch_and_Gaudi/Getting_Started_with_PyTorch.html?highlight=multiprocessing#torch-multiprocessing-for-dataloaders>
59 +
60 +It is perhaps more accurate to say that there are known problems with using
61 +`fork` after initializing these dependencies.
62 +
63 +## Current State (v0)
64 +
65 +The environment variable `VLLM_WORKER_MULTIPROC_METHOD` can be used to control which method is used by vLLM. The current default is `fork`.
66 +
67 +- <https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/envs.py#L339-L342>
68 +
69 +When we know we own the process because the `vllm` command was used, we use
70 +`spawn` because it's the most widely compatible.
71 +
72 +- <https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/scripts.py#L123-L140>
73 +
74 +The `multiproc_xpu_executor` forces the use of `spawn`.
75 +
76 +- <https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/executor/multiproc_xpu_executor.py#L14-L18>
77 +
78 +There are other miscellaneous places hard-coding the use of `spawn`:
79 +
80 +- <https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/distributed/device_communicators/custom_all_reduce_utils.py#L135>
81 +- <https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/entrypoints/openai/api_server.py#L184>
82 +
83 +Related PRs:
84 +
85 +- <https://github.com/vllm-project/vllm/pull/8823>
86 +
87 +## Prior State in v1
88 +
89 +There was an environment variable to control whether multiprocessing is used in
90 +the v1 engine core, `VLLM_ENABLE_V1_MULTIPROCESSING`. This defaulted to off.
91 +
92 +- <https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/envs.py#L452-L454>
93 +
94 +When it was enabled, the v1 `LLMEngine` would create a new process to run the
95 +engine core.
96 +
97 +- <https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/v1/engine/llm_engine.py#L93-L95>
98 +- <https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/v1/engine/llm_engine.py#L70-L77>
99 +- https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/v1/engine/core_client.py#L44-L45
100 +
101 +It was off by default for all the reasons mentioned above - compatibility with
102 +dependencies and code using vLLM as a library.
103 +
104 +### Changes Made in v1
105 +
106 +There is not an easy solution with Python's `multiprocessing` that will work
107 +everywhere. As a first step, we can get v1 into a state where it does "best
108 +effort" choice of multiprocessing method to maximize compatibility.
109 +
110 +- Default to `fork`.
111 +- Use `spawn` when we know we control the main process (`vllm` was executed).
112 +- If we detect `cuda` was previously initialized, force `spawn` and emit a
113 + warning. We know `fork` will break, so this is the best we can do.
114 +
115 +The case that is known to still break in this scenario is code using vLLM as a
116 +library that initializes `cuda` before calling vLLM. The warning we emit should
117 +instruct users to either add a `__main__` guard or to disable multiprocessing.
118 +
119 +If that known-failure case occurs, the user will see two messages that explain
120 +what is happening. First, a log message from vLLM:
121 +
122 +```
123 + WARNING 12-11 14:50:37 multiproc_worker_utils.py:281] CUDA was previously
124 + initialized. We must use the `spawn` multiprocessing start method. Setting
125 + VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See
126 + https://docs.vllm.ai/en/latest/getting_started/debugging.html#python-multiprocessing
127 + for more information.
128 +```
129 +
130 +Second, Python itself will raise an exception with a nice explanation:
131 +
132 +```
133 +RuntimeError:
134 + An attempt has been made to start a new process before the
135 + current process has finished its bootstrapping phase.
136 +
137 + This probably means that you are not using fork to start your
138 + child processes and you have forgotten to use the proper idiom
139 + in the main module:
140 +
141 + if __name__ == '__main__':
142 + freeze_support()
143 + ...
144 +
145 + The "freeze_support()" line can be omitted if the program
146 + is not going to be frozen to produce an executable.
147 +
148 + To fix this issue, refer to the "Safe importing of main module"
149 + section in https://docs.python.org/3/library/multiprocessing.html
150 +```
151 +
152 +## Alternatives Considered
153 +
154 +### Detect if a `__main__` guard is present
155 +
156 +It has been suggested that we could behave better if we could detect whether
157 +code using vLLM as a library has a `__main__` guard in place. This [post on
158 +stackoverflow](https://stackoverflow.com/questions/77220442/multiprocessing-pool-in-a-python-class-without-name-main-guard)
159 +was from a library author facing the same question.
160 +
161 +It is possible to detect whether we are in the original, `__main__` process, or
162 +a subsequent spawned process. However, it does not appear to be straight forward
163 +to detect whether a `__main__` guard is present in the code.
164 +
165 +This option has been discarded as impractical.
166 +
167 +### Use `forkserver`
168 +
169 +At first it appears that `forkserver` is a nice solution to the problem.
170 +However, the way it works presents the same challenges that `spawn` does when
171 +vLLM is used as a library.
172 +
173 +### Force `spawn` all the time
174 +
175 +One way to clean this up is to just force the use of `spawn` all the time and
176 +document that the use of a `__main__` guard is required when using vLLM as a
177 +library. This would unfortunately break existing code and make vLLM harder to
178 +use, violating the desire to make the `LLM` class as easy as possible to use.
179 +
180 +Instead of pushing this on our users, we will retain the complexity to do our
181 +best to make things work.
182 +
183 +## Future Work
184 +
185 +We may want to consider a different worker management approach in the future
186 +that works around these challenges.
187 +
188 +1. We could implement something `forkserver`-like, but have the process manager
189 + be something we initially launch by running our own subprocess and a custom
190 + entrypoint for worker management (launch a `vllm-manager` process).
191 +
192 +2. We can explore other libraries that may better suit our needs. Examples to
193 + consider:
194 +
195 +- <https://github.com/joblib/loky>
docs/source/getting_started/debugging.rst
+56
@@ -136,6 +136,62 @@ If the test script hangs or crashes, usually it means the hardware/drivers are b
136
137 Adjust ``--nproc-per-node``, ``--nnodes``, and ``--node-rank`` according to your setup, being sure to execute different commands (with different ``--node-rank``) on different nodes.
138
139 +Python multiprocessing
140 +----------------------
141 +
142 +`RuntimeError` Exception
143 +^^^^^^^^^^^^^^^^^^^^^^^^
144 +
145 +If you have seen a warning in your logs like this:
146 +
147 +.. code-block:: console
148 +
149 + WARNING 12-11 14:50:37 multiproc_worker_utils.py:281] CUDA was previously
150 + initialized. We must use the `spawn` multiprocessing start method. Setting
151 + VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See
152 + https://docs.vllm.ai/en/latest/getting_started/debugging.html#python-multiprocessing
153 + for more information.
154 +
155 +or an error from Python that looks like this:
156 +
157 +.. code-block:: console
158 +
159 + RuntimeError:
160 + An attempt has been made to start a new process before the
161 + current process has finished its bootstrapping phase.
162 +
163 + This probably means that you are not using fork to start your
164 + child processes and you have forgotten to use the proper idiom
165 + in the main module:
166 +
167 + if __name__ == '__main__':
168 + freeze_support()
169 + ...
170 +
171 + The "freeze_support()" line can be omitted if the program
172 + is not going to be frozen to produce an executable.
173 +
174 + To fix this issue, refer to the "Safe importing of main module"
175 + section in https://docs.python.org/3/library/multiprocessing.html
176 +
177 +then you must update your Python code to guard usage of ``vllm`` behind a ``if
178 +__name__ == '__main__':`` block. For example, instead of this:
179 +
180 +.. code-block:: python
181 +
182 + import vllm
183 +
184 + llm = vllm.LLM(...)
185 +
186 +try this instead:
187 +
188 +.. code-block:: python
189 +
190 + if __name__ == '__main__':
191 + import vllm
192 +
193 + llm = vllm.LLM(...)
194 +
195 Known Issues
196 ----------------------------------------
197 - In ``v0.5.2``, ``v0.5.3``, and ``v0.5.3.post1``, there is a bug caused by `zmq <https://github.com/zeromq/pyzmq/issues/2000>`_ , which can occasionally cause vLLM to hang depending on the machine configuration. The solution is to upgrade to the latest version of ``vllm`` to include the `fix <https://github.com/vllm-project/vllm/pull/6759>`_.
docs/source/index.rst
+1
@@ -173,6 +173,7 @@ Documentation
173 design/input_processing/model_inputs_index
174 design/kernel/paged_attention
175 design/multimodal/multimodal_index
176 + design/multiprocessing
177
178 .. For Developers: contributing to the vLLM project
179
vllm/entrypoints/llm.py
+4
@@ -232,6 +232,10 @@ class LLM:
232
233 self.request_counter = Counter()
234
235 + def __del__(self):
236 + if self.llm_engine and hasattr(self.llm_engine, "shutdown"):
237 + self.llm_engine.shutdown()
238 +
239 @staticmethod
240 def get_engine_class() -> Type[LLMEngine]:
241 if envs.VLLM_USE_V1:
vllm/envs.py
+2 -2
@@ -69,7 +69,7 @@ if TYPE_CHECKING:
69 VLLM_SKIP_P2P_CHECK: bool = False
70 VLLM_DISABLED_KERNELS: List[str] = []
71 VLLM_USE_V1: bool = False
72 - VLLM_ENABLE_V1_MULTIPROCESSING: bool = False
72 + VLLM_ENABLE_V1_MULTIPROCESSING: bool = True
73 VLLM_LOG_BATCHSIZE_INTERVAL: float = -1
74
75
@@ -460,7 +460,7 @@ environment_variables: Dict[str, Callable[[], Any]] = {
460
461 # If set, enable multiprocessing in LLM for the V1 code path.
462 "VLLM_ENABLE_V1_MULTIPROCESSING":
463 - lambda: bool(int(os.getenv("VLLM_ENABLE_V1_MULTIPROCESSING", "0"))),
463 + lambda: bool(int(os.getenv("VLLM_ENABLE_V1_MULTIPROCESSING", "1"))),
464 "VLLM_LOG_BATCHSIZE_INTERVAL":
465 lambda: float(os.getenv("VLLM_LOG_BATCHSIZE_INTERVAL", "-1")),
466 }
vllm/executor/multiproc_worker_utils.py
+14 -6
@@ -274,7 +274,20 @@ def _add_prefix(file: TextIO, worker_name: str, pid: int) -> None:
274 file.write = write_with_prefix # type: ignore[method-assign]
275
276
277 +def _check_multiproc_method():
278 + if (cuda_is_initialized()
279 + and os.environ.get("VLLM_WORKER_MULTIPROC_METHOD") != "spawn"):
280 + logger.warning("CUDA was previously initialized. We must use "
281 + "the `spawn` multiprocessing start method. Setting "
282 + "VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. "
283 + "See https://docs.vllm.ai/en/latest/getting_started/"
284 + "debugging.html#python-multiprocessing "
285 + "for more information.")
286 + os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"
287 +
288 +
289 def get_mp_context():
290 + _check_multiproc_method()
291 mp_method = envs.VLLM_WORKER_MULTIPROC_METHOD
292 return multiprocessing.get_context(mp_method)
293
@@ -284,12 +297,7 @@ def set_multiprocessing_worker_envs(parallel_config):
297 in a multiprocessing environment. This should be called by the parent
298 process before worker processes are created"""
299
287 - if (cuda_is_initialized()
288 - and os.environ.get("VLLM_WORKER_MULTIPROC_METHOD") != "spawn"):
289 - logger.warning("CUDA was previously initialized. We must use "
290 - "the `spawn` multiprocessing start method. Setting "
291 - "VLLM_WORKER_MULTIPROC_METHOD to 'spawn'.")
292 - os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"
300 + _check_multiproc_method()
301
302 # Configure thread parallelism if OMP_NUM_THREADS isn't set
303 #
vllm/v1/engine/core.py
+2 -6
@@ -1,4 +1,3 @@
1 -import multiprocessing
1 import pickle
2 import queue
3 import signal
@@ -13,6 +12,7 @@ import zmq.asyncio
12 from msgspec import msgpack
13
14 from vllm.config import CacheConfig, VllmConfig
15 +from vllm.executor.multiproc_worker_utils import get_mp_context
16 from vllm.logger import init_logger
17 from vllm.usage.usage_lib import UsageContext
18 from vllm.v1.core.scheduler import Scheduler
@@ -210,11 +210,7 @@ class EngineCoreProc(EngineCore):
210 output_path: str,
211 ready_path: str,
212 ) -> EngineCoreProcHandle:
213 - # The current process might have CUDA context,
214 - # so we need to spawn a new process.
215 - # NOTE(rob): this is a problem for using EngineCoreProc w/
216 - # LLM, since we need a if __name__ == "__main__" guard.
217 - context = multiprocessing.get_context("spawn")
213 + context = get_mp_context()
214
215 process_kwargs = {
216 "input_path": input_path,
vllm/v1/engine/core_client.py
+9 -2
@@ -159,10 +159,16 @@ class MPClient(EngineCoreClient):
159 atexit.register(self.shutdown)
160
161 def shutdown(self):
162 + # During final garbage collection in process shutdown, atexit may be
163 + # None.
164 + if atexit:
165 + # in case shutdown gets called via __del__ first
166 + atexit.unregister(self.shutdown)
167 +
168 # Shut down the zmq context.
169 self.ctx.destroy(linger=0)
170
165 - if hasattr(self, "proc_handle"):
171 + if hasattr(self, "proc_handle") and self.proc_handle:
172 # Shutdown the process if needed.
173 if self.proc_handle.proc.is_alive():
174 self.proc_handle.proc.terminate()
@@ -178,8 +184,9 @@ class MPClient(EngineCoreClient):
184 ]
185 for ipc_socket in ipc_sockets:
186 socket_file = ipc_socket.replace("ipc://", "")
181 - if os.path.exists(socket_file):
187 + if os and os.path.exists(socket_file):
188 os.remove(socket_file)
189 + self.proc_handle = None
190
191 def __del__(self):
192 self.shutdown()
vllm/v1/engine/llm_engine.py
+7
@@ -196,3 +196,10 @@ class LLMEngine:
196 f"found type: {type(tokenizer_group)}")
197
198 return tokenizer_group
199 +
200 + def __del__(self):
201 + self.shutdown()
202 +
203 + def shutdown(self):
204 + if engine_core := getattr(self, "engine_core", None):
205 + engine_core.shutdown()
vllm/v1/executor/multiproc_executor.py
+9 -1
@@ -163,6 +163,10 @@ class MultiprocExecutor:
163 termination and kill signals if needed."""
164
165 def wait_for_termination(procs, timeout):
166 + if not time:
167 + # If we are in late stage shutdown, the interpreter may replace
168 + # `time` with `None`.
169 + return all(not proc.is_alive() for proc in procs)
170 start_time = time.time()
171 while time.time() - start_time < timeout:
172 if all(not proc.is_alive() for proc in procs):
@@ -187,10 +191,14 @@ class MultiprocExecutor:
191 for w in self.workers:
192 # Remove the zmq ipc socket file
193 socket_path = w.ready_path.replace("ipc://", "")
190 - if os.path.exists(socket_path):
194 + if os and os.path.exists(socket_path):
195 os.remove(socket_path)
196
197 def shutdown(self):
198 + if atexit:
199 + # in case shutdown was called explicitly, we don't need to call it
200 + # again
201 + atexit.unregister(self.shutdown)
202 """Properly shut down the executor and its workers"""
203 if (hasattr(self, 'workers') and self.workers is not None):
204 for w in self.workers: #TODO: not sure if needed