A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 17:15 UTC
[Bugfix] Fix failure to launch in Tensor Parallel TP mode on macOS. (#14948)
Signed-off-by: Kebe <mail@kebe7jun.com> Signed-off-by: youkaichao <youkaichao@gmail.com> Co-authored-by: youkaichao <youkaichao@gmail.com>
Kebe committed
Mar 28, 2025 at 10:13 UTC
4e0f6076be71532272f114429a24a559f2656bef
3 files changed
+17
-4
docs/source/design/multiprocessing.md
+2
-2
@@ -24,7 +24,7 @@ This document describes how vLLM deals with these challenges.
24
[Python multiprocessing methods](https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods) include:
25
26
- `spawn` - spawn a new Python process. This will be the default as of Python
27
- 3.14.
27
+ 3.14. In macOS, this is already the default.
28
29
- `fork` - Use `os.fork()` to fork the Python interpreter. This is the default
30
in Python versions prior to 3.14.
@@ -34,7 +34,7 @@ This document describes how vLLM deals with these challenges.
34
### Tradeoffs
35
36
`fork` is the fastest method, but is incompatible with dependencies that use
37
-threads.
37
+threads. If you are under macOS, using `fork` may cause the process to crash.
38
39
`spawn` is more compatible with dependencies, but can be problematic when vLLM
40
is used as a library. If the consuming code does not use a `__main__` guard (`if
vllm/distributed/device_communicators/shm_broadcast.py
+7
-2
@@ -125,8 +125,13 @@ class ShmRingBuffer:
125
lambda *args, **kwargs: None):
126
try:
127
self.shared_memory = shared_memory.SharedMemory(name=name)
128
- assert (
129
- self.shared_memory.size == self.total_bytes_of_buffer)
128
+ # See https://docs.python.org/3/library/multiprocessing.shared_memory.html # noqa
129
+ # Some platforms allocate memory based on page size,
130
+ # so the shared memory block size may be larger or equal
131
+ # to the requested size. The size parameter is ignored
132
+ # when attaching to an existing block.
133
+ assert (self.shared_memory.size
134
+ >= self.total_bytes_of_buffer)
135
except FileNotFoundError:
136
# we might deserialize the object in a different node
137
# in this case, this object is not used,
vllm/platforms/cpu.py
+8
@@ -1,6 +1,7 @@
1
# SPDX-License-Identifier: Apache-2.0
2
3
import os
4
+import sys
5
from typing import TYPE_CHECKING, Optional
6
7
import psutil
@@ -148,6 +149,13 @@ class CpuPlatform(Platform):
149
# To hint IPEX uses shared memory based AllReduce
150
os.environ["LOCAL_WORLD_SIZE"] = str(
151
vllm_config.parallel_config.tensor_parallel_size)
152
+ if sys.platform == "darwin" and \
153
+ envs.VLLM_WORKER_MULTIPROC_METHOD == "fork":
154
+ if os.environ.get('VLLM_WORKER_MULTIPROC_METHOD', None) is None:
155
+ logger.warning(
156
+ "Default to spawn method on MacOS. If this is not desired,"
157
+ " set VLLM_WORKER_MULTIPROC_METHOD to fork explicitly.")
158
+ os.environ['VLLM_WORKER_MULTIPROC_METHOD'] = 'spawn'
159
160
@classmethod
161
def is_pin_memory_available(cls) -> bool: