A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 11, 12:30 UTC
[Doc] Add documents for multi-node distributed serving with MP backend (#30509)
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Isotr0py committed
Dec 14, 2025 at 02:02 UTC
7c16f3fbcc45e95491b90811fe9af1e6dfe297bc
2 files changed
+24
-4
docs/serving/parallelism_scaling.md
+23
-1
@@ -62,7 +62,7 @@ If a single node lacks sufficient GPUs to hold the model, deploy vLLM across mul
62
63
### What is Ray?
64
65
-Ray is a distributed computing framework for scaling Python programs. Multi-node vLLM deployments require Ray as the runtime engine.
65
+Ray is a distributed computing framework for scaling Python programs. Multi-node vLLM deployments can use Ray as the runtime engine.
66
67
vLLM uses Ray to manage the distributed execution of tasks across multiple nodes and control where execution happens.
68
@@ -130,6 +130,28 @@ vllm serve /path/to/the/model/in/the/container \
130
--distributed-executor-backend ray
131
```
132
133
+### Running vLLM with MultiProcessing
134
+
135
+Besides Ray, Multi-node vLLM deployments can also use `multiprocessing` as the runtime engine. Here's an example to deploy model across 2 nodes (8 GPUs per node) with `tp_size=8` and `pp_size=2`.
136
+
137
+Choose one node as the head node and run:
138
+
139
+```bash
140
+vllm serve /path/to/the/model/in/the/container \
141
+ --tensor-parallel-size 8 --pipeline-parallel-size 2 \
142
+ --nnodes 2 --node-rank 0 \
143
+ --master-addr <HEAD_NODE_IP>
144
+```
145
+
146
+On the other worker node, run:
147
+
148
+```bash
149
+vllm serve /path/to/the/model/in/the/container \
150
+ --tensor-parallel-size 8 --pipeline-parallel-size 2 \
151
+ --nnodes 2 --node-rank 1 \
152
+ --master-addr <HEAD_NODE_IP> --headless
153
+```
154
+
155
## Optimizing network communication for tensor parallelism
156
157
Efficient tensor parallelism requires fast inter-node communication, preferably through high-speed network adapters such as InfiniBand.
vllm/v1/executor/multiproc_executor.py
+1
-3
@@ -124,9 +124,7 @@ class MultiprocExecutor(Executor):
124
# Set multiprocessing envs
125
set_multiprocessing_worker_envs()
126
127
- # Multiprocessing-based executor does not support multi-node setting.
128
- # Since it only works for single node, we can use the loopback address
129
- # get_loopback_ip() for communication.
127
+ # use the loopback address get_loopback_ip() for communication.
128
distributed_init_method = get_distributed_init_method(
129
get_loopback_ip(), get_open_port()
130
)