A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 12, 16:00 UTC
docs: fixes distributed executor backend config for multi-node vllm (#29173)
Signed-off-by: Michael Act <michael.a.c.tulenan@gdplabs.id> Co-authored-by: Michael Goin <mgoin64@gmail.com>
Michael Act committed
Nov 23, 2025 at 09:58 UTC
3ed767ec064fbebbf5d8de829d390fa4a1bf0a0b
1 file changed
+4
-2
docs/serving/parallelism_scaling.md
+4
-2
@@ -118,14 +118,16 @@ The common practice is to set the tensor parallel size to the number of GPUs in
118
```bash
119
vllm serve /path/to/the/model/in/the/container \
120
--tensor-parallel-size 8 \
121
- --pipeline-parallel-size 2
121
+ --pipeline-parallel-size 2 \
122
+ --distributed-executor-backend ray
123
```
124
125
Alternatively, you can set `tensor_parallel_size` to the total number of GPUs in the cluster:
126
127
```bash
128
vllm serve /path/to/the/model/in/the/container \
128
- --tensor-parallel-size 16
129
+ --tensor-parallel-size 16 \
130
+ --distributed-executor-backend ray
131
```
132
133
## Optimizing network communication for tensor parallelism