A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 02:15 UTC
[doc] Add more details for Ray-based DP (#20948)
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
Rui Qiao committed
Jul 15, 2025 at 05:37 UTC
d91278181d89686b73b2ec88c2db4d55c6c506cb
1 file changed
+10
-2
docs/serving/data_parallel_deployment.md
+10
-2
@@ -57,12 +57,20 @@ vllm serve $MODEL --headless --data-parallel-size 4 --data-parallel-size-local 4
57
--data-parallel-address 10.99.48.128 --data-parallel-rpc-port 13345
58
```
59
60
-This DP mode can also be used with Ray, in which case only a single launch command is needed irrespective of the number of nodes:
60
+This DP mode can also be used with Ray by specifying `--data-parallel-backend=ray`:
61
62
```bash
63
-vllm serve $MODEL --data-parallel-size 16 --tensor-parallel-size 2 --data-parallel-backend=ray
63
+vllm serve $MODEL --data-parallel-size 4 --data-parallel-size-local 2 \
64
+ --data-parallel-backend=ray
65
```
66
67
+There are several notable differences when using Ray:
68
+
69
+- A single launch command (on any node) is needed to start all local and remote DP ranks, therefore it is more convenient compared to launching on each node
70
+- There is no need to specify `--data-parallel-address`, and the node where the command is run is used as `--data-parallel-address`
71
+- There is no need to specify `--data-parallel-rpc-port`
72
+- Remote DP ranks will be allocated based on node resources of the Ray cluster
73
+
74
Currently, the internal DP load balancing is done within the API server process(es) and is based on the running and waiting queues in each of the engines. This could be made more sophisticated in future by incorporating KV cache aware logic.
75
76
When deploying large DP sizes using this method, the API server process can become a bottleneck. In this case, the orthogonal `--api-server-count` command line option can be used to scale this out (for example `--api-server-count=4`). This is transparent to users - a single HTTP endpoint / port is still exposed. Note that this API server scale-out is "internal" and still confined to the "head" node.