A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 20:45 UTC
[doc][distributed] fix doc argument order (#6691)
youkaichao committed
Jul 23, 2024 at 08:55 UTC
71950af726faa12bef0070a5d5d0305c503d7c35
1 file changed
+4
-4
docs/source/serving/distributed_serving.rst
+4
-4
@@ -66,8 +66,8 @@ Pick a node as the head node, and run the following command:
66
$ bash run_cluster.sh \
67
$ vllm/vllm-openai \
68
$ ip_of_head_node \
69
- $ /path/to/the/huggingface/home/in/this/node \
70
- $ --head
69
+ $ --head \
70
+ $ /path/to/the/huggingface/home/in/this/node
71
72
On the rest of the worker nodes, run the following command:
73
@@ -76,8 +76,8 @@ On the rest of the worker nodes, run the following command:
76
$ bash run_cluster.sh \
77
$ vllm/vllm-openai \
78
$ ip_of_head_node \
79
- $ /path/to/the/huggingface/home/in/this/node \
80
- $ --worker
79
+ $ --worker \
80
+ $ /path/to/the/huggingface/home/in/this/node
81
82
Then you get a ray cluster of containers. Note that you need to keep the shells running these commands alive to hold the cluster. Any shell disconnect will terminate the cluster.
83