@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 71950af726
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 20:45 UTC

[doc][distributed] fix doc argument order (#6691)

youkaichao committed Jul 23, 2024 at 08:55 UTC 71950af726faa12bef0070a5d5d0305c503d7c35
1 file changed +4 -4
docs/source/serving/distributed_serving.rst
+4 -4
@@ -66,8 +66,8 @@ Pick a node as the head node, and run the following command:
66 $ bash run_cluster.sh \
67 $ vllm/vllm-openai \
68 $ ip_of_head_node \
69 - $ /path/to/the/huggingface/home/in/this/node \
70 - $ --head
69 + $ --head \
70 + $ /path/to/the/huggingface/home/in/this/node
71
72 On the rest of the worker nodes, run the following command:
73
@@ -76,8 +76,8 @@ On the rest of the worker nodes, run the following command:
76 $ bash run_cluster.sh \
77 $ vllm/vllm-openai \
78 $ ip_of_head_node \
79 - $ /path/to/the/huggingface/home/in/this/node \
80 - $ --worker
79 + $ --worker \
80 + $ /path/to/the/huggingface/home/in/this/node
81
82 Then you get a ray cluster of containers. Note that you need to keep the shells running these commands alive to hold the cluster. Any shell disconnect will terminate the cluster.
83