A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 23:15 UTC
[doc][faq] add warning to download models for every nodes (#5783)
youkaichao committed
Jun 24, 2024 at 00:37 UTC
c2462129521a64b62ace77b28641d2e3bec5831c
1 file changed
+4
-1
docs/source/serving/distributed_serving.rst
+4
-1
@@ -35,4 +35,7 @@ To scale vLLM beyond a single machine, install and start a `Ray runtime <https:/
35
$ # On worker nodes
36
$ ray start --address=<ray-head-address>
37
38
-After that, you can run inference and serving on multiple machines by launching the vLLM process on the head node by setting :code:`tensor_parallel_size` to the number of GPUs to be the total number of GPUs across all machines.
\ No newline at end of file
38
+After that, you can run inference and serving on multiple machines by launching the vLLM process on the head node by setting :code:`tensor_parallel_size` to the number of GPUs to be the total number of GPUs across all machines.
39
+
40
+.. warning::
41
+ Please make sure you downloaded the model to all the nodes, or the model is downloaded to some distributed file system that is accessible by all nodes.