@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / c246212952
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 23:15 UTC

[doc][faq] add warning to download models for every nodes (#5783)

youkaichao committed Jun 24, 2024 at 00:37 UTC c2462129521a64b62ace77b28641d2e3bec5831c
1 file changed +4 -1
docs/source/serving/distributed_serving.rst
+4 -1
@@ -35,4 +35,7 @@ To scale vLLM beyond a single machine, install and start a `Ray runtime <https:/
35 $ # On worker nodes
36 $ ray start --address=<ray-head-address>
37
38 -After that, you can run inference and serving on multiple machines by launching the vLLM process on the head node by setting :code:`tensor_parallel_size` to the number of GPUs to be the total number of GPUs across all machines.
\ No newline at end of file
38 +After that, you can run inference and serving on multiple machines by launching the vLLM process on the head node by setting :code:`tensor_parallel_size` to the number of GPUs to be the total number of GPUs across all machines.
39 +
40 +.. warning::
41 + Please make sure you downloaded the model to all the nodes, or the model is downloaded to some distributed file system that is accessible by all nodes.