A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 20:45 UTC
[Docs] Update PP docs (#6598)
Murali Andoorveedu committed
Jul 19, 2024 at 19:38 UTC
45ceb85a0c9032ea8bfa3d2526e3d7c1a09a48f0
1 file changed
+2
-3
docs/source/serving/distributed_serving.rst
+2
-3
@@ -44,11 +44,10 @@ You can also additionally specify :code:`--pipeline-parallel-size` to enable pip
44
45
$ vllm serve gpt2 \
46
$ --tensor-parallel-size 4 \
47
- $ --pipeline-parallel-size 2 \
48
- $ --distributed-executor-backend ray
47
+ $ --pipeline-parallel-size 2
48
49
.. note::
51
- Pipeline parallel is a beta feature. It is only supported for online serving and the ray backend for now, as well as LLaMa and GPT2 style models.
50
+ Pipeline parallel is a beta feature. It is only supported for online serving as well as LLaMa, GPT2, and Mixtral style models.
51
52
To scale vLLM beyond a single machine, install and start a `Ray runtime <https://docs.ray.io/en/latest/ray-core/starting-ray.html>`_ via CLI before running vLLM:
53