@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 45ceb85a0c
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 20:45 UTC

[Docs] Update PP docs (#6598)

Murali Andoorveedu committed Jul 19, 2024 at 19:38 UTC 45ceb85a0c9032ea8bfa3d2526e3d7c1a09a48f0
1 file changed +2 -3
docs/source/serving/distributed_serving.rst
+2 -3
@@ -44,11 +44,10 @@ You can also additionally specify :code:`--pipeline-parallel-size` to enable pip
44
45 $ vllm serve gpt2 \
46 $ --tensor-parallel-size 4 \
47 - $ --pipeline-parallel-size 2 \
48 - $ --distributed-executor-backend ray
47 + $ --pipeline-parallel-size 2
48
49 .. note::
51 - Pipeline parallel is a beta feature. It is only supported for online serving and the ray backend for now, as well as LLaMa and GPT2 style models.
50 + Pipeline parallel is a beta feature. It is only supported for online serving as well as LLaMa, GPT2, and Mixtral style models.
51
52 To scale vLLM beyond a single machine, install and start a `Ray runtime <https://docs.ray.io/en/latest/ray-core/starting-ray.html>`_ via CLI before running vLLM:
53