@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 226 lines 11.8 KB
Rendered Raw
1 # Parallelism and Scaling
2
3 ## Distributed inference strategies for a single-model replica
4
5 To choose a distributed inference strategy for a single-model replica, use the following guidelines:
6
7 - **Single GPU (no distributed inference):** if the model fits on a single GPU, distributed inference is probably unnecessary. Run inference on that GPU.
8 - **Single-node multi-GPU using tensor parallel inference:** if the model is too large for a single GPU but fits on a single node with multiple GPUs, use *tensor parallelism*. For example, set `tensor_parallel_size=4` when using a node with 4 GPUs.
9 - **Multi-node multi-GPU using tensor parallel and pipeline parallel inference:** if the model is too large for a single node, combine *tensor parallelism* with *pipeline parallelism*. Set `tensor_parallel_size` to the number of GPUs per node and `pipeline_parallel_size` to the number of nodes. For example, set `tensor_parallel_size=8` and `pipeline_parallel_size=2` when using 2 nodes with 8 GPUs per node.
10
11 Increase the number of GPUs and nodes until there is enough GPU memory for the model. Set `tensor_parallel_size` to the number of GPUs per node and `pipeline_parallel_size` to the number of nodes.
12
13 After you provision sufficient resources to fit the model, run `vllm`. Look for log messages like:
14
15 ```text
16 INFO 07-23 13:56:04 [kv_cache_utils.py:775] GPU KV cache size: 643,232 tokens
17 INFO 07-23 13:56:04 [kv_cache_utils.py:779] Maximum concurrency for 40,960 tokens per request: 15.70x
18 ```
19
20 The `GPU KV cache size` line reports the total number of tokens that can be stored in the GPU KV cache at once. The `Maximum concurrency` line provides an estimate of how many requests can be served concurrently if each request requires the specified number of tokens (40,960 in the example above). The tokens-per-request number is taken from the model configuration's maximum sequence length, `ModelConfig.max_model_len`. If these numbers are lower than your throughput requirements, add more GPUs or nodes to your cluster.
21
22 !!! note "Edge case: uneven GPU splits"
23 If the model fits within a single node but the GPU count doesn't evenly divide the model size, enable pipeline parallelism, which splits the model along layers and supports uneven splits. In this scenario, set `tensor_parallel_size=1` and `pipeline_parallel_size` to the number of GPUs. Furthermore, if the GPUs on the node do not have NVLINK interconnect (e.g. L40S), leverage pipeline parallelism instead of tensor parallelism for higher throughput and lower communication overhead.
24
25 ### Distributed serving of *Mixture of Experts* (*MoE*) models
26
27 It's often advantageous to exploit the inherent parallelism of experts by using a separate parallelism strategy for the expert layers. vLLM supports large-scale deployment combining Data Parallel attention with Expert or Tensor Parallel MoE layers. For more information, see [Data Parallel Deployment](data_parallel_deployment.md).
28
29 ## Single-node deployment
30
31 vLLM supports distributed tensor-parallel and pipeline-parallel inference and serving. The implementation includes [Megatron-LM's tensor parallel algorithm](https://arxiv.org/pdf/1909.08053.pdf).
32
33 The default distributed runtimes are [Ray](https://github.com/ray-project/ray) for multi-node inference and native Python `multiprocessing` for single-node inference. You can override the defaults by setting `distributed_executor_backend` in the `LLM` class or `--distributed-executor-backend` in the API server. Use `mp` for `multiprocessing` or `ray` for Ray.
34
35 For multi-GPU inference, set `tensor_parallel_size` in the `LLM` class to the desired GPU count. For example, to run inference on 4 GPUs:
36
37 ```python
38 from vllm import LLM
39 llm = LLM("facebook/opt-13b", tensor_parallel_size=4)
40 output = llm.generate("San Francisco is a")
41 ```
42
43 For multi-GPU serving, include `--tensor-parallel-size` when starting the server. For example, to run the API server on 4 GPUs:
44
45 ```bash
46 vllm serve facebook/opt-13b \
47 --tensor-parallel-size 4
48 ```
49
50 To enable pipeline parallelism, add `--pipeline-parallel-size`. For example, to run the API server on 8 GPUs with pipeline parallelism and tensor parallelism:
51
52 ```bash
53 # Eight GPUs total
54 vllm serve gpt2 \
55 --tensor-parallel-size 4 \
56 --pipeline-parallel-size 2
57 ```
58
59 ## Multi-node deployment
60
61 If a single node lacks sufficient GPUs to hold the model, deploy vLLM across multiple nodes. Ensure that every node provides an identical execution environment, including the model path and Python packages. Using container images is recommended because they provide a convenient way to keep environments consistent and to hide host heterogeneity.
62
63 ### What is Ray?
64
65 Ray is a distributed computing framework for scaling Python programs. Multi-node vLLM deployments can use Ray as the runtime engine.
66
67 vLLM uses Ray to manage the distributed execution of tasks across multiple nodes and control where execution happens.
68
69 Ray also offers high-level APIs for large-scale [offline batch inference](https://docs.ray.io/en/latest/data/working-with-llms.html) and [online serving](https://docs.ray.io/en/latest/serve/llm) that can leverage vLLM as the engine. These APIs add production-grade fault tolerance, scaling, and distributed observability to vLLM workloads.
70
71 Ray is an optional dependency. Install it explicitly before using Ray-based execution, for example:
72
73 ```bash
74 pip install "ray[cgraph]"
75 ```
76
77 For details, see the [Ray documentation](https://docs.ray.io/en/latest/index.html).
78
79 ### Ray cluster setup with containers
80
81 The helper script [examples/ray_serving/run_cluster.sh](../../examples/ray_serving/run_cluster.sh) starts containers across nodes and initializes Ray. By default, the script runs Docker without administrative privileges, which prevents access to the GPU performance counters when profiling or tracing. To enable admin privileges, add the `--cap-add=CAP_SYS_ADMIN` flag to the Docker command.
82
83 Choose one node as the head node and run:
84
85 ```bash
86 bash run_cluster.sh \
87 vllm/vllm-openai \
88 <HEAD_NODE_IP> \
89 --head \
90 /path/to/the/huggingface/home/in/this/node \
91 -e VLLM_HOST_IP=<HEAD_NODE_IP>
92 ```
93
94 On each worker node, run:
95
96 ```bash
97 bash run_cluster.sh \
98 vllm/vllm-openai \
99 <HEAD_NODE_IP> \
100 --worker \
101 /path/to/the/huggingface/home/in/this/node \
102 -e VLLM_HOST_IP=<WORKER_NODE_IP>
103 ```
104
105 Note that `VLLM_HOST_IP` is unique for each worker. Keep the shells running these commands open; closing any shell terminates the cluster. Ensure that all nodes can communicate with each other through their IP addresses.
106
107 !!! warning "Network security"
108 For security, set `VLLM_HOST_IP` to an address on a private network segment. Traffic sent over this network is unencrypted, and the endpoints exchange data in a format that can be exploited to execute arbitrary code if an adversary gains network access. Ensure that untrusted parties cannot reach the network.
109
110 From any node, enter a container and run `ray status` and `ray list nodes` to verify that Ray finds the expected number of nodes and GPUs.
111
112 !!! tip
113 Alternatively, set up the Ray cluster using KubeRay. For more information, see [KubeRay vLLM documentation](https://docs.ray.io/en/latest/cluster/kubernetes/examples/rayserve-llm-example.html).
114
115 ### Running vLLM on a Ray cluster
116
117 !!! tip
118 If Ray is running inside containers, run the commands in the remainder of this guide *inside the containers*, not on the host. To open a shell inside a container, connect to a node and use `docker exec -it <container_name> /bin/bash`.
119
120 Once a Ray cluster is running, use vLLM as you would in a single-node setting. All resources across the Ray cluster are visible to vLLM, so a single `vllm` command on a single node is sufficient.
121
122 The common practice is to set the tensor parallel size to the number of GPUs in each node, and the pipeline parallel size to the number of nodes. For example, if you have 16 GPUs across 2 nodes (8 GPUs per node), set the tensor parallel size to 8 and the pipeline parallel size to 2:
123
124 ```bash
125 vllm serve /path/to/the/model/in/the/container \
126 --tensor-parallel-size 8 \
127 --pipeline-parallel-size 2 \
128 --distributed-executor-backend ray
129 ```
130
131 Alternatively, you can set `tensor_parallel_size` to the total number of GPUs in the cluster:
132
133 ```bash
134 vllm serve /path/to/the/model/in/the/container \
135 --tensor-parallel-size 16 \
136 --distributed-executor-backend ray
137 ```
138
139 ### Running vLLM with MultiProcessing
140
141 Besides Ray, Multi-node vLLM deployments can also use `multiprocessing` as the runtime engine. Here's an example to deploy model across 2 nodes (8 GPUs per node) with `tp_size=8` and `pp_size=2`.
142
143 Choose one node as the head node and run:
144
145 ```bash
146 vllm serve /path/to/the/model/in/the/container \
147 --tensor-parallel-size 8 --pipeline-parallel-size 2 \
148 --nnodes 2 --node-rank 0 \
149 --master-addr <HEAD_NODE_IP>
150 ```
151
152 On the other worker node, run:
153
154 ```bash
155 vllm serve /path/to/the/model/in/the/container \
156 --tensor-parallel-size 8 --pipeline-parallel-size 2 \
157 --nnodes 2 --node-rank 1 \
158 --master-addr <HEAD_NODE_IP> --headless
159 ```
160
161 ## Optimizing network communication for tensor parallelism
162
163 Efficient tensor parallelism requires fast internode communication, preferably through high-speed network adapters such as InfiniBand.
164 To set up the cluster to use InfiniBand, append additional arguments like `--privileged -e NCCL_IB_HCA=mlx5` to the
165 [examples/ray_serving/run_cluster.sh](../../examples/ray_serving/run_cluster.sh) helper script.
166 Contact your system administrator for more information about the required flags.
167
168 ## Enabling GPUDirect RDMA
169
170 GPUDirect RDMA (Remote Direct Memory Access) is an NVIDIA technology that allows network adapters to directly access GPU memory, bypassing the CPU and system memory. This direct access reduces latency and CPU overhead, which is beneficial for large data transfers between GPUs across nodes.
171
172 To enable GPUDirect RDMA with vLLM, configure the following settings:
173
174 - `IPC_LOCK` security context: add the `IPC_LOCK` capability to the container's security context to lock memory pages and prevent swapping to disk.
175 - Shared memory with `/dev/shm`: mount `/dev/shm` in the pod spec to provide shared memory for interprocess communication (IPC).
176
177 If you use Docker, set up the container as follows:
178
179 ```bash
180 docker run --gpus all \
181 --ipc=host \
182 --shm-size=16G \
183 -v /dev/shm:/dev/shm \
184 vllm/vllm-openai
185 ```
186
187 If you use Kubernetes, set up the pod spec as follows:
188
189 ```yaml
190 ...
191 spec:
192 containers:
193 - name: vllm
194 image: vllm/vllm-openai
195 securityContext:
196 capabilities:
197 add: ["IPC_LOCK"]
198 volumeMounts:
199 - mountPath: /dev/shm
200 name: dshm
201 resources:
202 limits:
203 nvidia.com/gpu: 8
204 requests:
205 nvidia.com/gpu: 8
206 volumes:
207 - name: dshm
208 emptyDir:
209 medium: Memory
210 ...
211 ```
212
213 !!! tip "Confirm GPUDirect RDMA operation"
214 To confirm your InfiniBand card is using GPUDirect RDMA, run vLLM with detailed NCCL logs: `NCCL_DEBUG=TRACE vllm serve ...`.
215
216 Then look for the NCCL version and the network used.
217
218 - If you find `[send] via NET/IB/GDRDMA` in the logs, then NCCL is using InfiniBand with GPUDirect RDMA, which *is* efficient.
219 - If you find `[send] via NET/Socket` in the logs, NCCL used a raw TCP socket, which *is not* efficient for cross-node tensor parallelism.
220
221 !!! tip "Pre-download Hugging Face models"
222 If you use Hugging Face models, downloading the model before starting vLLM is recommended. Download the model on every node to the same path, or store the model on a distributed file system accessible by all nodes. Then pass the path to the model in place of the repository ID. Otherwise, supply a Hugging Face token by appending `-e HF_TOKEN=<TOKEN>` to `run_cluster.sh`.
223
224 ## Troubleshooting distributed deployments
225
226 For information about distributed debugging, see [Troubleshooting distributed deployments](distributed_troubleshooting.md).