@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 320 lines 16.7 KB
Rendered Raw
1 # Expert Parallel Deployment
2
3 vLLM supports Expert Parallelism (EP), which allows experts in Mixture-of-Experts (MoE) models to be deployed on separate GPUs, increasing locality, efficiency, and throughput overall.
4
5 EP is typically coupled with Data Parallelism (DP). While DP can be used independently of EP, EP is more efficient when used in conjunction with DP. You can read more about data parallelism [here](data_parallel_deployment.md).
6
7 ## Prerequisites
8
9 Before using EP, you need to install the necessary dependencies. We are actively working on making this easier in the future:
10
11 1. **Install DeepEP**: Set up host environment following vLLM's guide for EP kernels [here](../../tools/ep_kernels).
12 2. **Install DeepGEMM library**: Follow the [official instructions](https://github.com/deepseek-ai/DeepGEMM#installation).
13 3. **For disaggregated serving**: Install `gdrcopy` by running the [`install_gdrcopy.sh`](../../tools/install_gdrcopy.sh) script (e.g., `install_gdrcopy.sh "${GDRCOPY_OS_VERSION}" "12.8" "x64"`). You can find available OS versions [here](https://developer.download.nvidia.com/compute/redist/gdrcopy/CUDA%2012.8/).
14
15 !!! note "NCCL version (CUDA 13+)"
16 The `deepep_v2` backend requires NCCL >= 2.30.4. PyTorch ships an older
17 NCCL, so you must upgrade it before building or running DeepEP. See the
18 [EP kernels guide](../../tools/ep_kernels) for instructions.
19
20 ### Backend Selection Guide
21
22 vLLM provides multiple communication backends for EP. Use `--all2all-backend` to select one:
23
24 | Backend | Use Case | Features | Best For |
25 | ------- | -------- | -------- | -------- |
26 | `allgather_reducescatter` | Default backend | Standard all2all using allgather/reducescatter primitives | General purpose, works with any EP+DP configuration |
27 | `deepep_high_throughput` | Multi-node prefill | Grouped GEMM with continuous layout, optimized for prefill | Prefill-dominated workloads, high-throughput scenarios |
28 | `deepep_low_latency` | Multi-node decode | CUDA graph support, masked layout, optimized for decode | Decode-dominated workloads, low-latency scenarios |
29 | `flashinfer_nvlink_one_sided` | MNNVL systems | FlashInfer's one-sided A2A strategy for multi-node NVLink | High-throughput workloads |
30 | `flashinfer_nvlink_two_sided` | MNNVL systems | FlashInfer's two-sided A2A strategy for multi-node NVLink | Systems with NVLink across nodes |
31
32 ## Single Node Deployment
33
34 ### Configuration
35
36 Enable EP by setting the `--enable-expert-parallel` flag. The EP size is automatically calculated as:
37
38 ```text
39 EP_SIZE = TP_SIZE × DP_SIZE
40 ```
41
42 Where:
43
44 - `TP_SIZE`: Tensor parallel size
45 - `DP_SIZE`: Data parallel size
46 - `EP_SIZE`: Expert parallel size (computed automatically)
47
48 ### Layer Behavior with EP Enabled
49
50 When EP is enabled, different layers in MoE models behave differently:
51
52 | Layer Type | Behavior | Parallelism Used |
53 | ---------- | -------- | ---------------- |
54 | **Expert (MoE) Layers** | Sharded across all EP ranks | Expert Parallel (EP) of size `TP × DP` |
55 | **Attention Layers** | Behavior depends on TP size | See below |
56
57 **Attention layer parallelism:**
58
59 - **When `TP = 1`**: Attention weights are **replicated** across all DP ranks (data parallelism)
60 - **When `TP > 1`**: Attention weights are **sharded** using tensor parallelism across TP ranks within each DP group
61
62 For example, with `TP=2, DP=4` (8 GPUs total):
63
64 - Expert layers form an EP group of size 8, with experts distributed across all GPUs
65 - Attention layers use TP=2 within each of the 4 DP groups
66
67 !!! note "Key Difference from Data Parallel Deployment"
68 Without `--enable-expert-parallel`, MoE layers would use tensor parallelism (forming a TP group of size `TP × DP`), similar to dense models. With EP enabled, expert layers switch to expert parallelism, which can provide better efficiency and locality for MoE models.
69
70 ### Example Command
71
72 The following command serves a `DeepSeek-V3-0324` model with 1-way tensor parallel, 8-way (attention) data parallel, and 8-way expert parallel. The attention weights are replicated across all GPUs, while the expert weights are split across GPUs. It will work on a H200 (or H20) node with 8 GPUs. For H100, you can try to serve a smaller model or refer to the multi-node deployment section.
73
74 ```bash
75 # Single node EP deployment
76 vllm serve deepseek-ai/DeepSeek-V3-0324 \
77 --tensor-parallel-size 1 \ # Tensor parallelism across 1 GPU
78 --data-parallel-size 8 \ # Data parallelism across 8 processes
79 --enable-expert-parallel # Enable expert parallelism
80 ```
81
82 ## Multi-Node Deployment
83
84 For multi-node deployment, use the DeepEP communication kernel with one of two modes (see [Backend Selection Guide](#backend-selection-guide) above).
85
86 ### Deployment Steps
87
88 1. **Run one command per node** - Each node requires its own launch command
89 2. **Configure networking** - Ensure proper IP addresses and port configurations
90 3. **Set node roles** - First node handles requests, additional nodes run in headless mode
91
92 ### Example: 2-Node Deployment
93
94 The following example deploys `DeepSeek-V3-0324` across 2 nodes using `deepep_low_latency` mode:
95
96 ```bash
97 # Node 1 (Primary - handles incoming requests)
98 vllm serve deepseek-ai/DeepSeek-V3-0324 \
99 --all2all-backend deepep_low_latency \
100 --tensor-parallel-size 1 \ # TP size per node
101 --enable-expert-parallel \ # Enable EP
102 --data-parallel-size 16 \ # Total DP size across all nodes
103 --data-parallel-size-local 8 \ # Local DP size on this node (8 GPUs per node)
104 --data-parallel-address 192.168.1.100 \ # Replace with actual IP of Node 1
105 --data-parallel-rpc-port 13345 \ # RPC communication port, can be any port as long as reachable by all nodes
106 --api-server-count=8 # Number of API servers for load handling (scaling this out to # local ranks is recommended)
107
108 # Node 2 (Secondary - headless mode, no API server)
109 vllm serve deepseek-ai/DeepSeek-V3-0324 \
110 --all2all-backend deepep_low_latency \
111 --tensor-parallel-size 1 \ # TP size per node
112 --enable-expert-parallel \ # Enable EP
113 --data-parallel-size 16 \ # Total DP size across all nodes
114 --data-parallel-size-local 8 \ # Local DP size on this node
115 --data-parallel-start-rank 8 \ # Starting rank offset for this node
116 --data-parallel-address 192.168.1.100 \ # IP of primary node (Node 1)
117 --data-parallel-rpc-port 13345 \ # Same RPC port as primary
118 --headless # No API server, worker only
119 ```
120
121 ### Key Configuration Notes
122
123 - **Headless mode**: Secondary nodes run with `--headless` flag, meaning all client requests are handled by the primary node
124 - **Rank calculation**: `--data-parallel-start-rank` should equal the cumulative local DP size of previous nodes
125 - **Load scaling**: Adjust `--api-server-count` on the primary node to handle higher request loads
126
127 ### Network Configuration
128
129 !!! important "InfiniBand Clusters"
130 On InfiniBand networked clusters, set this environment variable to prevent initialization hangs:
131 ```bash
132 export GLOO_SOCKET_IFNAME=eth0
133 ```
134 This ensures torch distributed group discovery uses Ethernet instead of InfiniBand for initial setup.
135
136 ## Expert Parallel Load Balancer (EPLB)
137
138 While MoE models are typically trained so that each expert receives a similar number of tokens, in practice the distribution of tokens across experts can be highly skewed. vLLM provides an Expert Parallel Load Balancer (EPLB) to redistribute expert mappings across EP ranks, evening the load across experts.
139
140 ### Configuration
141
142 Enable EPLB with the `--enable-eplb` flag.
143
144 When enabled, vLLM collects load statistics with every forward pass and periodically rebalances expert distribution.
145
146 ### EPLB Parameters
147
148 Configure EPLB with the `--eplb-config` argument, which accepts a JSON string. The available keys and their descriptions are:
149
150 | Parameter | Description | Default |
151 | --------- | ----------- | ------- |
152 | `window_size` | Number of engine steps to track for rebalancing decisions | 1000 |
153 | `step_interval` | Frequency of rebalancing (every N engine steps) | 3000 |
154 | `log_balancedness` | Log balancedness metrics (avg tokens per expert ÷ max tokens per expert) | `false` |
155 | `num_redundant_experts` | Additional global experts per EP rank beyond equal distribution | `0` |
156 | `use_async` | Use non-blocking EPLB for reduced latency overhead | `true` |
157 | `policy` | The policy type for expert parallel load balancing | `"default"` |
158 | `communicator` | Backend for expert weight transfers: `"torch_nccl"`, `"torch_gloo"`, `"pynccl"`, `"nixl"`, or `null` (auto) | `null` |
159
160 For example:
161
162 ```bash
163 vllm serve Qwen/Qwen3-30B-A3B \
164 --enable-eplb \
165 --eplb-config '{"window_size":1000,"step_interval":3000,"num_redundant_experts":2,"log_balancedness":true}'
166 ```
167
168 ??? tip "Prefer individual arguments instead of JSON?"
169
170 ```bash
171 vllm serve Qwen/Qwen3-30B-A3B \
172 --enable-eplb \
173 --eplb-config.window_size 1000 \
174 --eplb-config.step_interval 3000 \
175 --eplb-config.num_redundant_experts 2 \
176 --eplb-config.log_balancedness true
177 ```
178
179 ### Expert Distribution Formula
180
181 - **Default**: Each EP rank has `NUM_TOTAL_EXPERTS ÷ NUM_EP_RANKS` experts
182 - **With redundancy**: Each EP rank has `(NUM_TOTAL_EXPERTS + NUM_REDUNDANT_EXPERTS) ÷ NUM_EP_RANKS` experts
183
184 ### Memory Footprint Overhead
185
186 EPLB uses redundant experts that need to fit in GPU memory. This means that EPLB may not be a good fit for memory constrained environments or when KV cache space is at a premium.
187
188 This overhead equals `NUM_MOE_LAYERS * BYTES_PER_EXPERT * (NUM_TOTAL_EXPERTS + NUM_REDUNDANT_EXPERTS) ÷ NUM_EP_RANKS`.
189 For DeepSeekV3, this is approximately `2.4 GB` for one redundant expert per EP rank.
190
191 ### Example Command
192
193 Single node deployment with EPLB enabled:
194
195 ```bash
196 # Single node with EPLB load balancing
197 vllm serve deepseek-ai/DeepSeek-V3-0324 \
198 --tensor-parallel-size 1 \ # Tensor parallelism
199 --data-parallel-size 8 \ # Data parallelism
200 --enable-expert-parallel \ # Enable EP
201 --enable-eplb \ # Enable load balancer
202 --eplb-config '{"window_size":1000,"step_interval":3000,"num_redundant_experts":2,"log_balancedness":true}'
203 ```
204
205 For multi-node deployment, add these EPLB flags to each node's command. We recommend setting `--eplb-config '{"num_redundant_experts":32}'` to 32 in large scale use cases so the most popular experts are always available.
206
207 ## Advanced Configuration
208
209 ### Performance Optimization
210
211 - **DeepEP kernels**: The `high_throughput` and `low_latency` kernels are optimized for disaggregated serving and may show poor performance for mixed workloads
212 - **Dual Batch Overlap**: Use `--enable-dbo` to overlap all-to-all communication with compute. See [Dual Batch Overlap](../design/dbo.md) for more details.
213 - **Async scheduling (experimental)**: Try `--async-scheduling` to overlap scheduling with model execution.
214
215 ### Troubleshooting
216
217 - **`non-zero status: 7 cannot register cq buf`**: When using Infiniband/RoCE, make sure host VM and pods show `ulimit -l` "unlimited".
218 - **`init failed for transport: IBGDA`**: The InfiniBand GDA kernel modules are missing. Run `tools/ep_kernels/configure_system_drivers.sh` on each GPU node and reboot. Also fixes error `NVSHMEM API called before NVSHMEM initialization has completed`.
219 - **NVSHMEM peer disconnect**: Usually a networking misconfiguration. If deploying via Kubernetes, verify that every pod runs with `hostNetwork: true`, `securityContext.privileged: true` to access Infiniband.
220
221 ### Benchmarking
222
223 - Use simulator flags `VLLM_MOE_ROUTING_SIMULATION_STRATEGY=uniform_random` and `VLLM_RANDOMIZE_DP_DUMMY_INPUTS=1` so token routing is balanced across EP ranks.
224
225 ## Disaggregated Serving (Prefill/Decode Split)
226
227 For production deployments requiring strict SLA guarantees for time-to-first-token and inter-token latency, disaggregated serving allows independent scaling of prefill and decode operations.
228
229 ### Architecture Overview
230
231 - **Prefill Instance**: Uses `deepep_high_throughput` backend for optimal prefill performance
232 - **Decode Instance**: Uses `deepep_low_latency` backend for minimal decode latency
233 - **KV Cache Transfer**: Connects instances via NIXL or other KV connectors
234
235 ### Setup Steps
236
237 1. **Install gdrcopy/ucx/nixl**: For maximum performance, run the [install_gdrcopy.sh](../../tools/install_gdrcopy.sh) script to install `gdrcopy` (e.g., `install_gdrcopy.sh "${GDRCOPY_OS_VERSION}" "12.8" "x64"`). You can find available OS versions [here](https://developer.download.nvidia.com/compute/redist/gdrcopy/CUDA%2012.8/). If `gdrcopy` is not installed, things will still work with a plain `pip install nixl`, just with lower performance. `nixl` and `ucx` are installed as dependencies via pip. For non-cuda platform to install nixl with non-cuda UCX build, run the [install_nixl_from_source_ubuntu.py](../../tools/install_nixl_from_source_ubuntu.py) script.
238
239 2. **Configure Both Instances**: Add this flag to both prefill and decode instances `--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}`. Noted, you may also specify one or multiple NIXL_Backend. Such as: `--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both", "kv_connector_extra_config":{"backends":["UCX", "GDS"]}}'`
240
241 3. **Client Orchestration**: Use the client-side script below to coordinate prefill/decode operations. We are actively working on routing solutions.
242
243 ### Client Orchestration Example
244
245 ```python
246 from openai import OpenAI
247 import uuid
248
249 try:
250 # 1: Set up clients for prefill and decode instances
251 openai_api_key = "EMPTY" # vLLM doesn't require a real API key
252
253 # Replace these IP addresses with your actual instance addresses
254 prefill_client = OpenAI(
255 api_key=openai_api_key,
256 base_url="http://192.168.1.100:8000/v1", # Prefill instance URL
257 )
258 decode_client = OpenAI(
259 api_key=openai_api_key,
260 base_url="http://192.168.1.101:8001/v1", # Decode instance URL
261 )
262
263 # Get model name from prefill instance
264 models = prefill_client.models.list()
265 model = models.data[0].id
266 print(f"Using model: {model}")
267
268 # 2: Prefill Phase
269 # Generate unique request ID to link prefill and decode operations
270 request_id = str(uuid.uuid4())
271 print(f"Request ID: {request_id}")
272
273 prefill_response = prefill_client.completions.create(
274 model=model,
275 # Prompt must exceed vLLM's block size (16 tokens) for PD to work
276 prompt="Write a detailed explanation of Paged Attention for Transformers works including the management of KV cache for multi-turn conversations",
277 max_tokens=1, # Force prefill-only operation
278 extra_body={
279 "kv_transfer_params": {
280 "do_remote_decode": True, # Enable remote decode
281 "do_remote_prefill": False, # This is the prefill instance
282 "remote_engine_id": None, # Will be populated by vLLM
283 "remote_block_ids": None, # Will be populated by vLLM
284 "remote_host": None, # Will be populated by vLLM
285 "remote_port": None, # Will be populated by vLLM
286 }
287 },
288 extra_headers={"X-Request-Id": request_id},
289 )
290
291 print("-" * 50)
292 print("✓ Prefill completed successfully")
293 print(f"Prefill response: {prefill_response.choices[0].text}")
294
295 # 3: Decode Phase
296 # Transfer KV cache parameters from prefill to decode instance
297 decode_response = decode_client.completions.create(
298 model=model,
299 prompt="This prompt is ignored during decode", # Original prompt not needed
300 max_tokens=150, # Generate up to 150 tokens
301 extra_body={
302 "kv_transfer_params": prefill_response.kv_transfer_params # Pass KV cache info
303 },
304 extra_headers={"X-Request-Id": request_id}, # Same request ID
305 )
306
307 print("-" * 50)
308 print("✓ Decode completed successfully")
309 print(f"Final response: {decode_response.choices[0].text}")
310
311 except Exception as e:
312 print(f"❌ Error during disaggregated serving: {e}")
313 print("Check that both prefill and decode instances are running and accessible")
314 ```
315
316 ### Benchmarking
317
318 - To simulate the decode deployment of disaggregated serving, pass `--kv-transfer-config '{"kv_connector":"DecodeBenchConnector","kv_role":"kv_both"}'` to the `vllm serve` invocation. The connector populates KV cache with random values so decode can be profiled in isolation. Add `"kv_connector_extra_config":{"startup_fill":true}` to fill the whole KV cache once at startup instead of in the decode steps that admit new requests (see the connector docstring for its limitations).
319
320 - **CUDAGraph capture**: Use `--compilation_config '{"cudagraph_mode": "FULL_DECODE_ONLY"}'` to enable CUDA graph capture for decode only and save KV cache.