A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 11, 17:30 UTC
[docs] Improve wide-EP performance + benchmarking documentation (#27933)
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
Seiji Eicher committed
Dec 10, 2025 at 17:15 UTC
b9e0951f964e1b8adfebb973c30462c0e0417c1f
3 files changed
+42
-4
docs/serving/data_parallel_deployment.md
+13
-1
@@ -24,7 +24,7 @@ There are two distinct modes supported for online deployments - self-contained w
24
25
vLLM supports "self-contained" data parallel deployments that expose a single API endpoint.
26
27
-It can be configured by simply including e.g. `--data-parallel-size=4` in the vllm serve command line arguments. This will require 4 GPUs. It can be combined with tensor parallel, for example `--data-parallel-size=4 --tensor-parallel-size=2`, which would require 8 GPUs.
27
+It can be configured by simply including e.g. `--data-parallel-size=4` in the vllm serve command line arguments. This will require 4 GPUs. It can be combined with tensor parallel, for example `--data-parallel-size=4 --tensor-parallel-size=2`, which would require 8 GPUs. When sizing DP deployments, remember that `--max-num-seqs` applies per DP rank.
28
29
Running a single data parallel deployment across multiple nodes requires a different `vllm serve` to be run on each node, specifying which DP ranks should run on that node. In this case, there will still be a single HTTP entrypoint - the API server(s) will run only on one node, but it doesn't necessarily need to be co-located with the DP ranks.
30
@@ -80,6 +80,18 @@ When deploying large DP sizes using this method, the API server process can beco
80
81
</figure>
82
83
+## Hybrid Load Balancing
84
+
85
+Hybrid load balancing sits between the internal and external approaches. Each node runs its own API server(s) that only queue requests to the data-parallel engines colocated on that node. An upstream load balancer (for example, an ingress controller or traffic router) spreads user requests across those per-node endpoints.
86
+
87
+Enable this mode with `--data-parallel-hybrid-lb` while still launching every node with the global data-parallel size. The key differences from internal load balancing are:
88
+
89
+- You must provide `--data-parallel-size-local` and `--data-parallel-start-rank` so each node knows which ranks it owns.
90
+- Not compatible with `--headless` since every node exposes an API endpoint.
91
+- Scale `--api-server-count` per node based on the number of local ranks
92
+
93
+In this configuration, each node keeps scheduling decisions local, which reduces cross-node traffic and avoids single node bottlenecks at larger DP sizes.
94
+
95
## External Load Balancing
96
97
For larger scale deployments especially, it can make sense to handle the orchestration and load balancing of data parallel ranks externally.
docs/serving/expert_parallel_deployment.md
+27
-1
@@ -83,7 +83,7 @@ vllm serve deepseek-ai/DeepSeek-V3-0324 \
83
--data-parallel-size-local 8 \ # Local DP size on this node (8 GPUs per node)
84
--data-parallel-address 192.168.1.100 \ # Replace with actual IP of Node 1
85
--data-parallel-rpc-port 13345 \ # RPC communication port, can be any port as long as reachable by all nodes
86
- --api-server-count=8 # Number of API servers for load handling (scaling this out to total ranks are recommended)
86
+ --api-server-count=8 # Number of API servers for load handling (scaling this out to # local ranks is recommended)
87
88
# Node 2 (Secondary - headless mode, no API server)
89
vllm serve deepseek-ai/DeepSeek-V3-0324 \
@@ -184,6 +184,26 @@ vllm serve deepseek-ai/DeepSeek-V3-0324 \
184
185
For multi-node deployment, add these EPLB flags to each node's command. We recommend setting `--eplb-config '{"num_redundant_experts":32}'` to 32 in large scale use cases so the most popular experts are always available.
186
187
+## Advanced Configuration
188
+
189
+### Performance Optimization
190
+
191
+- **DeepEP kernels**: The `high_throughput` and `low_latency` kernels are optimized for disaggregated serving and may show poor performance for mixed workloads
192
+- **Dual Batch Overlap**: Use `--enable-dbo` to overlap all-to-all communication with compute. See [Dual Batch Overlap](../design/dbo.md) for more details.
193
+- **Async scheduling (experimental)**: Try `--async-scheduling` to overlap scheduling with model execution.
194
+
195
+### Troubleshooting
196
+
197
+- **`non-zero status: 7 cannot register cq buf`**: When using Infiniband/RoCE, make sure host VM and pods show `ulimit -l` "unlimited".
198
+- **`init failed for transport: IBGDA`**: The InfiniBand GDA kernel modules are missing. Run `tools/ep_kernels/configure_system_drivers.sh` on each GPU node and reboot. Also fixes error `NVSHMEM API called before NVSHMEM initialization has completed`.
199
+- **NVSHMEM peer disconnect**: Usually a networking misconfiguration. If deploying via Kubernetes, verify that every pod runs with `hostNetwork: true`, `securityContext.privileged: true` to access Infiniband.
200
+
201
+### Benchmarking
202
+
203
+- Use simulator flags `VLLM_MOE_ROUTING_SIMULATION_STRATEGY=uniform_random` and `VLLM_RANDOMIZE_DP_DUMMY_INPUTS=1` so token routing is balanced across EP ranks.
204
+
205
+- Increasing `VLLM_MOE_DP_CHUNK_SIZE` may increase throughput by increasing the maximum batch size for inter-rank token transfers. This may cause DeepEP to throw `assert self.nvshmem_qp_depth >= (num_max_dispatch_tokens_per_rank + 1) * 2`, which can be fixed by increasing environment variable `NVSHMEM_QP_DEPTH`.
206
+
207
## Disaggregated Serving (Prefill/Decode Split)
208
209
For production deployments requiring strict SLA guarantees for time-to-first-token and inter-token latency, disaggregated serving allows independent scaling of prefill and decode operations.
@@ -274,3 +294,9 @@ except Exception as e:
294
print(f"❌ Error during disaggregated serving: {e}")
295
print("Check that both prefill and decode instances are running and accessible")
296
```
297
+
298
+### Benchmarking
299
+
300
+- To simulate the decode deployment of disaggregated serving, pass `--kv-transfer-config '{"kv_connector":"DecodeBenchConnector","kv_role":"kv_both"}'` to the `vllm serve` invocation. The connector populates KV cache with random values so decode can be profiled in isolation.
301
+
302
+- **CUDAGraph capture**: Use `--compilation_config '{"cudagraph_mode": "FULL_DECODE_ONLY"}'` to enable CUDA graph capture for decode only and save KV cache.
tools/ep_kernels/README.md
+2
-2
@@ -7,7 +7,7 @@ Here we break down the requirements in 2 steps:
7
1. Build and install the Python libraries (both [pplx-kernels](https://github.com/ppl-ai/pplx-kernels) and [DeepEP](https://github.com/deepseek-ai/DeepEP)), including necessary dependencies like NVSHMEM. This step does not require any privileged access. Any user can do this.
8
2. Configure NVIDIA driver to enable IBGDA. This step requires root access, and must be done on the host machine.
9
10
-2 is necessary for multi-node deployment.
10
+Step 2 is necessary for multi-node deployment.
11
12
All scripts accept a positional argument as workspace path for staging the build, defaulting to `$(pwd)/ep_kernels_workspace`.
13
@@ -23,6 +23,6 @@ TORCH_CUDA_ARCH_LIST="10.0" bash install_python_libraries.sh
23
Additional step for multi-node deployment:
24
25
```bash
26
-sudo bash configure_system_drivers.sh
26
+sudo bash configure_system_drivers.sh # update-initramfs can take several minutes
27
sudo reboot # Reboot is required to load the new driver
28
```