A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 11, 17:30 UTC
[Docs] Update EPLB docs (#30426)
Signed-off-by: mgoin <mgoin64@gmail.com>
Michael Goin committed
Dec 10, 2025 at 15:56 UTC
fcb894222f2b8a353072e1aea33b38f4403bbd7a
1 file changed
+5
-4
docs/serving/expert_parallel_deployment.md
+5
-4
@@ -40,10 +40,12 @@ EP_SIZE = TP_SIZE × DP_SIZE
40
41
Where:
42
43
-- `TP_SIZE`: Tensor parallel size (always 1 for now)
43
+- `TP_SIZE`: Tensor parallel size
44
- `DP_SIZE`: Data parallel size
45
- `EP_SIZE`: Expert parallel size (computed automatically)
46
47
+When EP is enabled, MoE layers use expert parallelism instead of tensor parallelism, while attention layers continue to use tensor parallelism if `TP_SIZE > 1`.
48
+
49
### Example Command
50
51
The following command serves a `DeepSeek-V3-0324` model with 1-way tensor parallel, 8-way (attention) data parallel, and 8-way expert parallel. The attention weights are replicated across all GPUs, while the expert weights are split across GPUs. It will work on a H200 (or H20) node with 8 GPUs. For H100, you can try to serve a smaller model or refer to the multi-node deployment section.
@@ -119,9 +121,6 @@ While MoE models are typically trained so that each expert receives a similar nu
121
122
Enable EPLB with the `--enable-eplb` flag.
123
122
-!!! note "Model Support"
123
- Currently only DeepSeek V3 architecture is supported.
124
-
124
When enabled, vLLM collects load statistics with every forward pass and periodically rebalances expert distribution.
125
126
### EPLB Parameters
@@ -134,6 +133,8 @@ Configure EPLB with the `--eplb-config` argument, which accepts a JSON string. T
133
| `step_interval`| Frequency of rebalancing (every N engine steps) | 3000 |
134
| `log_balancedness` | Log balancedness metrics (avg tokens per expert ÷ max tokens per expert) | `false` |
135
| `num_redundant_experts` | Additional global experts per EP rank beyond equal distribution | `0` |
136
+| `use_async` | Use non-blocking EPLB for reduced latency overhead | `false` |
137
+| `policy` | The policy type for expert parallel load balancing | `"default"` |
138
139
For example:
140