A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 05:30 UTC
[Docs] Document the extra memory footprint overhead when using EPLB (#24537)
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
Tyler Michael Smith committed
Sep 10, 2025 at 09:09 UTC
8b83b937397ca921ca59eb14515da0ebf0d73b6d
1 file changed
+7
docs/serving/expert_parallel_deployment.md
+7
@@ -156,6 +156,13 @@ vllm serve Qwen/Qwen3-30B-A3B \
156
- **Default**: Each EP rank has `NUM_TOTAL_EXPERTS ÷ NUM_EP_RANKS` experts
157
- **With redundancy**: Each EP rank has `(NUM_TOTAL_EXPERTS + NUM_REDUNDANT_EXPERTS) ÷ NUM_EP_RANKS` experts
158
159
+### Memory Footprint Overhead
160
+
161
+EPLB uses redundant experts to that need to fit in GPU memory. This means that EPLB may not be a good fit for memory constrained environments or when KV cache space is at a premium.
162
+
163
+This overhead equals `NUM_MOE_LAYERS * BYTES_PER_EXPERT * (NUM_TOTAL_EXPERTS + NUM_REDUNDANT_EXPERTS) ÷ NUM_EP_RANKS`.
164
+For DeepSeekV3, this is approximately `2.4 GB` for one redundant expert per rank.
165
+
166
### Example Command
167
168
Single node deployment with EPLB enabled: