A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 05:30 UTC
[Doc] Fix batch-level DP example (#23325)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Co-authored-by: youkaichao <youkaichao@gmail.com>
Cyrus Leung committed
Aug 21, 2025 at 21:16 UTC
5cc54f7c5bf2516b909fadb0ad09b68e65c42812
1 file changed
+6
-5
docs/configuration/optimization.md
+6
-5
@@ -153,13 +153,14 @@ from vllm import LLM
153
154
llm = LLM(
155
model="Qwen/Qwen2.5-VL-72B-Instruct",
156
- # Create two EngineCore instances, one per DP rank
157
- data_parallel_size=2,
158
- # Within each EngineCore instance:
159
- # The vision encoder uses TP=4 (not DP=2) to shard the input data
160
- # The language decoder uses TP=4 to shard the weights as usual
156
tensor_parallel_size=4,
157
+ # When mm_encoder_tp_mode="data",
158
+ # the vision encoder uses TP=4 (not DP=1) to shard the input data,
159
+ # so the TP size becomes the effective DP size.
160
+ # Note that this is independent of the DP size for language decoder which is used in expert parallel setting.
161
mm_encoder_tp_mode="data",
162
+ # The language decoder uses TP=4 to shard the weights regardless
163
+ # of the setting of mm_encoder_tp_mode
164
)
165
```
166