A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 05:30 UTC
[Doc] Add caution for API server scale-out (#23550)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Cyrus Leung committed
Aug 25, 2025 at 21:14 UTC
e269be2ba2c52ced7581b8499ae19f21383f3c56
1 file changed
+7
docs/configuration/optimization.md
+7
@@ -196,6 +196,13 @@ vllm serve Qwen/Qwen2.5-VL-3B-Instruct --api-server-count 4 -dp 2
196
!!! note
197
API server scale-out is only available for online inference.
198
199
+!!! warning
200
+ By default, 8 CPU threads are used in each API server to load media items (e.g. images)
201
+ from request data.
202
+
203
+ If you apply API server scale-out, consider adjusting `VLLM_MEDIA_LOADING_THREAD_COUNT`
204
+ to avoid CPU resource exhaustion.
205
+
206
!!! note
207
[Multi-modal processor cache](#processor-cache) is disabled when API server scale-out is enabled
208
because it requires a one-to-one correspondance between API and engine core processes.