A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 05:30 UTC
[MM][Doc] Add documentation for configurable mm profiling (#26200)
Signed-off-by: wwl2755 <wangwenlong2755@gmail.com>
Wenlong Wang committed
Oct 8, 2025 at 23:21 UTC
43ab8cfaa56e12cb6fa6bf9d70b84e911b8c09c6
1 file changed
+40
docs/configuration/conserving_memory.md
+40
@@ -122,6 +122,46 @@ llm = LLM(model="google/gemma-3-27b-it",
122
limit_mm_per_prompt={"image": 0})
123
```
124
125
+### Configurable options
126
+
127
+`limit_mm_per_prompt` also accepts configurable options per modality. In the configurable form, you still specify `count`, and you may optionally provide size hints that control how vLLM profiles and reserves memory for your multi‑modal inputs. This helps you tune memory for the actual media you expect, instead of the model’s absolute maxima.
128
+
129
+Configurable options by modality:
130
+
131
+- `image`: `{"count": int, "width": int, "height": int}`
132
+- `video`: `{"count": int, "num_frames": int, "width": int, "height": int}`
133
+- `audio`: `{"count": int, "length": int}`
134
+
135
+Details could be found in [`ImageDummyOptions`][vllm.config.multimodal.ImageDummyOptions], [`VideoDummyOptions`][vllm.config.multimodal.VideoDummyOptions], and [`AudioDummyOptions`][vllm.config.multimodal.AudioDummyOptions].
136
+
137
+Examples:
138
+
139
+```python
140
+from vllm import LLM
141
+
142
+# Up to 5 images per prompt, profile with 512x512.
143
+# Up to 1 video per prompt, profile with 32 frames at 640x640.
144
+llm = LLM(
145
+ model="Qwen/Qwen2.5-VL-3B-Instruct",
146
+ limit_mm_per_prompt={
147
+ "image": {"count": 5, "width": 512, "height": 512},
148
+ "video": {"count": 1, "num_frames": 32, "width": 640, "height": 640},
149
+ },
150
+)
151
+```
152
+
153
+For backward compatibility, passing an integer works as before and is interpreted as `{"count": <int>}`. For example:
154
+
155
+- `limit_mm_per_prompt={"image": 5}` is equivalent to `limit_mm_per_prompt={"image": {"count": 5}}`
156
+- You can mix formats: `limit_mm_per_prompt={"image": 5, "video": {"count": 1, "num_frames": 32, "width": 640, "height": 640}}`
157
+
158
+!!! note
159
+ - The size hints affect memory profiling only. They shape the dummy inputs used to compute reserved activation sizes. They do not change how inputs are actually processed at inference time.
160
+ - If a hint exceeds what the model can accept, vLLM clamps it to the model's effective maximum and may log a warning.
161
+
162
+!!! warning
163
+ These size hints currently only affect activation memory profiling. Encoder cache size is determined by the actual inputs at runtime and is not limited by these hints.
164
+
165
## Multi-modal processor arguments
166
167
For certain models, you can adjust the multi-modal processor arguments to