A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 13:30 UTC
[Doc] Split dummy_processor_inputs() in Multimodal Docs (#16915)
Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
Alex Brooks committed
Apr 21, 2025 at 05:10 UTC
b34f33438a9db25319b132720eeef5cea2fc6c69
2 files changed
+35
-29
docs/source/contributing/model/multimodal.md
+34
-28
@@ -128,11 +128,9 @@ HF processing as well as memory profiling.
128
129
### For memory profiling
130
131
-Override the abstract method {meth}`~vllm.multimodal.profiling.BaseDummyInputsBuilder.get_dummy_processor_inputs`
132
-to construct dummy inputs for memory profiling. This dummy input should result in the worst-case memory usage of
133
-the model so that vLLM can reserve the correct amount of memory for it.
131
+Override the abstract methods {meth}`~vllm.multimodal.profiling.BaseDummyInputsBuilder.get_dummy_text` and {meth}`~vllm.multimodal.profiling.BaseDummyInputsBuilder.get_dummy_mm_data` to construct dummy inputs for memory profiling. These dummy inputs should result in the worst-case memory usage of the model so that vLLM can reserve the correct amount of memory for it.
132
135
-Assuming that the memory usage increases with the number of tokens, the dummy input can be constructed to maximize the number of output embeddings, which is the same number as placeholder feature tokens.
133
+Assuming that the memory usage increases with the number of tokens, the dummy inputs can be constructed to maximize the number of output embeddings, which is the same number as placeholder feature tokens.
134
135
::::{tab-set}
136
:::{tab-item} Basic example: LLaVA
@@ -244,38 +242,45 @@ def get_num_image_tokens(
242
```
243
244
Notice that the number of image tokens doesn't depend on the image width and height.
247
-We can simply use a dummy `image_size`:
245
+We can simply use a dummy `image_size` to calculate the multimodal profiling data:
246
247
```python
248
+# NOTE: In actuality, this is usually implemented as part of the
249
+# model's subclass of `BaseProcessingInfo`, but we show it as is
250
+# here for simplicity.
251
def get_image_size_with_most_features(self) -> ImageSize:
252
hf_config = self.get_hf_config()
253
width = height = hf_config.image_size
254
return ImageSize(width=width, height=height)
255
255
-def get_dummy_processor_inputs(
256
+def get_dummy_mm_data(
257
self,
258
seq_len: int,
259
mm_counts: Mapping[str, int],
259
-) -> ProcessorInputs:
260
+) -> MultiModalDataDict:
261
num_images = mm_counts.get("image", 0)
262
262
- processor = self.info.get_hf_processor()
263
- image_token = processor.image_token
264
-
265
- hf_config = self.get_hf_config()
266
- target_width, target_height = self.info.get_image_size_with_most_features()
263
+ target_width, target_height = \
264
+ self.info.get_image_size_with_most_features()
265
268
- mm_data = {
266
+ return {
267
"image":
268
self._get_dummy_images(width=target_width,
269
height=target_height,
270
num_images=num_images)
271
}
272
+```
273
275
- return ProcessorInputs(
276
- prompt_text=image_token * num_images,
277
- mm_data=mm_data,
278
- )
274
+For the text, we simply expand the multimodal image token from the model config to match the desired number of images.
275
+
276
+```python
277
+def get_dummy_text(self, mm_counts: Mapping[str, int]) -> str:
278
+ num_images = mm_counts.get("image", 0)
279
+
280
+ processor = self.info.get_hf_processor()
281
+ image_token = processor.image_token
282
+
283
+ return image_token * num_images
284
```
285
286
:::
@@ -412,29 +417,30 @@ def get_image_size_with_most_features(self) -> ImageSize:
417
418
Fuyu does not expect image placeholders in the inputs to HF processor, so
419
the dummy prompt text is empty regardless of the number of images.
415
-Otherwise, the logic of this method is very similar to LLaVA:
420
421
```python
418
-def get_dummy_processor_inputs(
422
+def get_dummy_text(self, mm_counts: Mapping[str, int]) -> str:
423
+ return ""
424
+```
425
+
426
+For the multimodal image profiling data, the logic is very similar to LLaVA:
427
+
428
+```python
429
+def get_dummy_mm_data(
430
self,
431
seq_len: int,
432
mm_counts: Mapping[str, int],
422
-) -> ProcessorInputs:
433
+) -> MultiModalDataDict:
434
target_width, target_height = \
435
self.info.get_image_size_with_most_features()
436
num_images = mm_counts.get("image", 0)
437
427
- mm_data = {
438
+ return {
439
"image":
440
self._get_dummy_images(width=target_width,
430
- height=target_height,
431
- num_images=num_images)
441
+ height=target_height,
442
+ num_images=num_images)
443
}
433
-
434
- return ProcessorInputs(
435
- prompt_text="",
436
- mm_data=mm_data,
437
- )
444
```
445
446
:::
docs/source/design/mm_processing.md
+1
-1
@@ -47,7 +47,7 @@ Moreover, since the tokenized text has not passed through the HF processor, we h
47
48
### Dummy text
49
50
-We work around the first issue by requiring each model to define how to generate dummy text based on the number of multi-modal inputs, via {meth}`~vllm.multimodal.profiling.BaseDummyInputsBuilder.get_dummy_processor_inputs`. This lets us generate dummy text corresponding to the multi-modal inputs and input them together to obtain the processed multi-modal data.
50
+We work around the first issue by requiring each model to define how to generate dummy text based on the number of multi-modal inputs, via {meth}`~vllm.multimodal.profiling.BaseDummyInputsBuilder.get_dummy_text`. This lets us generate dummy text corresponding to the multi-modal inputs and input them together to obtain the processed multi-modal data.
51
52
(mm-automatic-prompt-updating)=
53