@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 743 lines 32 KB
Rendered Raw
1 # Multi-Modal Support
2
3 This document walks you through the steps to extend a basic model so that it accepts [multi-modal inputs](../../features/multimodal_inputs.md).
4
5 ## 1. Update the base vLLM model
6
7 It is assumed that you have already implemented the model in vLLM according to [these steps](basic.md).
8 Further update the model as follows:
9
10 - Implement [get_placeholder_str][vllm.model_executor.models.interfaces.SupportsMultiModal.get_placeholder_str] to define the placeholder string which is used to represent the multi-modal item in the text prompt. This should be consistent with the chat template of the model.
11
12 ??? code
13
14 ```python
15 class YourModelForImage2Seq(nn.Module):
16 ...
17
18 @classmethod
19 def get_placeholder_str(cls, modality: str, i: int) -> str | None:
20 if modality.startswith("image"):
21 return "<image>"
22
23 raise ValueError("Only image modality is supported")
24 ```
25
26 - Inside `__init__` method, initialize the language components of the model inside [_mark_language_model][vllm.model_executor.models.interfaces.SupportsMultiModal._mark_language_model], and the multimodal components of the model inside [_mark_tower_model][vllm.model_executor.models.interfaces.SupportsMultiModal._mark_tower_model], e.g.:
27
28 ```python
29 def __init__(self, *, vllm_config: VllmConfig, prefix: str = "") -> None:
30 super().__init__()
31
32 config = vllm_config.model_config.hf_config
33
34 with self._mark_tower_model(vllm_config, "image"):
35 self.vision_encoder = ...
36 self.multi_modal_projector = ...
37
38 with self._mark_language_model(vllm_config):
39 self.language_model = init_vllm_registered_model(
40 vllm_config=vllm_config,
41 hf_config=config.text_config,
42 prefix=maybe_prefix(prefix, "language_model"),
43 )
44 ```
45
46 - Remove the embedding part from the [forward][torch.nn.Module.forward] method:
47 - Move the multi-modal embedding to [embed_multimodal][vllm.model_executor.models.interfaces.SupportsMultiModal.embed_multimodal].
48 - The text embedding and embedding merge are handled automatically by a default implementation of [embed_input_ids][vllm.model_executor.models.interfaces.SupportsMultiModal.embed_input_ids]. It does not need to be overridden in most cases.
49
50 ```diff
51 def forward(
52 self,
53 input_ids: torch.Tensor | None,
54 - pixel_values: torch.Tensor,
55 positions: torch.Tensor,
56 intermediate_tensors: IntermediateTensors | None = None,
57 inputs_embeds: torch.Tensor | None = None,
58 ) -> torch.Tensor:
59 - if inputs_embeds is None:
60 - inputs_embeds = self.get_input_embeddings()(input_ids)
61 -
62 - if pixel_values is not None:
63 - image_features = self.get_image_features(
64 - pixel_values=pixel_values,
65 - )
66 - special_image_mask = self.get_placeholder_mask(
67 - input_ids,
68 - inputs_embeds=inputs_embeds,
69 - image_features=image_features,
70 - )
71 - inputs_embeds = inputs_embeds.masked_scatter(
72 - special_image_mask,
73 - image_features,
74 - )
75
76 hidden_states = self.language_model(
77 input_ids,
78 positions,
79 intermediate_tensors,
80 inputs_embeds=inputs_embeds,
81 )
82 ...
83
84 + def embed_multimodal(
85 + self,
86 + pixel_values: torch.Tensor,
87 + ) -> MultiModalEmbeddings | None:
88 + return self.get_image_features(
89 + pixel_values=pixel_values,
90 + )
91 ```
92
93 Below we provide a boilerplate of a typical implementation pattern of [embed_multimodal][vllm.model_executor.models.interfaces.SupportsMultiModal.embed_multimodal], but feel free to adjust it to your own needs.
94
95 ```python
96 def _process_image_input(self, image_input: YourModelImageInputs) -> torch.Tensor:
97 image_features = self.vision_encoder(image_input)
98 return self.multi_modal_projector(image_features)
99
100 def embed_multimodal(
101 self,
102 **kwargs: object,
103 ) -> MultiModalEmbeddings | None:
104 # Validate the multimodal input keyword arguments
105 image_input = self._parse_and_validate_image_input(**kwargs)
106 if image_input is None:
107 return None
108
109 # Run multimodal inputs through encoder and projector
110 vision_embeddings = self._process_image_input(image_input)
111 return vision_embeddings
112 ```
113
114 !!! important
115 The returned `multimodal_embeddings` must be either a **3D [torch.Tensor][]** of shape `(num_items, feature_size, hidden_size)`, or a **list / tuple of 2D [torch.Tensor][]'s** of shape `(feature_size, hidden_size)`, so that `multimodal_embeddings[i]` retrieves the embeddings generated from the `i`-th multimodal data item (e.g, image) of the request.
116
117 !!! note
118 By default, vLLM merges the multimodal embeddings into text embeddings depending on the information of their locations defined in
119 [PlaceholderRange][vllm.multimodal.inputs.PlaceholderRange] from input processing.
120 This logic can be found at [embed_input_ids][vllm.model_executor.models.interfaces.SupportsMultiModal.embed_input_ids].
121
122 You may override this method if additional logic is required for your model when merging embeddings.
123
124 - Once the above steps are done, update the model class with the [SupportsMultiModal][vllm.model_executor.models.interfaces.SupportsMultiModal] interface.
125
126 ```diff
127 + from vllm.model_executor.models.interfaces import SupportsMultiModal
128
129 - class YourModelForImage2Seq(nn.Module):
130 + class YourModelForImage2Seq(nn.Module, SupportsMultiModal):
131 ```
132
133 !!! note
134 The model class does not have to be named `*ForCausalLM`.
135 Check out [the HuggingFace Transformers documentation](https://huggingface.co/docs/transformers/model_doc/auto#multimodal) for some examples.
136
137 ## 2. Specify processing information
138
139 Next, create a subclass of [BaseProcessingInfo][vllm.multimodal.processing.BaseProcessingInfo]
140 to provide basic information related to HF processing.
141
142 ### Maximum number of input items
143
144 You need to override the abstract method [get_supported_mm_limits][vllm.multimodal.processing.BaseProcessingInfo.get_supported_mm_limits]
145 to return the maximum number of input items for each modality supported by the model.
146
147 For example, if the model supports any number of images but only one video per prompt:
148
149 ```python
150 def get_supported_mm_limits(self) -> Mapping[str, int | None]:
151 return {"image": None, "video": 1}
152 ```
153
154 ## 3. Specify dummy inputs
155
156 Then, inherit [BaseDummyInputsBuilder][vllm.multimodal.processing.BaseDummyInputsBuilder] to construct the dummy inputs
157 that are used for memory profiling.
158
159 Override the abstract methods [get_dummy_text][vllm.multimodal.processing.BaseDummyInputsBuilder.get_dummy_text] and [get_dummy_mm_data][vllm.multimodal.processing.BaseDummyInputsBuilder.get_dummy_mm_data] to construct dummy inputs. These dummy inputs should result in the worst-case memory usage of the model so that vLLM can reserve the correct amount of memory for it.
160
161 Besides profiling, the dummy text is passed to the HF processor by models whose HF processor requires text corresponding to the multi-modal items (see [Multi-modal fields](#multi-modal-fields)). If the HF processor can process multi-modal data without any text, the dummy text is not used during HF processing.
162
163 Assuming that the memory usage increases with the number of tokens, the dummy inputs can be constructed to maximize the number of output embeddings, which is the same number as placeholder feature tokens.
164
165 === "Basic example: LLaVA"
166
167 Looking at the code of HF's `LlavaForConditionalGeneration`:
168
169 ??? code
170
171 ```python
172 # https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/llava/modeling_llava.py#L530-L544
173 n_image_tokens = (input_ids == self.config.image_token_index).sum().item()
174 n_image_features = image_features.shape[0] * image_features.shape[1]
175
176 if n_image_tokens != n_image_features:
177 raise ValueError(
178 f"Image features and image tokens do not match: tokens: {n_image_tokens}, features {n_image_features}"
179 )
180 special_image_mask = (
181 (input_ids == self.config.image_token_index)
182 .unsqueeze(-1)
183 .expand_as(inputs_embeds)
184 .to(inputs_embeds.device)
185 )
186 image_features = image_features.to(inputs_embeds.device, inputs_embeds.dtype)
187 inputs_embeds = inputs_embeds.masked_scatter(special_image_mask, image_features)
188 ```
189
190 The number of placeholder feature tokens per image is `image_features.shape[1]`.
191 `image_features` is calculated inside the `get_image_features` method:
192
193 ??? code
194
195 ```python
196 # https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/llava/modeling_llava.py#L290-L300
197 image_outputs = self.vision_tower(pixel_values, output_hidden_states=True)
198
199 selected_image_feature = image_outputs.hidden_states[vision_feature_layer]
200 if vision_feature_select_strategy == "default":
201 selected_image_feature = selected_image_feature[:, 1:]
202 elif vision_feature_select_strategy == "full":
203 selected_image_feature = selected_image_feature
204 else:
205 raise ValueError(f"Unexpected select feature strategy: {self.config.vision_feature_select_strategy}")
206 image_features = self.multi_modal_projector(selected_image_feature)
207 return image_features
208 ```
209
210 We can infer that `image_features.shape[1]` is based on `image_outputs.hidden_states.shape[1]` from the vision tower
211 (`CLIPVisionModel` for the [`llava-hf/llava-1.5-7b-hf`](https://huggingface.co/llava-hf/llava-1.5-7b-hf) model).
212 Moreover, we only need the sequence length (the second dimension of the tensor) to get `image_features.shape[1]`.
213 The sequence length is determined by the initial hidden states in `CLIPVisionTransformer` since the attention
214 mechanism doesn't change the sequence length of the output hidden states.
215
216 ```python
217 # https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/clip/modeling_clip.py#L1094-L1102
218 hidden_states = self.embeddings(pixel_values, interpolate_pos_encoding=interpolate_pos_encoding)
219 hidden_states = self.pre_layrnorm(hidden_states)
220
221 encoder_outputs = self.encoder(
222 inputs_embeds=hidden_states,
223 output_attentions=output_attentions,
224 output_hidden_states=output_hidden_states,
225 return_dict=return_dict,
226 )
227 ```
228
229 To find the sequence length, we turn to the code of `CLIPVisionEmbeddings`:
230
231 ??? code
232
233 ```python
234 # https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/clip/modeling_clip.py#L247-L257
235 target_dtype = self.patch_embedding.weight.dtype
236 patch_embeds = self.patch_embedding(pixel_values.to(dtype=target_dtype)) # shape = [*, width, grid, grid]
237 patch_embeds = patch_embeds.flatten(2).transpose(1, 2)
238
239 class_embeds = self.class_embedding.expand(batch_size, 1, -1)
240 embeddings = torch.cat([class_embeds, patch_embeds], dim=1)
241 if interpolate_pos_encoding:
242 embeddings = embeddings + self.interpolate_pos_encoding(embeddings, height, width)
243 else:
244 embeddings = embeddings + self.position_embedding(self.position_ids)
245 return embeddings
246 ```
247
248 We can infer that `embeddings.shape[1] == self.num_positions`, where
249
250 ```python
251 # https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/clip/modeling_clip.py#L195-L196
252 self.num_patches = (self.image_size // self.patch_size) ** 2
253 self.num_positions = self.num_patches + 1
254 ```
255
256 Overall, the number of placeholder feature tokens for an image can be calculated as:
257
258 ??? code
259
260 ```python
261 def get_num_image_tokens(
262 self,
263 *,
264 image_width: int,
265 image_height: int,
266 ) -> int:
267 hf_config = self.get_hf_config()
268 hf_processor = self.get_hf_processor()
269
270 image_size = hf_config.vision_config.image_size
271 patch_size = hf_config.vision_config.patch_size
272
273 num_image_tokens = (image_size // patch_size) ** 2 + 1
274 if hf_processor.vision_feature_select_strategy == "default":
275 num_image_tokens -= 1
276
277 return num_image_tokens
278 ```
279
280 Notice that the number of image tokens doesn't depend on the image width and height.
281 We can simply use a dummy `image_size` to calculate the multimodal profiling data:
282
283 ??? code
284
285 ```python
286 # NOTE: In actuality, this is usually implemented as part of the
287 # model's subclass of `BaseProcessingInfo`, but we show it as is
288 # here for simplicity.
289 def get_image_size_with_most_features(self) -> ImageSize:
290 hf_config = self.get_hf_config()
291 width = height = hf_config.image_size
292 return ImageSize(width=width, height=height)
293
294 def get_dummy_mm_data(
295 self,
296 seq_len: int,
297 mm_counts: Mapping[str, int],
298 mm_options: Mapping[str, BaseDummyOptions],
299 ) -> MultiModalDataDict:
300 num_images = mm_counts.get("image", 0)
301
302 target_width, target_height = \
303 self.info.get_image_size_with_most_features()
304
305 image_overrides = mm_options.get("image")
306
307 return {
308 "image": self._get_dummy_images(
309 width=target_width,
310 height=target_height,
311 num_images=num_images,
312 overrides=image_overrides,
313 )
314 }
315 ```
316
317 For the text, we simply expand the multimodal image token from the model config to match the desired number of images.
318
319 ```python
320 def get_dummy_text(self, mm_counts: Mapping[str, int]) -> str:
321 num_images = mm_counts.get("image", 0)
322
323 processor = self.info.get_hf_processor()
324 image_token = processor.image_token
325
326 return image_token * num_images
327 ```
328
329 === "No input placeholders: PaliGemma"
330
331 Unlike LLaVA, PaliGemma's HF processor does not expect image placeholder
332 tokens in the input prompt; the placeholder feature tokens are instead
333 inserted afterwards (see [Prompt updates](#prompt-updates)). So the dummy
334 prompt text is empty regardless of the number of images:
335
336 ```python
337 def get_dummy_text(self, mm_counts: Mapping[str, int]) -> str:
338 return ""
339 ```
340
341 PaliGemma resizes every image to a square of `vision_config.image_size`, so
342 the number of placeholder feature tokens per image is fixed at
343 `(image_size // patch_size) ** 2`. This is computed by the SigLIP vision
344 encoder that PaliGemma uses:
345
346 ??? code
347
348 ```python
349 # vllm/model_executor/models/siglip.py
350 class SiglipEncoderInfo(VisionEncoderInfo[SiglipVisionConfig]):
351 def get_num_image_tokens(
352 self,
353 *,
354 image_width: int,
355 image_height: int,
356 ) -> int:
357 return self.get_patch_grid_length() ** 2
358
359 def get_patch_grid_length(self) -> int:
360 image_size, patch_size = self.get_image_size(), self.get_patch_size()
361 return image_size // patch_size
362 ```
363
364 Since the number of image tokens doesn't depend on the input image dimensions,
365 we can simply use a dummy image of the model's expected input size for the
366 multimodal profiling data:
367
368 ??? code
369
370 ```python
371 def get_dummy_mm_data(
372 self,
373 seq_len: int,
374 mm_counts: Mapping[str, int],
375 mm_options: Mapping[str, BaseDummyOptions],
376 ) -> MultiModalDataDict:
377 hf_config = self.info.get_hf_config()
378 vision_config = hf_config.vision_config
379 max_image_size = vision_config.image_size
380
381 num_images = mm_counts.get("image", 0)
382
383 image_overrides = mm_options.get("image")
384
385 return {
386 "image": self._get_dummy_images(
387 width=max_image_size,
388 height=max_image_size,
389 num_images=num_images,
390 overrides=image_overrides,
391 )
392 }
393 ```
394
395 ## 4. Specify processing details
396
397 Afterwards, create a subclass of [BaseMultiModalProcessor][vllm.multimodal.processing.BaseMultiModalProcessor]
398 to fill in the missing details about HF processing.
399
400 !!! info
401 [Multi-Modal Data Processing](../../design/mm_processing.md)
402
403 ### Multi-modal fields
404
405 Override [_get_mm_fields_config][vllm.multimodal.processing.BaseMultiModalProcessor._get_mm_fields_config] to
406 return a schema of the tensors outputted by the HF processor that are related to the input multi-modal items.
407
408 === "Basic example: LLaVA"
409
410 The output of `CLIPImageProcessor` is a simple tensor with shape
411 `(num_images, num_channels, image_height, image_width)`:
412
413
414 ```python
415 # https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/clip/image_processing_clip.py#L339-L345
416 images = [
417 to_channel_dimension_format(image, data_format, input_channel_dim=input_data_format)
418 for image in all_images
419 ]
420
421 data = {"pixel_values": images}
422 return BatchFeature(data=data, tensor_type=return_tensors)
423 ```
424
425 So, we override [_get_mm_fields_config][vllm.multimodal.processing.BaseMultiModalProcessor._get_mm_fields_config] as follows:
426
427 ```python
428 def _get_mm_fields_config(
429 self,
430 hf_inputs: BatchFeature,
431 hf_processor_mm_kwargs: Mapping[str, object],
432 ) -> Mapping[str, MultiModalFieldConfig]:
433 return dict(
434 pixel_values=MultiModalFieldConfig.batched("image"),
435 )
436 ```
437
438 !!! note
439 Our [actual code](../../../vllm/model_executor/models/llava.py) additionally supports
440 pre-computed image embeddings, which can be passed to be model via the `image_embeds` argument.
441
442 === "With postprocessing: Mistral3"
443
444 The `pixel_values` output of Mistral3's HF processor pads every image in the
445 batch to a common size, so that they can be stacked into a single tensor.
446
447 To use [MultiModalFieldConfig.batched][vllm.multimodal.inputs.MultiModalFieldConfig.batched]
448 like in LLaVA, each image's features must be independent of the others (which
449 is also required for prefix caching to work correctly). So, we un-pad each image
450 back to its own size by overriding
451 [BaseMultiModalProcessor._postprocess_hf_mm_data][vllm.multimodal.processing.BaseMultiModalProcessor._postprocess_hf_mm_data]
452 to post-process the outputs of the HF processor:
453
454 ??? code
455
456 ```python
457 def _get_hf_mm_text(self, mm_counts: Mapping[str, int]) -> str:
458 # Mistral3Processor requires text corresponding to the images
459 return self.dummy_inputs.get_dummy_text(mm_counts)
460
461 def _postprocess_hf_mm_data(
462 self,
463 hf_data: Mapping[str, object],
464 hf_kwargs: Mapping[str, object],
465 processed_data: BatchFeature,
466 ) -> BatchFeature:
467 if not hf_data:
468 return processed_data
469
470 pixel_values = processed_data.get("pixel_values")
471 if pixel_values is not None:
472 # Avoid padding since we need the output for each image to be
473 # independent of other images for the cache to work correctly
474 image_sizes = processed_data["image_sizes"]
475 assert len(pixel_values) == len(image_sizes)
476
477 processed_data["pixel_values"] = [
478 p[:, :h, :w] for p, (h, w) in zip(pixel_values, image_sizes)
479 ]
480
481 return processed_data
482 ```
483
484 The default implementation of
485 [_apply_hf_processor_main][vllm.multimodal.processing.BaseMultiModalProcessor._apply_hf_processor_main]
486 calls the HF processor on the multi-modal data without passing any text.
487 If the HF processor instead requires text corresponding to the multi-modal items,
488 you should override
489 [_get_hf_mm_text][vllm.multimodal.processing.BaseMultiModalProcessor._get_hf_mm_text]
490 to return the dummy text from
491 [BaseDummyInputsBuilder.get_dummy_text][vllm.multimodal.processing.BaseDummyInputsBuilder.get_dummy_text]
492 like in the example above. If the HF processor expects the multi-modal data
493 under different keys than those provided by the multi-modal items,
494 or requires additional keyword arguments
495 (e.g. `sampling_rate`), you should override
496 [_get_hf_mm_inputs][vllm.multimodal.processing.BaseMultiModalProcessor._get_hf_mm_inputs].
497 If you need to modify the output of the HF processor,
498 you should override
499 [_postprocess_hf_mm_data][vllm.multimodal.processing.BaseMultiModalProcessor._postprocess_hf_mm_data].
500 Passthrough fields have already been merged into `processed_data` when this
501 hook is called.
502 If you need to change how the normalized inputs are passed to the HF
503 processor (for example, to call a method other than `__call__`, or to use
504 different kwargs to construct and invoke it), override
505 [_call_hf_processor][vllm.multimodal.processing.BaseMultiModalProcessor._call_hf_processor].
506 If the HF processor applies additional transformations to the prompt tokens
507 that are not reflected in its multi-modal outputs (e.g. prepending a BOS token),
508 you should override
509 [_postprocess_prompt][vllm.multimodal.processing.BaseMultiModalProcessor._postprocess_prompt]
510 to replicate them.
511 Only processors that require multiple HF calls or manually construct the
512 complete output should override
513 [_apply_hf_processor_main][vllm.multimodal.processing.BaseMultiModalProcessor._apply_hf_processor_main]
514 directly. Such implementations must obtain their normalized inputs through
515 [_get_hf_mm_inputs][vllm.multimodal.processing.BaseMultiModalProcessor._get_hf_mm_inputs]
516 and return through
517 [_finalize_hf_mm_data][vllm.multimodal.processing.BaseMultiModalProcessor._finalize_hf_mm_data]
518 so passthrough fields and model-specific postprocessing are applied consistently.
519
520 Since `pixel_values` is now a list with one tensor per image, we can override
521 [_get_mm_fields_config][vllm.multimodal.processing.BaseMultiModalProcessor._get_mm_fields_config] as follows:
522
523 ```python
524 def _get_mm_fields_config(
525 self,
526 hf_inputs: BatchFeature,
527 hf_processor_mm_kwargs: Mapping[str, object],
528 ) -> Mapping[str, MultiModalFieldConfig]:
529 return dict(
530 pixel_values=MultiModalFieldConfig.batched("image"),
531 image_embeds=MultiModalFieldConfig.batched("image"),
532 )
533 ```
534
535 !!! note
536 See our [actual code](../../../vllm/model_executor/models/mistral3.py) for the full implementation.
537
538 ### Prompt updates
539
540 Override [_get_prompt_updates][vllm.multimodal.processing.BaseMultiModalProcessor._get_prompt_updates] to
541 return a list of [PromptUpdate][vllm.multimodal.processing.PromptUpdate] instances.
542
543 Each [PromptUpdate][vllm.multimodal.processing.PromptUpdate] instance specifies an update operation
544 (e.g.: insertion, replacement) performed by the HF processor.
545
546 !!! note
547 The target and content of each update are token sequences. When converting
548 text to token sequences, remember to encode it with
549 `cached_encode(..., add_special_tokens=False)`. Otherwise, the updated prompt
550 may contain duplicated special tokens or fail to match the target.
551
552 === "Basic example: LLaVA"
553
554 Looking at HF's `LlavaProcessor`:
555
556 ```python
557 # https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/llava/processing_llava.py#L167-L170
558 prompt_strings = []
559 for sample in text:
560 sample = sample.replace(self.image_token, self.image_token * num_image_tokens)
561 prompt_strings.append(sample)
562 ```
563
564 It simply repeats each input `image_token` a number of times equal to the number of placeholder feature tokens (`num_image_tokens`).
565 Based on this, we override [_get_prompt_updates][vllm.multimodal.processing.BaseMultiModalProcessor._get_prompt_updates] as follows:
566
567 ??? code
568
569 ```python
570 def _get_prompt_updates(
571 self,
572 mm_items: MultiModalDataItems,
573 hf_processor_mm_kwargs: Mapping[str, object],
574 out_mm_kwargs: MultiModalKwargsItems,
575 ) -> Sequence[PromptUpdate]:
576 hf_config = self.info.get_hf_config()
577 image_token_id = hf_config.image_token_index
578
579 def get_replacement(item_idx: int):
580 images = mm_items.get_items("image", ImageProcessorItems)
581
582 image_size = images.get_image_size(item_idx)
583 num_image_tokens = self.info.get_num_image_tokens(
584 image_width=image_size.width,
585 image_height=image_size.height,
586 )
587
588 return [image_token_id] * num_image_tokens
589
590 return [
591 PromptReplacement(
592 modality="image",
593 target=[image_token_id],
594 replacement=get_replacement,
595 ),
596 ]
597 ```
598
599 === "Handling additional tokens: PaliGemma"
600
601 PaliGemma's HF processor inserts, after the prompt's leading `<bos>` token, a
602 run of image tokens followed by a second `<bos>` token that marks the start of
603 the text prompt. We start by building the run of image tokens, one per
604 placeholder feature token:
605
606 ??? code
607
608 ```python
609 def get_insertion(item_idx: int):
610 images = mm_items.get_items(
611 "image", (ImageEmbeddingItems, ImageProcessorItems)
612 )
613
614 if isinstance(images, ImageEmbeddingItems):
615 num_image_tokens = images.get_feature_size(item_idx)
616 else:
617 image_size = images.get_image_size(item_idx)
618 num_image_tokens = self.info.get_num_image_tokens(
619 image_width=image_size.width,
620 image_height=image_size.height,
621 )
622
623 image_tokens = [image_token_id] * num_image_tokens
624 ...
625 ```
626
627 The trailing `<bos>` token is an additional token that must **not** receive a
628 vision embedding. To assign the vision embeddings to only the image tokens,
629 instead of returning the token ids directly you can return an instance of
630 [PromptUpdateDetails][vllm.multimodal.processing.PromptUpdateDetails] and mark
631 the embedding tokens with `embed_token_id`:
632
633 ??? code
634
635 ```python
636 return PromptUpdateDetails.select_token_id(
637 image_tokens + [bos_token_id],
638 embed_token_id=image_token_id,
639 )
640 ```
641
642 Putting it together, we override [_get_prompt_updates][vllm.multimodal.processing.BaseMultiModalProcessor._get_prompt_updates].
643 Since these tokens are inserted (rather than replacing an existing placeholder)
644 after the prompt's leading `<bos>`, we use [PromptInsertion][vllm.multimodal.processing.PromptInsertion]
645 with a prefix target:
646
647 ??? code
648
649 ```python
650 def _get_prompt_updates(
651 self,
652 mm_items: MultiModalDataItems,
653 hf_processor_mm_kwargs: Mapping[str, object],
654 out_mm_kwargs: MultiModalKwargsItems,
655 ) -> Sequence[PromptUpdate]:
656 hf_config = self.info.get_hf_config()
657 image_token_id = hf_config.image_token_index
658
659 tokenizer = self.info.get_tokenizer()
660
661 bos_token_id = tokenizer.bos_token_id
662 assert isinstance(bos_token_id, int)
663
664 def get_insertion(item_idx: int):
665 images = mm_items.get_items(
666 "image", (ImageEmbeddingItems, ImageProcessorItems)
667 )
668
669 if isinstance(images, ImageEmbeddingItems):
670 num_image_tokens = images.get_feature_size(item_idx)
671 else:
672 image_size = images.get_image_size(item_idx)
673 num_image_tokens = self.info.get_num_image_tokens(
674 image_width=image_size.width,
675 image_height=image_size.height,
676 )
677
678 image_tokens = [image_token_id] * num_image_tokens
679
680 return PromptUpdateDetails.select_token_id(
681 image_tokens + [bos_token_id],
682 embed_token_id=image_token_id,
683 )
684
685 return [
686 PromptInsertion(
687 modality="image",
688 target=PromptIndexTargets.prefix(
689 [bos_token_id] if tokenizer.add_bos_token else []
690 ),
691 insertion=get_insertion,
692 )
693 ]
694 ```
695
696 ## 5. Register processor-related classes
697
698 After you have defined [BaseProcessingInfo][vllm.multimodal.processing.BaseProcessingInfo] (Step 2),
699 [BaseDummyInputsBuilder][vllm.multimodal.processing.BaseDummyInputsBuilder] (Step 3),
700 and [BaseMultiModalProcessor][vllm.multimodal.processing.BaseMultiModalProcessor] (Step 4),
701 decorate the model class with [MULTIMODAL_REGISTRY.register_processor][vllm.multimodal.registry.MultiModalRegistry.register_processor]
702 to register them to the multi-modal registry:
703
704 ```diff
705 from vllm.model_executor.models.interfaces import SupportsMultiModal
706 + from vllm.multimodal import MULTIMODAL_REGISTRY
707
708 + @MULTIMODAL_REGISTRY.register_processor(
709 + YourMultiModalProcessor,
710 + info=YourProcessingInfo,
711 + dummy_inputs=YourDummyInputsBuilder,
712 + )
713 class YourModelForImage2Seq(nn.Module, SupportsMultiModal):
714 ```
715
716 ## Notes
717
718 ### Inserting feature tokens without replacement
719
720 Some HF processors directly insert feature tokens without replacing anything in the original prompt. In that case, you can use [PromptInsertion][vllm.multimodal.processing.PromptInsertion] instead of [PromptReplacement][vllm.multimodal.processing.PromptReplacement] inside [_get_prompt_updates][vllm.multimodal.processing.BaseMultiModalProcessor._get_prompt_updates].
721
722 Examples:
723
724 - BLIP-2 (insert at start of prompt): [vllm/model_executor/models/blip2.py](../../../vllm/model_executor/models/blip2.py)
725 - Molmo (insert after `<|endoftext|>` token): [vllm/model_executor/models/molmo.py](../../../vllm/model_executor/models/molmo.py)
726
727 ### Handling prompt updates unrelated to multi-modal data
728
729 [_get_prompt_updates][vllm.multimodal.processing.BaseMultiModalProcessor._get_prompt_updates] assumes that each application of prompt update corresponds to one multi-modal item. If the HF processor performs additional processing on the prompt regardless of how many multi-modal items there are, you should override [_postprocess_prompt][vllm.multimodal.processing.BaseMultiModalProcessor._postprocess_prompt] so that the prompt token IDs are consistent with the result of applying the HF processor on text inputs. This is because token inputs bypass the HF processor according to [our design](../../design/mm_processing.md).
730
731 Examples:
732
733 - Molmo2 (prepends `bos_token`): [vllm/model_executor/models/molmo2.py](../../../vllm/model_executor/models/molmo2.py)
734 - Molmo (applies chat template which is not defined elsewhere): [vllm/model_executor/models/molmo.py](../../../vllm/model_executor/models/molmo.py)
735
736 ### Custom HF processor
737
738 Some models don't define an HF processor class on HF Hub. In that case, you can define a custom HF processor that has the same call signature as HF processors and return it from [BaseProcessingInfo.get_hf_processor][vllm.multimodal.processing.BaseProcessingInfo.get_hf_processor]. The default [_call_hf_processor][vllm.multimodal.processing.BaseMultiModalProcessor._call_hf_processor] applies it to the normalized multi-modal data via [InputProcessingContext.call_hf_processor][vllm.multimodal.processing.InputProcessingContext.call_hf_processor]. Override `_call_hf_processor` if the custom processor instead exposes a different callable interface.
739
740 Examples:
741
742 - DeepSeek-VL2: [vllm/model_executor/models/deepseek_vl2.py](../../../vllm/model_executor/models/deepseek_vl2.py)
743 - InternVL: [vllm/model_executor/models/internvl.py](../../../vllm/model_executor/models/internvl.py)