[MM] Simplify _apply_hf_processor_main (#53275)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Cyrus Leung committed
Aug 22, 2026 at 16:49 UTC
0b19ebcacd0e9578460206035c4b3575361c19c4
90 files changed
+1810
-1668
docs/contributing/model/multimodal.md
+46
-17
@@ -153,11 +153,13 @@ def get_supported_mm_limits(self) -> Mapping[str, int | None]:
153
154
## 3. Specify dummy inputs
155
156
-Then, inherit [BaseDummyInputsBuilder][vllm.multimodal.processing.BaseDummyInputsBuilder] to construct dummy inputs for
157
-HF processing. The processed outputs are also used for memory profiling.
156
+Then, inherit [BaseDummyInputsBuilder][vllm.multimodal.processing.BaseDummyInputsBuilder] to construct the dummy inputs
157
+that are used for memory profiling.
158
159
Override the abstract methods [get_dummy_text][vllm.multimodal.processing.BaseDummyInputsBuilder.get_dummy_text] and [get_dummy_mm_data][vllm.multimodal.processing.BaseDummyInputsBuilder.get_dummy_mm_data] to construct dummy inputs. These dummy inputs should result in the worst-case memory usage of the model so that vLLM can reserve the correct amount of memory for it.
160
161
+Besides profiling, the dummy text is passed to the HF processor by models whose HF processor requires text corresponding to the multi-modal items (see [Multi-modal fields](#multi-modal-fields)). If the HF processor can process multi-modal data without any text, the dummy text is not used during HF processing.
162
+
163
Assuming that the memory usage increases with the number of tokens, the dummy inputs can be constructed to maximize the number of output embeddings, which is the same number as placeholder feature tokens.
164
165
=== "Basic example: LLaVA"
@@ -446,37 +448,64 @@ return a schema of the tensors outputted by the HF processor that are related to
448
like in LLaVA, each image's features must be independent of the others (which
449
is also required for prefix caching to work correctly). So, we un-pad each image
450
back to its own size by overriding
449
- [BaseMultiModalProcessor._call_hf_processor][vllm.multimodal.processing.BaseMultiModalProcessor._call_hf_processor]:
451
+ [BaseMultiModalProcessor._postprocess_hf_mm_data][vllm.multimodal.processing.BaseMultiModalProcessor._postprocess_hf_mm_data]
452
+ to post-process the outputs of the HF processor:
453
454
??? code
455
456
```python
454
- def _call_hf_processor(
457
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
458
+ # Mistral3Processor requires text corresponding to the images
459
+ return self.dummy_inputs.get_dummy_text(mm_counts)
460
+
461
+ def _postprocess_hf_mm_data(
462
self,
456
- prompt: str,
463
mm_data: Mapping[str, object],
458
- mm_kwargs: Mapping[str, object],
464
+ hf_processor_mm_kwargs: Mapping[str, object],
465
+ processed_data: BatchFeature,
466
) -> BatchFeature:
460
- processed_outputs = super()._call_hf_processor(
461
- prompt=prompt,
462
- mm_data=mm_data,
463
- mm_kwargs=mm_kwargs,
464
- )
467
+ if not mm_data:
468
+ return processed_data
469
466
- pixel_values = processed_outputs.get("pixel_values")
470
+ pixel_values = processed_data.get("pixel_values")
471
if pixel_values is not None:
472
# Avoid padding since we need the output for each image to be
473
# independent of other images for the cache to work correctly
470
- image_sizes = processed_outputs["image_sizes"]
474
+ image_sizes = processed_data["image_sizes"]
475
assert len(pixel_values) == len(image_sizes)
476
473
- processed_outputs["pixel_values"] = [
477
+ processed_data["pixel_values"] = [
478
p[:, :h, :w] for p, (h, w) in zip(pixel_values, image_sizes)
479
]
480
477
- return processed_outputs
481
+ return processed_data
482
```
483
484
+ The default implementation of
485
+ [_apply_hf_processor_main][vllm.multimodal.processing.BaseMultiModalProcessor._apply_hf_processor_main]
486
+ calls the HF processor on the multi-modal data without passing any text.
487
+ If the HF processor instead requires text corresponding to the multi-modal items,
488
+ you should override
489
+ [_get_hf_processor_text][vllm.multimodal.processing.BaseMultiModalProcessor._get_hf_processor_text]
490
+ to return the dummy text from
491
+ [BaseDummyInputsBuilder.get_dummy_text][vllm.multimodal.processing.BaseDummyInputsBuilder.get_dummy_text]
492
+ like in the example above. If the HF processor expects the multi-modal data
493
+ under different keys than those provided by the multi-modal items
494
+ (e.g. `audio` instead of `audios`), or requires additional keyword arguments
495
+ (e.g. `sampling_rate`), you should override
496
+ [_preprocess_hf_mm_data][vllm.multimodal.processing.BaseMultiModalProcessor._preprocess_hf_mm_data].
497
+ If you need to modify the output of the HF processor,
498
+ you should override
499
+ [_postprocess_hf_mm_data][vllm.multimodal.processing.BaseMultiModalProcessor._postprocess_hf_mm_data].
500
+ If the HF processor applies additional transformations to the prompt tokens
501
+ that are not reflected in its multi-modal outputs (e.g. prepending a BOS token),
502
+ you should override
503
+ [_postprocess_prompt][vllm.multimodal.processing.BaseMultiModalProcessor._postprocess_prompt]
504
+ to replicate them.
505
+ For even more control over how the HF processor is called, you can override
506
+ [_apply_hf_processor_main][vllm.multimodal.processing.BaseMultiModalProcessor._apply_hf_processor_main]
507
+ directly.
508
+
509
Since `pixel_values` is now a list with one tensor per image, we can override
510
[_get_mm_fields_config][vllm.multimodal.processing.BaseMultiModalProcessor._get_mm_fields_config] as follows:
511
@@ -680,7 +709,7 @@ Examples:
709
710
### Handling prompt updates unrelated to multi-modal data
711
683
-[_get_prompt_updates][vllm.multimodal.processing.BaseMultiModalProcessor._get_prompt_updates] assumes that each application of prompt update corresponds to one multi-modal item. If the HF processor performs additional processing regardless of how many multi-modal items there are, you should override [_apply_hf_processor_tokens_only][vllm.multimodal.processing.BaseMultiModalProcessor._apply_hf_processor_tokens_only] so that the processed token inputs are consistent with the result of applying the HF processor on text inputs. This is because token inputs bypass the HF processor according to [our design](../../design/mm_processing.md).
712
+[_get_prompt_updates][vllm.multimodal.processing.BaseMultiModalProcessor._get_prompt_updates] assumes that each application of prompt update corresponds to one multi-modal item. If the HF processor performs additional processing on the prompt regardless of how many multi-modal items there are, you should override [_postprocess_prompt][vllm.multimodal.processing.BaseMultiModalProcessor._postprocess_prompt] so that the prompt token IDs are consistent with the result of applying the HF processor on text inputs. This is because token inputs bypass the HF processor according to [our design](../../design/mm_processing.md).
713
714
Examples:
715
@@ -690,7 +719,7 @@ Examples:
719
720
### Custom HF processor
721
693
-Some models don't define an HF processor class on HF Hub. In that case, you can define a custom HF processor that has the same call signature as HF processors and pass it to [_call_hf_processor][vllm.multimodal.processing.BaseMultiModalProcessor._call_hf_processor].
722
+Some models don't define an HF processor class on HF Hub. In that case, you can define a custom HF processor that has the same call signature as HF processors and return it from [BaseProcessingInfo.get_hf_processor][vllm.multimodal.processing.BaseProcessingInfo.get_hf_processor]. It is then applied to the multi-modal data via [InputProcessingContext.call_hf_processor][vllm.multimodal.processing.InputProcessingContext.call_hf_processor] inside [_apply_hf_processor_main][vllm.multimodal.processing.BaseMultiModalProcessor._apply_hf_processor_main].
723
724
Examples:
725
docs/design/mm_processing.md
+11
-42
@@ -2,7 +2,15 @@
2
3
To enable various optimizations in vLLM such as [chunked prefill](../configuration/optimization.md#chunked-prefill) and [prefix caching](../features/automatic_prefix_caching.md), we use [BaseMultiModalProcessor][vllm.multimodal.processing.BaseMultiModalProcessor] to provide the correspondence between placeholder feature tokens (e.g. `<image>`) and multi-modal inputs (e.g. the raw input image) based on the outputs of HF processor.
4
5
-Here are the main features of [BaseMultiModalProcessor][vllm.multimodal.processing.BaseMultiModalProcessor]:
5
+In vLLM's rendering pipeline (see [BaseRenderer][vllm.renderers.base.BaseRenderer]), tokenization is performed as a separate step before multi-modal processing. Therefore, `BaseMultiModalProcessor` needs to recreate the output of calling HF processor end-to-end, while not being able to see the original text. This is achieved through **Dummy Input Text** and **Prompt Update Detection**.
6
+
7
+## Dummy Input Text
8
+
9
+Since Transformers 5.10, `ProcessorMixin` now allows multi-modal inputs to be passed by themselves. However, certain subclasses (such as `ChameleonProcessor`) and older out-of-tree implementations may still define their own `__call__` method that assumes the presence of text with corresponding placeholder tokens. This causes a problem as we don't have the original text anymore to pass to these processors.
10
+
11
+To work around this, each model defines how to generate dummy text based on the number of multi-modal inputs, via [get_dummy_text][vllm.multimodal.processing.BaseDummyInputsBuilder.get_dummy_text], which its override of [_get_hf_processor_text][vllm.multimodal.processing.BaseMultiModalProcessor._get_hf_processor_text] returns so that [_apply_hf_processor_main][vllm.multimodal.processing.BaseMultiModalProcessor._apply_hf_processor_main] passes it to the HF processor together with the multi-modal inputs to obtain the processed multi-modal data.
12
+
13
+Similarly, since the multi-modal data extracted by vLLM may not match what a specific HF processor expects, [_apply_hf_processor_main][vllm.multimodal.processing.BaseMultiModalProcessor._apply_hf_processor_main] allows each model to adapt the inputs via [_preprocess_hf_mm_data][vllm.multimodal.processing.BaseMultiModalProcessor._preprocess_hf_mm_data] (e.g. renaming keys like `audios` to `audio` or injecting extra keyword arguments such as `sampling_rate`) and the outputs via [_postprocess_hf_mm_data][vllm.multimodal.processing.BaseMultiModalProcessor._postprocess_hf_mm_data], without having to reimplement the entire method.
14
15
## Prompt Update Detection
16
@@ -13,46 +21,9 @@ One of the main responsibilities of HF processor is to update the prompt with pl
21
22
The information about which tokens have been updated is key to finding the correspondence between placeholder feature tokens and multi-modal inputs.
23
16
-In vLLM, this information is specified using [PromptUpdate][vllm.multimodal.processing.PromptUpdate] in [_get_prompt_updates][vllm.multimodal.processing.BaseMultiModalProcessor._get_prompt_updates]. We can automatically detect whether HF has updated the prompt by checking the existence of the updated tokens.
17
-
18
-## Tokenized Prompt Inputs
19
-
20
-To enable tokenization in a separate process, we support passing input token IDs alongside multi-modal data.
21
-
22
-### The problem
23
-
24
-Consider that HF processors follow these main steps:
25
-
26
-1. Tokenize the text
27
-2. Process multi-modal inputs
28
-3. Perform prompt updates
29
-
30
-And we require that:
31
-
32
-- For text + multi-modal inputs, apply all steps 1--3.
33
-- For tokenized + multi-modal inputs, apply only steps 2--3.
24
+Since we call HF processor without the input text, we have to perform this update by ourselves. In vLLM, we represent the necessary information using [PromptUpdate][vllm.multimodal.processing.PromptUpdate] in [_get_prompt_updates][vllm.multimodal.processing.BaseMultiModalProcessor._get_prompt_updates], and apply them via [_apply_prompt_updates][vllm.multimodal.processing.BaseMultiModalProcessor._apply_prompt_updates].
25
35
-How can we achieve this without rewriting HF processors? We can try to call the HF processor several times on different inputs:
36
-
37
-- For text + multi-modal inputs, simply call the HF processor directly.
38
-- For tokenized + multi-modal inputs, call the processor only on the multi-modal inputs.
39
-
40
-While HF processors support text + multi-modal inputs natively, this is not so for tokenized + multi-modal inputs: an error is thrown if the number of input placeholder tokens do not correspond to the number of multi-modal inputs.
41
-
42
-Moreover, since the tokenized text has not passed through the HF processor, we have to apply Step 3 by ourselves to keep the output tokens and multi-modal data consistent with each other.
43
-
44
-### Dummy text
45
-
46
-We work around the first issue by requiring each model to define how to generate dummy text based on the number of multi-modal inputs, via [get_dummy_text][vllm.multimodal.processing.BaseDummyInputsBuilder.get_dummy_text]. This lets us generate dummy text corresponding to the multi-modal inputs and input them together to obtain the processed multi-modal data.
47
-
48
-### Automatic prompt updating
49
-
50
-We address the second issue by implementing model-agnostic code in
51
-[_apply_prompt_updates][vllm.multimodal.processing.BaseMultiModalProcessor._apply_prompt_updates] to automatically update the prompt with feature placeholder tokens based on the specification outputted by [_get_prompt_updates][vllm.multimodal.processing.BaseMultiModalProcessor._get_prompt_updates].
52
-
53
-### Summary
54
-
55
-With the help of dummy text and automatic prompt updating, our multi-modal processor can finally accept both text and token prompts with multi-modal data. The detailed logic is shown in [_apply_hf_processor_main][vllm.multimodal.processing.BaseMultiModalProcessor._apply_hf_processor_main].
26
+Some HF processors additionally transform the prompt itself regardless of the multi-modal inputs (such as `ChameleonProcessor` appending a sep token for chat mode). Since the prompt tokens likewise bypass the HF processor, such transformations are replicated via [_postprocess_prompt][vllm.multimodal.processing.BaseMultiModalProcessor._postprocess_prompt] before the prompt updates are located or applied.
27
28
## Processor Output Caching
29
@@ -60,8 +31,6 @@ Some HF processors, such as the one for Qwen2-VL, are [very slow](https://github
31
32
When new data is passed in, we first check which items are in the cache, and which ones are missing. The missing items are passed into the HF processor in a single batch and cached, before being merged with the existing items in the cache.
33
63
-Since we only process the missing multi-modal data items, the number of input placeholder tokens no longer corresponds to the number of the multi-modal inputs, so they can't be passed alongside the text prompt to HF processor. Therefore, we process the text and multi-modal inputs separately, using [dummy text](#dummy-text) to avoid HF errors. Since this skips HF's prompt updating code, we apply [automatic prompt updating](#automatic-prompt-updating) afterwards to keep the output tokens and multi-modal data consistent with each other.
64
-
34
## Speeding Up Multi‑Modal Data Processing
35
36
### Fused Normalisation on the Device
requirements/common.txt
+1
-1
@@ -7,7 +7,7 @@ requests >= 2.26.0
7
tqdm
8
blake3
9
py-cpuinfo
10
-transformers >= 5.5.3
10
+transformers >= 5.10.4
11
huggingface_hub >= 1.28.0
12
tokenizers >= 0.21.1 # Required for fast incremental detokenization.
13
safetensors >= 0.6.2 # MXFP4/MXFP6 dtype support (F8_E8M0, F4) added in 0.6.0: https://github.com/huggingface/safetensors/pull/611
tests/models/multimodal/processing/test_audio_in_video.py
+1
-1
@@ -136,7 +136,7 @@ def test_use_audio_in_video_without_audio_track(model_id: str) -> None:
136
# no audio track" case. (The audio limit above is 1, not 0, so that
137
# `mm_counts["audio"]` is still declared for this request; a 0 limit
138
# would make vLLM omit the "audio" key from `mm_counts` entirely, which
139
- # trips an unrelated assertion in `_apply_hf_processor_mm_only`.)
139
+ # trips an unrelated assertion in `_apply_hf_processor_main`.)
140
mm_data = {"video": [video]}
141
142
with pytest.raises(ValueError, match="doesn't have audio track"):
tests/models/multimodal/processing/test_audioflamingo3.py
+2
-2
@@ -105,9 +105,9 @@ def test_audio_chunk_counting(mock_ctx):
105
audio_2 = np.zeros(75 * sr)
106
107
mm_data = {"audio": [audio_1, audio_2]}
108
- prompt = "<|user|>Listen.<|end|>"
108
110
- processed = processor._call_hf_processor(prompt, mm_data, {})
109
+ mm_items = processor.info.parse_mm_data(mm_data, validate=False)
110
+ processed = processor._apply_hf_processor_main(mm_items, {})
111
112
chunk_counts = processed["chunk_counts"]
113
tests/models/multimodal/processing/test_common.py
+7
-15
@@ -385,17 +385,19 @@ def test_processing_correctness(
385
num_batches: int,
386
simplify_rate: float,
387
):
388
- if model_id == "google/gemma-3n-E2B-it":
389
- pytest.skip("Fix later")
390
- if model_id == "OpenGVLab/InternVL2-2B":
391
- pytest.skip("Fix later")
388
if model_id == "openvla/openvla-7b":
389
pytest.skip(
390
"OpenVLA uses a custom vLLM processor because its HF remote "
391
"processor is incompatible with current Transformers."
392
)
393
if model_id == "jinaai/jina-reranker-m0":
398
- pytest.skip("Fix later")
394
+ pytest.skip(
395
+ "JinaVL reverses the multi-modal item order inside "
396
+ "_apply_hf_processor_main to match its document-before-query "
397
+ "score template, which breaks the positional correspondence "
398
+ "between multi-modal kwargs and hashes assumed by the "
399
+ "processing cache. Needs a fix that keeps both consistent."
400
+ )
401
if model_id == "mistralai/Voxtral-Mini-4B-Realtime-2602":
402
pytest.skip(
403
"Voxtral Realtime doesn't make use of any place-holder "
@@ -403,22 +405,12 @@ def test_processing_correctness(
405
"correctness test as is. Let's revisit adapting this "
406
"test once more realtime models exist."
407
)
406
- if model_id == "CohereLabs/cohere-transcribe-03-2026":
407
- pytest.skip("Fix later")
408
if model_id.startswith("OpenMOSS-Team/MOSS-Audio-"):
409
pytest.skip(
410
"MOSS-Audio uses a custom processor that dynamically expands "
411
"audio placeholders from processed audio lengths. Its vLLM "
412
"processor paths are covered by test_moss_audio.py."
413
)
414
- # TODO: Remove when transformers 5.15.0 is released, which contains
415
- # https://github.com/huggingface/transformers/pull/47483.
416
- if model_id == "microsoft/VibeVoice-ASR-HF":
417
- pytest.skip(
418
- "VibeVoice ASR requires audio as a positional argument and hence "
419
- "cannot pass the processing correctness test as is. Its generation "
420
- "is covered by test_transformers_audio.py."
421
- )
414
if model_id == "lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct":
415
pytest.skip(
416
"LLaVA-OneVision-2 video processing routes frames through custom "
tests/models/multimodal/processing/test_llava_onevision2.py
+1
-1
@@ -202,7 +202,7 @@ def test_validate_relative_bare_path_blocked():
202
# ---------------------------------------------------------------------------
203
# Frame backend marker -> PIL + timestamps (_frame_video_to_pil_and_timestamps)
204
#
205
-# Non-codec videos reach _call_hf_processor as a ``(frames_ndarray, metadata)``
205
+# Non-codec videos reach _apply_hf_processor_main as a ``(frames_ndarray, metadata)``
206
# tuple -- produced by the registered ``LlavaOnevision2VideoBackend`` for real
207
# ``video_url`` inputs, or by the dummy-inputs builder during profiling
208
# (``video_needs_metadata=True``). The helper materialises PIL frames and
tests/models/multimodal/processing/test_qwen3_vl.py
+2
-2
@@ -103,7 +103,7 @@ def test_processor_multi_video(
103
"""Verify that multi-video processing produces correct placeholders.
104
105
This exercises the token-level replacement path in
106
- ``_call_hf_processor`` which avoids the quadratic text-level
106
+ ``_apply_hf_processor_main`` which avoids the quadratic text-level
107
prompt expansion.
108
"""
109
ctx = build_model_context(
@@ -153,7 +153,7 @@ def test_processor_multi_video_list_kwargs(
153
``mm_processor_kwargs`` (one ``fps``/``num_frames`` per video) must not
154
crash.
155
156
- Before the fix, ``_call_hf_processor`` copied the whole kwargs to every
156
+ Before the fix, ``_apply_hf_processor_main`` copied the whole kwargs to every
157
video without slicing, so ``_get_video_second_idx`` received the list
158
where a scalar was expected and raised ``TypeError``.
159
"""
vllm/model_executor/models/audioflamingo3.py
+27
-20
@@ -377,35 +377,41 @@ class AudioFlamingo3MultiModalDataParser(MultiModalDataParser):
377
class AudioFlamingo3MultiModalProcessor(
378
BaseMultiModalProcessor[AudioFlamingo3ProcessingInfo]
379
):
380
- def _call_hf_processor(
380
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
381
+ return self.dummy_inputs.get_dummy_text(mm_counts)
382
+
383
+ def _preprocess_hf_mm_data(
384
self,
382
- prompt: str,
385
mm_data: Mapping[str, object],
384
- mm_kwargs: Mapping[str, Any],
385
- ) -> BatchFeature:
386
- processor_mm_data = dict(mm_data)
387
- audios = processor_mm_data.pop("audios", None)
386
+ hf_processor_mm_kwargs: Mapping[str, object],
387
+ ) -> tuple[Mapping[str, object], Mapping[str, object]]:
388
+ mm_data = dict(mm_data)
389
+ audios = mm_data.pop("audios", None)
390
if audios is not None:
389
- processor_mm_data["audio"] = audios
391
+ mm_data["audio"] = audios
392
391
- outputs = super()._call_hf_processor(
392
- prompt=prompt,
393
- mm_data=processor_mm_data,
394
- mm_kwargs=mm_kwargs,
395
- )
393
+ return mm_data, hf_processor_mm_kwargs
394
397
- if "input_features_mask" in outputs:
398
- outputs["feature_attention_mask"] = outputs.pop("input_features_mask")
395
+ def _postprocess_hf_mm_data(
396
+ self,
397
+ mm_data: Mapping[str, object],
398
+ hf_processor_mm_kwargs: Mapping[str, object],
399
+ processed_data: BatchFeature,
400
+ ) -> BatchFeature:
401
+ if "input_features_mask" in processed_data:
402
+ processed_data["feature_attention_mask"] = processed_data.pop(
403
+ "input_features_mask"
404
+ )
405
400
- audio_data = processor_mm_data.get("audio")
406
+ audio_data = mm_data.get("audio")
407
if audio_data is None:
402
- return outputs
408
+ return processed_data
409
410
audio_list = audio_data if isinstance(audio_data, list) else [audio_data]
411
if len(audio_list) == 0:
406
- return outputs
412
+ return processed_data
413
408
- processor = self.info.get_hf_processor(**mm_kwargs)
414
+ processor = self.info.get_hf_processor(**hf_processor_mm_kwargs)
415
feature_extractor = processor.feature_extractor
416
sampling_rate = feature_extractor.sampling_rate
417
chunk_length = feature_extractor.chunk_length
@@ -423,8 +429,9 @@ class AudioFlamingo3MultiModalProcessor(
429
n_win = max_windows
430
chunk_counts.append(n_win)
431
426
- outputs["chunk_counts"] = torch.tensor(chunk_counts, dtype=torch.long)
427
- return outputs
432
+ processed_data["chunk_counts"] = torch.tensor(chunk_counts, dtype=torch.long)
433
+
434
+ return processed_data
435
436
def _get_mm_fields_config(
437
self,
vllm/model_executor/models/blip2.py
-18
@@ -467,24 +467,6 @@ class Blip2DummyInputsBuilder(BaseDummyInputsBuilder[Blip2ProcessingInfo]):
467
468
469
class Blip2MultiModalProcessor(BaseMultiModalProcessor[Blip2ProcessingInfo]):
470
- def _call_hf_processor(
471
- self,
472
- prompt: str,
473
- mm_data: Mapping[str, object],
474
- mm_kwargs: Mapping[str, object],
475
- ) -> BatchFeature:
476
- if not mm_data:
477
- # HF processor always adds placeholders even when there's no image
478
- tokenizer = self.info.get_tokenizer()
479
- prompt_ids = tokenizer.encode(prompt)
480
- return BatchFeature(dict(input_ids=[prompt_ids]), tensor_type="pt")
481
-
482
- return super()._call_hf_processor(
483
- prompt=prompt,
484
- mm_data=mm_data,
485
- mm_kwargs=mm_kwargs,
486
- )
487
-
470
def _get_mm_fields_config(
471
self,
472
hf_inputs: BatchFeature,
vllm/model_executor/models/chameleon.py
+6
-24
@@ -135,34 +135,16 @@ class ChameleonDummyInputsBuilder(BaseDummyInputsBuilder[ChameleonProcessingInfo
135
136
137
class ChameleonMultiModalProcessor(BaseMultiModalProcessor[ChameleonProcessingInfo]):
138
- def _call_hf_processor(
139
- self,
140
- prompt: str,
141
- mm_data: Mapping[str, object],
142
- mm_kwargs: Mapping[str, object],
143
- ) -> BatchFeature:
144
- if not mm_data:
145
- prompt_ids = self.info.get_tokenizer().encode(prompt)
146
- prompt_ids = self._apply_hf_processor_tokens_only(prompt_ids)
147
- return BatchFeature(dict(input_ids=[prompt_ids]), tensor_type="pt")
148
-
149
- return super()._call_hf_processor(
150
- prompt=prompt,
151
- mm_data=mm_data,
152
- mm_kwargs=mm_kwargs,
153
- )
138
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
139
+ # ChameleonProcessor requires text corresponding to the images
140
+ return self.dummy_inputs.get_dummy_text(mm_counts)
141
155
- def _apply_hf_processor_tokens_only(
156
- self,
157
- prompt_tokens: list[int],
158
- ) -> list[int]:
142
+ def _postprocess_prompt(self, prompt: list[int]) -> list[int]:
143
# HF processor adds sep token for chat mode
144
tokenizer = self.info.get_tokenizer()
161
- vocab = tokenizer.get_vocab()
162
-
163
- sep_token_id = vocab[tokenizer.sep_token] # type: ignore
145
+ sep_token_id = tokenizer.get_vocab()[tokenizer.sep_token] # type: ignore
146
165
- return prompt_tokens + [sep_token_id]
147
+ return [*prompt, sep_token_id]
148
149
def _get_mm_fields_config(
150
self,
vllm/model_executor/models/cheers.py
-9
@@ -16,7 +16,6 @@ import torch
16
import torch.nn as nn
17
import torch.nn.functional as F
18
from einops import rearrange
19
-from transformers import BatchFeature
19
20
from vllm.config import VllmConfig
21
from vllm.config.multimodal import BaseDummyOptions
@@ -492,14 +491,6 @@ class CheersDummyInputsBuilder(BaseDummyInputsBuilder[CheersProcessingInfo]):
491
class CheersMultiModalProcessor(BaseMultiModalProcessor[CheersProcessingInfo]):
492
"""Multimodal processor for Cheers model."""
493
495
- def _call_hf_processor(
496
- self,
497
- prompt: str,
498
- mm_data: Mapping[str, object],
499
- mm_kwargs: Mapping[str, object],
500
- ) -> BatchFeature:
501
- return super()._call_hf_processor(prompt, mm_data, mm_kwargs)
502
-
494
def _get_prompt_updates(
495
self,
496
mm_items: MultiModalDataItems,
vllm/model_executor/models/cohere2_vision.py
+13
-14
@@ -222,24 +222,23 @@ class Cohere2VisionDummyInputsBuilder(
222
class Cohere2VisionMultiModalProcessor(
223
BaseMultiModalProcessor[Cohere2VisionProcessingInfo]
224
):
225
- def _call_hf_processor(
225
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
226
+ return self.dummy_inputs.get_dummy_text(mm_counts)
227
+
228
+ def _postprocess_hf_mm_data(
229
self,
227
- prompt: str,
230
mm_data: Mapping[str, object],
229
- mm_kwargs: Mapping[str, object],
231
+ hf_processor_mm_kwargs: Mapping[str, object],
232
+ processed_data: BatchFeature,
233
) -> BatchFeature:
231
- processed_outputs = super()._call_hf_processor(
232
- prompt,
233
- mm_data,
234
- mm_kwargs,
235
- )
234
+ if not mm_data:
235
+ return processed_data
236
237
- # Ensure num_patches is available for proper tensor splitting
237
if (
239
- "num_patches" not in processed_outputs
238
+ "num_patches" not in processed_data
239
and (images := mm_data.get("images")) is not None
240
):
242
- hf_processor = self.info.get_hf_processor(**mm_kwargs)
241
+ hf_processor = self.info.get_hf_processor(**hf_processor_mm_kwargs)
242
243
# Fallback calculation if HF processor didn't provide num_patches
244
mm_items = self.info.parse_mm_data({"image": images}, validate=False)
@@ -250,13 +249,13 @@ class Cohere2VisionMultiModalProcessor(
249
image_width=parsed_images.get_image_size(i).width,
250
image_height=parsed_images.get_image_size(i).height,
251
processor=hf_processor,
253
- mm_kwargs=mm_kwargs,
252
+ mm_kwargs=hf_processor_mm_kwargs,
253
)
254
for i in range(len(parsed_images))
255
]
257
- processed_outputs["num_patches"] = torch.tensor(num_patches)
256
+ processed_data["num_patches"] = torch.tensor(num_patches)
257
259
- return processed_outputs
258
+ return processed_data
259
260
def _get_mm_fields_config(
261
self,
vllm/model_executor/models/cohere_asr.py
+46
-19
@@ -8,7 +8,7 @@ from typing import Any, ClassVar
8
import torch
9
import torch.nn.functional as F
10
from torch import nn
11
-from transformers import PretrainedConfig
11
+from transformers import BatchFeature, PretrainedConfig
12
13
from vllm.compilation.decorators import support_torch_compile
14
from vllm.config import CacheConfig, ModelConfig, SpeechToTextConfig, VllmConfig
@@ -45,9 +45,12 @@ from vllm.multimodal.processing import (
45
BaseDummyInputsBuilder,
46
BaseProcessingInfo,
47
EncDecMultiModalProcessor,
48
+ ProcessorInputs,
49
PromptReplacement,
50
PromptUpdate,
51
+ TimingContext,
52
)
53
+from vllm.multimodal.processing.processor import MultiModalProcessingInfo
54
from vllm.renderers import TokenizeParams
55
from vllm.tokenizers import cached_tokenizer_from_config
56
from vllm.transformers_utils.processors.cohere_asr import (
@@ -1941,27 +1944,51 @@ class CohereASRMultiModalProcessor(EncDecMultiModalProcessor[CohereASRProcessing
1944
) -> list[int]:
1945
return [0]
1946
1944
- def _call_hf_processor(
1947
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
1948
+ return self.dummy_inputs.get_dummy_text(mm_counts)
1949
+
1950
+ def _preprocess_hf_mm_data(
1951
self,
1946
- prompt: str,
1952
mm_data: Mapping[str, object],
1948
- mm_kwargs: Mapping[str, object],
1949
- ):
1950
- if mm_data:
1951
- feature_extractor = self.info.get_feature_extractor(**mm_kwargs)
1952
- mm_data = dict(audio=dict(mm_data).pop("audios"))
1953
- mm_kwargs = dict(
1954
- **mm_kwargs,
1955
- sampling_rate=feature_extractor.sampling_rate,
1956
- )
1957
- processed_outputs = super()._call_hf_processor(
1958
- prompt=prompt,
1959
- mm_data=mm_data,
1960
- mm_kwargs=mm_kwargs,
1953
+ hf_processor_mm_kwargs: Mapping[str, object],
1954
+ ) -> tuple[Mapping[str, object], Mapping[str, object]]:
1955
+ feature_extractor = self.info.get_feature_extractor(**hf_processor_mm_kwargs)
1956
+
1957
+ mm_data = dict(mm_data)
1958
+ mm_data["audio"] = mm_data.pop("audios")
1959
+
1960
+ hf_processor_mm_kwargs = dict(
1961
+ **hf_processor_mm_kwargs,
1962
+ sampling_rate=feature_extractor.sampling_rate,
1963
)
1962
- if "labels" in processed_outputs:
1963
- processed_outputs["input_ids"] = processed_outputs.pop("labels")
1964
- return processed_outputs
1964
+
1965
+ return mm_data, hf_processor_mm_kwargs
1966
+
1967
+ def _postprocess_hf_mm_data(
1968
+ self,
1969
+ mm_data: Mapping[str, object],
1970
+ hf_processor_mm_kwargs: Mapping[str, object],
1971
+ processed_data: BatchFeature,
1972
+ ) -> BatchFeature:
1973
+ if "labels" in processed_data:
1974
+ processed_data["input_ids"] = processed_data.pop("labels")
1975
+
1976
+ return processed_data
1977
+
1978
+ def _cached_apply_hf_processor(
1979
+ self,
1980
+ inputs: ProcessorInputs,
1981
+ timing_ctx: TimingContext,
1982
+ ) -> MultiModalProcessingInfo:
1983
+ # Dithering injects noise into the extracted features, so the
1984
+ # feature extractor is not a pure function of its input. Since the
1985
+ # processing cache assumes that processor outputs are invariant
1986
+ # across calls, bypass the cache when dithering is active.
1987
+ preproc = self.info.get_hf_config().preprocessor
1988
+ if preproc.get("dither", 1e-05) > 0:
1989
+ return self._apply_hf_processor(inputs, timing_ctx)
1990
+
1991
+ return super()._cached_apply_hf_processor(inputs, timing_ctx)
1992
1993
def _get_mm_fields_config(
1994
self,
vllm/model_executor/models/colmodernvbert.py
+17
-6
@@ -154,16 +154,25 @@ class ColModernVBertDummyInputsBuilder(
154
class ColModernVBertMultiModalProcessor(
155
BaseMultiModalProcessor[ColModernVBertProcessingInfo],
156
):
157
- def _call_hf_processor(
157
+ def _apply_hf_processor_main(
158
self,
159
- prompt: str,
160
- mm_data: Mapping[str, object],
161
- mm_kwargs: Mapping[str, object],
159
+ mm_items: MultiModalDataItems,
160
+ hf_processor_mm_kwargs: Mapping[str, object],
161
) -> BatchFeature:
162
+ valid_mm_items = mm_items.select(
163
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
164
+ )
165
+ mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
166
+
167
+ if not mm_data:
168
+ return BatchFeature(dict(passthrough_data))
169
+
170
+ prompt_text = self.dummy_inputs.get_dummy_text(mm_items.get_all_counts())
171
+
172
tokenizer = self.info.get_tokenizer()
173
assert isinstance(tokenizer, HfTokenizer)
174
text_encoding = tokenizer(
166
- prompt,
175
+ prompt_text,
176
return_tensors="pt",
177
)
178
result = BatchFeature(data=dict(text_encoding))
@@ -183,7 +192,9 @@ class ColModernVBertMultiModalProcessor(
192
)
193
result.update(image_outputs)
194
186
- return result
195
+ processed_data = result
196
+ processed_data.update(passthrough_data)
197
+ return processed_data
198
199
def _get_mm_fields_config(
200
self,
vllm/model_executor/models/colpali.py
+24
-16
@@ -28,6 +28,7 @@ from vllm.model_executor.layers.pooler.tokwise import pooler_for_token_embed
28
from vllm.model_executor.layers.pooler.tokwise.heads import TokenPoolerHead
29
from vllm.model_executor.model_loader.weight_utils import default_weight_loader
30
from vllm.multimodal import MULTIMODAL_REGISTRY
31
+from vllm.multimodal.parse import MultiModalDataItems
32
33
from .interfaces import SupportsLateInteraction
34
from .interfaces_base import default_pooling_type
@@ -59,24 +60,31 @@ class ColPaliProcessingInfo(PaliGemmaProcessingInfo):
60
class ColPaliMultiModalProcessor(PaliGemmaMultiModalProcessor):
61
"""Multimodal processor for ColPali."""
62
62
- def _call_hf_processor(
63
+ def _apply_hf_processor_main(
64
self,
64
- prompt: str,
65
- mm_data: Mapping[str, object],
66
- mm_kwargs: Mapping[str, object],
65
+ mm_items: MultiModalDataItems,
66
+ hf_processor_mm_kwargs: Mapping[str, object],
67
) -> BatchFeature:
68
- if mm_data:
69
- # The ColPali tokenizer_config.json ships with a small default
70
- # max_length (50) that truncates the 1024 image tokens inserted
71
- # by PaliGemmaProcessor, causing a token-count mismatch.
72
- # vLLM enforces its own max_model_len, so we disable HF
73
- # truncation to keep all image + text tokens intact.
74
- return self.info.ctx.call_hf_processor(
75
- self.info.get_hf_processor(**mm_kwargs),
76
- dict(text=prompt, **mm_data),
77
- dict(**mm_kwargs, truncation=False),
78
- )
79
- return super()._call_hf_processor(prompt, mm_data, mm_kwargs)
68
+ valid_mm_items = mm_items.select(
69
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
70
+ )
71
+ mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
72
+
73
+ if not mm_data:
74
+ return BatchFeature(dict(passthrough_data))
75
+
76
+ # The ColPali tokenizer_config.json ships with a small default
77
+ # max_length (50) that truncates the 1024 image tokens inserted
78
+ # by PaliGemmaProcessor, causing a token-count mismatch.
79
+ # vLLM enforces its own max_model_len, so we disable HF
80
+ # truncation to keep all image + text tokens intact.
81
+ processed_data = self.info.ctx.call_hf_processor(
82
+ self.info.get_hf_processor(**hf_processor_mm_kwargs),
83
+ mm_data,
84
+ dict(**hf_processor_mm_kwargs, truncation=False),
85
+ )
86
+ processed_data.update(passthrough_data)
87
+ return processed_data
88
89
90
@default_pooling_type(seq_pooling_type="CLS", tok_pooling_type="ALL")
vllm/model_executor/models/deepseek_ocr.py
+18
-11
@@ -287,27 +287,34 @@ class DeepseekOCRDummyInputsBuilder(BaseDummyInputsBuilder[DeepseekOCRProcessing
287
class DeepseekOCRMultiModalProcessor(
288
BaseMultiModalProcessor[DeepseekOCRProcessingInfo]
289
):
290
- def _call_hf_processor(
290
+ def _apply_hf_processor_main(
291
self,
292
- prompt: str,
293
- mm_data: Mapping[str, object],
294
- mm_kwargs: Mapping[str, object],
292
+ mm_items: MultiModalDataItems,
293
+ hf_processor_mm_kwargs: Mapping[str, object],
294
) -> BatchFeature:
295
+ valid_mm_items = mm_items.select(
296
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
297
+ )
298
+ mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
299
+
300
+ prompt_text = self.dummy_inputs.get_dummy_text(mm_items.get_all_counts())
301
+
302
if mm_data:
297
- processed_outputs = self.info.ctx.call_hf_processor(
298
- self.info.get_hf_processor(**mm_kwargs),
299
- dict(prompt=prompt, **mm_data),
300
- mm_kwargs,
303
+ processed_data = self.info.ctx.call_hf_processor(
304
+ self.info.get_hf_processor(**hf_processor_mm_kwargs),
305
+ dict(prompt=prompt_text, **mm_data),
306
+ hf_processor_mm_kwargs,
307
)
308
309
else:
310
tokenizer = self.info.get_tokenizer()
311
assert isinstance(tokenizer, HfTokenizer)
306
- processed_outputs = tokenizer(
307
- prompt, add_special_tokens=True, return_tensors="pt"
312
+ processed_data = tokenizer(
313
+ prompt_text, add_special_tokens=True, return_tensors="pt"
314
)
315
310
- return processed_outputs
316
+ processed_data.update(passthrough_data)
317
+ return processed_data
318
319
def _get_mm_fields_config(
320
self,
vllm/model_executor/models/deepseek_ocr2.py
+19
-11
@@ -158,27 +158,35 @@ class DeepseekOCR2DummyInputsBuilder(
158
class DeepseekOCR2MultiModalProcessor(
159
BaseMultiModalProcessor[DeepseekOCR2ProcessingInfo]
160
):
161
- def _call_hf_processor(
161
+ def _apply_hf_processor_main(
162
self,
163
- prompt: str,
164
- mm_data: Mapping[str, object],
165
- mm_kwargs: Mapping[str, object],
163
+ mm_items: MultiModalDataItems,
164
+ hf_processor_mm_kwargs: Mapping[str, object],
165
) -> BatchFeature:
166
+ valid_mm_items = mm_items.select(
167
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
168
+ )
169
+ mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
170
+
171
+ prompt_text = self.dummy_inputs.get_dummy_text(mm_items.get_all_counts())
172
+
173
if mm_data:
168
- processed_outputs = self.info.ctx.call_hf_processor(
169
- self.info.get_hf_processor(**mm_kwargs),
170
- dict(prompt=prompt, **mm_data),
171
- mm_kwargs,
174
+ processed_data = self.info.ctx.call_hf_processor(
175
+ self.info.get_hf_processor(**hf_processor_mm_kwargs),
176
+ dict(prompt=prompt_text, **mm_data),
177
+ hf_processor_mm_kwargs,
178
)
179
180
else:
181
tokenizer = self.info.get_tokenizer()
182
assert isinstance(tokenizer, HfTokenizer)
177
- processed_outputs = tokenizer(
178
- prompt, add_special_tokens=True, return_tensors="pt"
183
+ processed_data = tokenizer(
184
+ prompt_text, add_special_tokens=True, return_tensors="pt"
185
)
186
181
- return processed_outputs
187
+ processed_data.update(passthrough_data)
188
+
189
+ return processed_data
190
191
def _get_mm_fields_config(
192
self,
vllm/model_executor/models/deepseek_vl2.py
+11
-17
@@ -43,7 +43,6 @@ from vllm.multimodal.processing.processor import (
43
)
44
from vllm.sequence import IntermediateTensors
45
from vllm.tokenizers import cached_tokenizer_from_config
46
-from vllm.tokenizers.hf import HfTokenizer
46
from vllm.transformers_utils.configs.deepseek_vl2 import (
47
DeepseekVLV2Config,
48
MlpProjectorConfig,
@@ -236,28 +235,23 @@ class DeepseekVL2DummyInputsBuilder(BaseDummyInputsBuilder[DeepseekVL2Processing
235
class DeepseekVL2MultiModalProcessor(
236
BaseMultiModalProcessor[DeepseekVL2ProcessingInfo]
237
):
239
- def _call_hf_processor(
238
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
239
+ return self.dummy_inputs.get_dummy_text(mm_counts)
240
+
241
+ def _postprocess_hf_mm_data(
242
self,
241
- prompt: str,
243
mm_data: Mapping[str, object],
243
- mm_kwargs: Mapping[str, object],
244
+ hf_processor_mm_kwargs: Mapping[str, object],
245
+ processed_data: BatchFeature,
246
) -> BatchFeature:
247
if not mm_data:
246
- tokenizer = self.info.get_tokenizer()
247
- assert isinstance(tokenizer, HfTokenizer)
248
- return tokenizer(prompt, add_special_tokens=True, return_tensors="pt")
249
-
250
- processed_outputs = super()._call_hf_processor(
251
- prompt=prompt,
252
- mm_data=mm_data,
253
- mm_kwargs=mm_kwargs,
254
- )
248
+ return processed_data
249
256
- processed_outputs["num_patches"] = (
257
- processed_outputs["images_spatial_crop"].prod(-1) + 1
250
+ processed_data["num_patches"] = (
251
+ processed_data["images_spatial_crop"].prod(-1) + 1
252
)
253
260
- return processed_outputs
254
+ return processed_data
255
256
def _get_mm_fields_config(
257
self,
@@ -315,7 +309,7 @@ class DeepseekVL2MultiModalProcessor(
309
self,
310
inputs: ProcessorInputs,
311
timing_ctx: TimingContext,
318
- ) -> tuple[list[int], MultiModalProcessingInfo, bool]:
312
+ ) -> MultiModalProcessingInfo:
313
# The processor logic is different for len(images) <= 2 vs > 2
314
# Since the processing cache assumes that the processor output is
315
# invariant of how many images are passed per prompt, we only
vllm/model_executor/models/ernie45_vl.py
+21
-18
@@ -1110,21 +1110,20 @@ class Ernie4_5VLMultiModalProcessor(BaseMultiModalProcessor[Ernie4_5_VLProcessin
1110
pixel_values = pixel_values.to(hf_config.dtype)
1111
return pixel_values
1112
1113
- def _call_hf_processor(
1113
+ def _apply_hf_processor_main(
1114
self,
1115
- prompt: str,
1116
- mm_data: Mapping[str, object],
1117
- mm_kwargs: Mapping[str, object],
1115
+ mm_items: MultiModalDataItems,
1116
+ hf_processor_mm_kwargs: Mapping[str, object],
1117
) -> BatchFeature:
1119
- # when the prompt is not empty but the multimodal data is empty,
1120
- # directly invoke the tokenizer.
1121
- if "images" not in mm_data and "videos" not in mm_data and prompt != "":
1122
- tokenizer = self.info.get_tokenizer()
1123
- prompt_ids = tokenizer.encode(prompt)
1124
- tokenizer_output = BatchFeature(
1125
- dict(input_ids=[prompt_ids]), tensor_type="pt"
1126
- )
1127
- return tokenizer_output
1118
+ valid_mm_items = mm_items.select(
1119
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
1120
+ )
1121
+ mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
1122
+
1123
+ if "images" not in mm_data and "videos" not in mm_data:
1124
+ return BatchFeature(dict(passthrough_data))
1125
+
1126
+ prompt_text = self.dummy_inputs.get_dummy_text(mm_items.get_all_counts())
1127
1128
if "images" not in mm_data:
1129
mm_data["images"] = []
@@ -1132,7 +1131,7 @@ class Ernie4_5VLMultiModalProcessor(BaseMultiModalProcessor[Ernie4_5_VLProcessin
1131
mm_data["videos"] = []
1132
1133
# Check if HF processor supports video metadata
1135
- hf_processor = self.info.get_hf_processor(**mm_kwargs)
1134
+ hf_processor = self.info.get_hf_processor(**hf_processor_mm_kwargs)
1135
supports_video_metadata = getattr(
1136
hf_processor, "supports_video_metadata", False
1137
)
@@ -1149,8 +1148,10 @@ class Ernie4_5VLMultiModalProcessor(BaseMultiModalProcessor[Ernie4_5_VLProcessin
1148
1149
processor_output = self.info.ctx.call_hf_processor(
1150
hf_processor,
1152
- dict(text=[prompt], images=mm_data["images"], videos=mm_data["videos"]),
1153
- mm_kwargs,
1151
+ dict(
1152
+ text=[prompt_text], images=mm_data["images"], videos=mm_data["videos"]
1153
+ ),
1154
+ hf_processor_mm_kwargs,
1155
)
1156
1157
# Divide the processor_output into two modalities: image and video.
@@ -1158,7 +1159,7 @@ class Ernie4_5VLMultiModalProcessor(BaseMultiModalProcessor[Ernie4_5_VLProcessin
1159
pixel_values = processor_output["images"]
1160
if pixel_values is not None:
1161
processor_output["images"] = self._pixel_values_norm(
1161
- pixel_values, mm_kwargs
1162
+ pixel_values, hf_processor_mm_kwargs
1163
)
1164
for key in list(processor_output.keys()):
1165
if processor_output[key] is None:
@@ -1184,7 +1185,9 @@ class Ernie4_5VLMultiModalProcessor(BaseMultiModalProcessor[Ernie4_5_VLProcessin
1185
]
1186
del processor_output["images"]
1187
1187
- return processor_output
1188
+ processed_data = processor_output
1189
+ processed_data.update(passthrough_data)
1190
+ return processed_data
1191
1192
def _get_prompt_updates(
1193
self,
vllm/model_executor/models/fireredasr2.py
+27
-18
@@ -228,27 +228,36 @@ class FireRedASR2DummyInputsBuilder(BaseDummyInputsBuilder[FireRedASR2Processing
228
class FireRedASR2MultiModalProcessor(
229
BaseMultiModalProcessor[FireRedASR2ProcessingInfo]
230
):
231
- def _call_hf_processor(
231
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
232
+ return self.dummy_inputs.get_dummy_text(mm_counts)
233
+
234
+ def _preprocess_hf_mm_data(
235
self,
233
- prompt: str,
236
mm_data: Mapping[str, object],
235
- mm_kwargs: Mapping[str, object],
236
- ) -> BatchFeature:
237
- if mm_data:
238
- feature_extractor = self.info.get_feature_extractor(**mm_kwargs)
239
- mm_data = dict(audio=mm_data.pop("audios"))
240
- mm_kwargs = dict(
241
- **mm_kwargs,
242
- sampling_rate=feature_extractor.sampling_rate,
243
- )
244
- processed_outputs = super()._call_hf_processor(
245
- prompt=prompt,
246
- mm_data=mm_data,
247
- mm_kwargs=mm_kwargs,
237
+ hf_processor_mm_kwargs: Mapping[str, object],
238
+ ) -> tuple[Mapping[str, object], Mapping[str, object]]:
239
+ feature_extractor = self.info.get_feature_extractor(**hf_processor_mm_kwargs)
240
+
241
+ mm_data = dict(mm_data)
242
+ mm_data["audio"] = mm_data.pop("audios")
243
+
244
+ hf_processor_mm_kwargs = dict(
245
+ **hf_processor_mm_kwargs,
246
+ sampling_rate=feature_extractor.sampling_rate,
247
)
249
- if "labels" in processed_outputs:
250
- processed_outputs["input_ids"] = processed_outputs.pop("labels")
251
- return processed_outputs
248
+
249
+ return mm_data, hf_processor_mm_kwargs
250
+
251
+ def _postprocess_hf_mm_data(
252
+ self,
253
+ mm_data: Mapping[str, object],
254
+ hf_processor_mm_kwargs: Mapping[str, object],
255
+ processed_data: BatchFeature,
256
+ ) -> BatchFeature:
257
+ if "labels" in processed_data:
258
+ processed_data["input_ids"] = processed_data.pop("labels")
259
+
260
+ return processed_data
261
262
def _get_mm_fields_config(
263
self,
vllm/model_executor/models/fireredlid.py
+27
-18
@@ -451,27 +451,36 @@ class FireRedLIDMultiModalProcessor(
451
# Dummy encoder prompt for profiling (encoder only processes audio).
452
return [0]
453
454
- def _call_hf_processor(
454
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
455
+ return self.dummy_inputs.get_dummy_text(mm_counts)
456
+
457
+ def _preprocess_hf_mm_data(
458
self,
456
- prompt: str,
459
mm_data: Mapping[str, object],
458
- mm_kwargs: Mapping[str, object],
459
- ) -> BatchFeature:
460
- if mm_data:
461
- feature_extractor = self.info.get_feature_extractor(**mm_kwargs)
462
- mm_data = dict(audio=mm_data.pop("audios"))
463
- mm_kwargs = dict(
464
- **mm_kwargs,
465
- sampling_rate=feature_extractor.sampling_rate,
466
- )
467
- processed_outputs = super()._call_hf_processor(
468
- prompt=prompt,
469
- mm_data=mm_data,
470
- mm_kwargs=mm_kwargs,
460
+ hf_processor_mm_kwargs: Mapping[str, object],
461
+ ) -> tuple[Mapping[str, object], Mapping[str, object]]:
462
+ feature_extractor = self.info.get_feature_extractor(**hf_processor_mm_kwargs)
463
+
464
+ mm_data = dict(mm_data)
465
+ mm_data["audio"] = mm_data.pop("audios")
466
+
467
+ hf_processor_mm_kwargs = dict(
468
+ **hf_processor_mm_kwargs,
469
+ sampling_rate=feature_extractor.sampling_rate,
470
)
472
- if "labels" in processed_outputs:
473
- processed_outputs["input_ids"] = processed_outputs.pop("labels")
474
- return processed_outputs
471
+
472
+ return mm_data, hf_processor_mm_kwargs
473
+
474
+ def _postprocess_hf_mm_data(
475
+ self,
476
+ mm_data: Mapping[str, object],
477
+ hf_processor_mm_kwargs: Mapping[str, object],
478
+ processed_data: BatchFeature,
479
+ ) -> BatchFeature:
480
+ if "labels" in processed_data:
481
+ processed_data["input_ids"] = processed_data.pop("labels")
482
+
483
+ return processed_data
484
485
def _get_mm_fields_config(
486
self,
vllm/model_executor/models/funasr.py
+44
-18
@@ -44,9 +44,12 @@ from vllm.multimodal.processing import (
44
BaseDummyInputsBuilder,
45
BaseMultiModalProcessor,
46
BaseProcessingInfo,
47
+ ProcessorInputs,
48
PromptReplacement,
49
PromptUpdate,
50
+ TimingContext,
51
)
52
+from vllm.multimodal.processing.processor import MultiModalProcessingInfo
53
from vllm.transformers_utils.processor import cached_processor_from_config
54
from vllm.transformers_utils.processors.funasr import FunASRFeatureExtractor
55
from vllm.utils.tensor_schema import TensorSchema, TensorShape
@@ -747,27 +750,50 @@ class FunASRDummyInputsBuilder(BaseDummyInputsBuilder[FunASRProcessingInfo]):
750
751
752
class FunASRMultiModalProcessor(BaseMultiModalProcessor[FunASRProcessingInfo]):
750
- def _call_hf_processor(
753
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
754
+ return self.dummy_inputs.get_dummy_text(mm_counts)
755
+
756
+ def _preprocess_hf_mm_data(
757
+ self,
758
+ mm_data: Mapping[str, object],
759
+ hf_processor_mm_kwargs: Mapping[str, object],
760
+ ) -> tuple[Mapping[str, object], Mapping[str, object]]:
761
+ feature_extractor = self.info.get_feature_extractor(**hf_processor_mm_kwargs)
762
+
763
+ mm_data = dict(mm_data)
764
+ mm_data["audio"] = mm_data.pop("audios")
765
+
766
+ hf_processor_mm_kwargs = dict(
767
+ **hf_processor_mm_kwargs,
768
+ sampling_rate=feature_extractor.sampling_rate,
769
+ )
770
+
771
+ return mm_data, hf_processor_mm_kwargs
772
+
773
+ def _postprocess_hf_mm_data(
774
self,
752
- prompt: str,
775
mm_data: Mapping[str, object],
754
- mm_kwargs: Mapping[str, object],
776
+ hf_processor_mm_kwargs: Mapping[str, object],
777
+ processed_data: BatchFeature,
778
) -> BatchFeature:
756
- if mm_data:
757
- feature_extractor = self.info.get_feature_extractor(**mm_kwargs)
758
- mm_data = dict(audio=mm_data.pop("audios"))
759
- mm_kwargs = dict(
760
- **mm_kwargs,
761
- sampling_rate=feature_extractor.sampling_rate,
762
- )
763
- processed_outputs = super()._call_hf_processor(
764
- prompt=prompt,
765
- mm_data=mm_data,
766
- mm_kwargs=mm_kwargs,
767
- )
768
- if "labels" in processed_outputs:
769
- processed_outputs["input_ids"] = processed_outputs.pop("labels")
770
- return processed_outputs
779
+ if "labels" in processed_data:
780
+ processed_data["input_ids"] = processed_data.pop("labels")
781
+
782
+ return processed_data
783
+
784
+ def _cached_apply_hf_processor(
785
+ self,
786
+ inputs: ProcessorInputs,
787
+ timing_ctx: TimingContext,
788
+ ) -> MultiModalProcessingInfo:
789
+ # Dithering injects noise into the extracted features, so the
790
+ # feature extractor is not a pure function of its input. Since the
791
+ # processing cache assumes that processor outputs are invariant
792
+ # across calls, bypass the cache when dithering is active.
793
+ if self.info.get_feature_extractor().dither > 0:
794
+ return self._apply_hf_processor(inputs, timing_ctx)
795
+
796
+ return super()._cached_apply_hf_processor(inputs, timing_ctx)
797
798
def _get_mm_fields_config(
799
self,
vllm/model_executor/models/funaudiochat.py
+21
-8
@@ -612,20 +612,31 @@ class FunAudioChatDummyInputsBuilder(
612
class FunAudioChatMultiModalProcessor(
613
BaseMultiModalProcessor[FunAudioChatProcessingInfo]
614
):
615
- def _call_hf_processor(
615
+ def _apply_hf_processor_main(
616
self,
617
- prompt: str,
618
- mm_data: Mapping[str, object],
619
- mm_kwargs: Mapping[str, object],
617
+ mm_items: MultiModalDataItems,
618
+ hf_processor_mm_kwargs: Mapping[str, object],
619
) -> BatchFeature:
620
+ valid_mm_items = mm_items.select(
621
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
622
+ )
623
+ mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
624
+
625
+ if not mm_data:
626
+ return BatchFeature(dict(passthrough_data))
627
+
628
+ prompt_text = self.dummy_inputs.get_dummy_text(mm_items.get_all_counts())
629
+
630
tokenizer = self.info.get_tokenizer()
622
- input_ids = torch.tensor([tokenizer.encode(prompt)])
631
+ input_ids = torch.tensor([tokenizer.encode(prompt_text)])
632
633
audios = mm_data.get("audios", [])
634
if not audios:
626
- return BatchFeature({"input_ids": input_ids})
635
+ processed_data = BatchFeature({"input_ids": input_ids})
636
+ processed_data.update(passthrough_data)
637
+ return processed_data
638
628
- feature_extractor = self.info.get_feature_extractor(**mm_kwargs)
639
+ feature_extractor = self.info.get_feature_extractor(**hf_processor_mm_kwargs)
640
sr = int(feature_extractor.sampling_rate)
641
min_samples = int(getattr(feature_extractor, "n_fft", 400) or 400)
642
@@ -676,7 +687,9 @@ class FunAudioChatMultiModalProcessor(
687
"feature_exist_mask": torch.ones((len(wavs),), dtype=torch.bool),
688
}
689
679
- return BatchFeature({"input_ids": input_ids, **mm_inputs})
690
+ processed_data = BatchFeature({"input_ids": input_ids, **mm_inputs})
691
+ processed_data.update(passthrough_data)
692
+ return processed_data
693
694
def _get_mm_fields_config(
695
self,
vllm/model_executor/models/gemma3_mm.py
+12
-13
@@ -261,39 +261,38 @@ class Gemma3DummyInputsBuilder(BaseDummyInputsBuilder[Gemma3ProcessingInfo]):
261
262
263
class Gemma3MultiModalProcessor(BaseMultiModalProcessor[Gemma3ProcessingInfo]):
264
- def _call_hf_processor(
264
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
265
+ return self.dummy_inputs.get_dummy_text(mm_counts)
266
+
267
+ def _postprocess_hf_mm_data(
268
self,
266
- prompt: str,
269
mm_data: Mapping[str, object],
268
- mm_kwargs: Mapping[str, object],
270
+ hf_processor_mm_kwargs: Mapping[str, object],
271
+ processed_data: BatchFeature,
272
) -> BatchFeature:
270
- processed_outputs = super()._call_hf_processor(
271
- prompt,
272
- mm_data,
273
- mm_kwargs,
274
- )
273
+ if not mm_data:
274
+ return processed_data
275
276
- # HF processor pops the `num_crops` kwarg, which is needed by vLLM
276
if (images := mm_data.get("images")) is not None:
277
mm_items = self.info.parse_mm_data({"image": images}, validate=False)
278
parsed_images = mm_items.get_items("image", ImageProcessorItems)
279
image_sizes = [
280
parsed_images.get_image_size(i) for i in range(len(parsed_images))
281
]
283
- hf_processor = self.info.get_hf_processor(**mm_kwargs)
282
+ hf_processor = self.info.get_hf_processor(**hf_processor_mm_kwargs)
283
284
num_crops = [
285
self.info.get_num_crops(
286
image_width=size.width,
287
image_height=size.height,
288
processor=hf_processor,
290
- mm_kwargs=mm_kwargs,
289
+ mm_kwargs=hf_processor_mm_kwargs,
290
)
291
for size in image_sizes
292
]
294
- processed_outputs["num_patches"] = torch.tensor(num_crops) + 1
293
+ processed_data["num_patches"] = torch.tensor(num_crops) + 1
294
296
- return processed_outputs
295
+ return processed_data
296
297
def _get_mm_fields_config(
298
self,
vllm/model_executor/models/gemma3n_mm.py
+57
-27
@@ -1,5 +1,6 @@
1
# SPDX-License-Identifier: Apache-2.0
2
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
+import inspect
4
from collections.abc import Iterable, Mapping, Sequence
5
from typing import Annotated, Any, Literal
6
@@ -257,40 +258,69 @@ class Gemma3nDummyInputsBuilder(BaseDummyInputsBuilder[Gemma3nProcessingInfo]):
258
259
260
class Gemma3nMultiModalProcessor(BaseMultiModalProcessor[Gemma3nProcessingInfo]):
260
- def _call_hf_processor(
261
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
262
+ return self.dummy_inputs.get_dummy_text(mm_counts)
263
+
264
+ def _preprocess_hf_mm_data(
265
self,
262
- prompt: str,
266
mm_data: Mapping[str, object],
264
- mm_kwargs: Mapping[str, object],
265
- ) -> BatchFeature:
266
- # HF Transformers audio processor no longer accepts `audios` key.
267
- # We pop `audios` and replace it with `audio` key to suppress
268
- # the warning.
267
+ hf_processor_mm_kwargs: Mapping[str, object],
268
+ ) -> tuple[Mapping[str, object], Mapping[str, object]]:
269
+ mm_data = dict(mm_data)
270
if "audios" in mm_data:
271
mm_data["audio"] = mm_data.pop("audios")
271
- processed_outputs = super()._call_hf_processor(
272
- prompt,
273
- mm_data,
274
- mm_kwargs,
275
- )
272
277
- if "input_features" in processed_outputs:
278
- # Padding enables audio_tower to run in batched mode
279
- processed_outputs["input_features_padded"] = processed_outputs[
280
- "input_features"
281
- ]
273
+ return mm_data, hf_processor_mm_kwargs
274
283
- # Unpad features here since we need the output of each item to be
284
- # independent of other items for the cache to work correctly
285
- unpadded_features = [
286
- f[mask]
287
- for f, mask in zip(
288
- processed_outputs["input_features"],
289
- processed_outputs["input_features_mask"],
290
- )
275
+ def _postprocess_hf_mm_data(
276
+ self,
277
+ mm_data: Mapping[str, object],
278
+ hf_processor_mm_kwargs: Mapping[str, object],
279
+ processed_data: BatchFeature,
280
+ ) -> BatchFeature:
281
+ if "input_features" in processed_data:
282
+ # The feature extractor pads every clip to the longest one in
283
+ # the batch, so each clip's frame count depends on the other
284
+ # clips it was processed with. Trim each clip to the number of
285
+ # frames it would have if processed alone so that each item's
286
+ # output (and thus its cache entry) is independent of the other
287
+ # items in the batch. `batch_audio_features` re-pads the
288
+ # per-item features so audio_tower can still run batched.
289
+ num_frames = [
290
+ self._get_num_audio_frames_alone(len(audio))
291
+ for audio in mm_data["audio"]
292
]
292
- processed_outputs["input_features"] = unpadded_features
293
- return processed_outputs
293
+ processed_data["input_features_padded"] = [
294
+ features[:n]
295
+ for features, n in zip(processed_data.pop("input_features"), num_frames)
296
+ ]
297
+ processed_data["input_features_mask"] = [
298
+ mask[:n]
299
+ for mask, n in zip(processed_data["input_features_mask"], num_frames)
300
+ ]
301
+
302
+ return processed_data
303
+
304
+ def _get_num_audio_frames_alone(self, num_samples: int) -> int:
305
+ """
306
+ Get the number of mel frames that the feature extractor produces
307
+ for a single audio clip of the given length, independent of batch
308
+ padding.
309
+ """
310
+ feature_extractor = self.info.get_feature_extractor()
311
+ call_params = inspect.signature(type(feature_extractor).__call__).parameters
312
+ max_length = call_params["max_length"].default
313
+ pad_to_multiple_of = call_params["pad_to_multiple_of"].default
314
+
315
+ num_samples = min(num_samples, max_length)
316
+ remainder = num_samples % pad_to_multiple_of
317
+ if remainder:
318
+ num_samples += pad_to_multiple_of - remainder
319
+
320
+ frame_size = feature_extractor.frame_length + 1
321
+ num_frames = (num_samples - frame_size) // feature_extractor.hop_length + 1
322
+
323
+ return max(num_frames, 0)
324
325
def _get_mm_fields_config(
326
self,
vllm/model_executor/models/gemma4_mm.py
+38
-28
@@ -560,13 +560,22 @@ class Gemma4DummyInputsBuilder(BaseDummyInputsBuilder[Gemma4ProcessingInfo]):
560
561
562
class Gemma4MultiModalProcessor(BaseMultiModalProcessor[Gemma4ProcessingInfo]):
563
- def _call_hf_processor(
563
+ def _apply_hf_processor_main(
564
self,
565
- prompt: str,
566
- mm_data: Mapping[str, object],
567
- mm_kwargs: Mapping[str, object],
565
+ mm_items: MultiModalDataItems,
566
+ hf_processor_mm_kwargs: Mapping[str, object],
567
) -> BatchFeature:
569
- merged_kwargs = self.info.ctx.get_merged_mm_kwargs(mm_kwargs)
568
+ valid_mm_items = mm_items.select(
569
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
570
+ )
571
+ mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
572
+
573
+ if not mm_data:
574
+ return BatchFeature(dict(passthrough_data))
575
+
576
+ prompt_text = self.dummy_inputs.get_dummy_text(mm_items.get_all_counts())
577
+
578
+ merged_kwargs = self.info.ctx.get_merged_mm_kwargs(hf_processor_mm_kwargs)
579
val, is_top_level_max_soft_tokens = _get_max_soft_tokens(merged_kwargs)
580
581
if val is not None and val not in _SUPPORTED_SOFT_TOKENS:
@@ -611,15 +620,15 @@ class Gemma4MultiModalProcessor(BaseMultiModalProcessor[Gemma4ProcessingInfo]):
620
timestamps = [idx / fps for idx in frame_indices]
621
622
# Process frames as images with max_soft_tokens=70
614
- video_mm_kwargs = dict(mm_kwargs)
623
+ video_mm_kwargs = dict(hf_processor_mm_kwargs)
624
video_mm_kwargs["max_soft_tokens"] = _VIDEO_MAX_SOFT_TOKENS
625
626
dummy_prompt = ("\t" + processor.image_token) * len(frames)
627
619
- frame_outputs = super()._call_hf_processor(
620
- prompt=dummy_prompt,
621
- mm_data={"images": frames},
622
- mm_kwargs=video_mm_kwargs,
628
+ frame_outputs = self.info.ctx.call_hf_processor(
629
+ self.info.get_hf_processor(**video_mm_kwargs),
630
+ dict(text=dummy_prompt, **{"images": frames}),
631
+ video_mm_kwargs,
632
)
633
634
# Remap HF key name
@@ -660,7 +669,7 @@ class Gemma4MultiModalProcessor(BaseMultiModalProcessor[Gemma4ProcessingInfo]):
669
# split-replace bug where replacement text (which itself
670
# contains <|video|> tokens) collides with later splits.
671
vt = processor.video_token
663
- parts = prompt.split(vt, len(video_replacements))
672
+ parts = prompt_text.split(vt, len(video_replacements))
673
674
# NOTE: len(parts) <= len(video_replacements) + 1
675
parts_with_repl: list[str] = []
@@ -668,7 +677,7 @@ class Gemma4MultiModalProcessor(BaseMultiModalProcessor[Gemma4ProcessingInfo]):
677
parts_with_repl.extend([part, repl])
678
parts_with_repl.extend(parts[len(video_replacements) :])
679
671
- prompt = "".join(parts_with_repl)
680
+ prompt_text = "".join(parts_with_repl)
681
682
video_outputs = {
683
"pixel_values_videos": torch.cat(all_video_pixel_values, dim=0),
@@ -712,45 +721,46 @@ class Gemma4MultiModalProcessor(BaseMultiModalProcessor[Gemma4ProcessingInfo]):
721
# NOTE: This requires a corresponding type annotation on the
722
# HF side (Gemma4ProcessorKwargs.images_kwargs) so that
723
# _merge_kwargs routes max_soft_tokens into images_kwargs.
715
- patched_mm_kwargs = dict(mm_kwargs)
724
+ patched_mm_kwargs = dict(hf_processor_mm_kwargs)
725
if val is not None and is_top_level_max_soft_tokens:
726
patched_mm_kwargs["max_soft_tokens"] = val
727
719
- processed_outputs = super()._call_hf_processor(
720
- prompt,
721
- mm_data,
728
+ processed_data = self.info.ctx.call_hf_processor(
729
+ self.info.get_hf_processor(**patched_mm_kwargs),
730
+ dict(text=prompt_text, **mm_data),
731
patched_mm_kwargs,
732
)
733
734
# HF uses 'image_position_ids'; vLLM uses 'pixel_position_ids'.
735
# Remap here to keep a single translation point.
727
- if "image_position_ids" in processed_outputs:
728
- processed_outputs["pixel_position_ids"] = processed_outputs.pop(
736
+ if "image_position_ids" in processed_data:
737
+ processed_data["pixel_position_ids"] = processed_data.pop(
738
"image_position_ids"
739
)
740
732
- if "input_features" in processed_outputs:
741
+ if "input_features" in processed_data:
742
# Unpad per-item so each item's cache entry is
743
# self-contained. The batched() field config in
744
# _get_mm_fields_config stacks equal-length items and leaves
745
# the rest as a list; _process_audio_input re-pads that list
746
# to the batch's max length.
738
- masks = processed_outputs["input_features_mask"]
747
+ masks = processed_data["input_features_mask"]
748
unpadded_features = [
749
f[mask]
750
for f, mask in zip(
742
- processed_outputs["input_features"],
751
+ processed_data["input_features"],
752
masks,
753
)
754
]
755
unpadded_masks = [mask[mask] for mask in masks]
747
- processed_outputs["input_features"] = unpadded_features
748
- processed_outputs["input_features_padded"] = unpadded_features
749
- processed_outputs["input_features_mask"] = unpadded_masks
756
+ processed_data["input_features"] = unpadded_features
757
+ processed_data["input_features_padded"] = unpadded_features
758
+ processed_data["input_features_mask"] = unpadded_masks
759
+
760
+ processed_data.update(video_outputs)
761
+ processed_data.update(passthrough_data)
762
751
- # Merge video outputs into the final result
752
- combined_outputs = dict(processed_outputs, **video_outputs)
753
- return BatchFeature(combined_outputs)
763
+ return processed_data
764
765
def _get_mm_fields_config(
766
self,
@@ -815,7 +825,7 @@ class Gemma4MultiModalProcessor(BaseMultiModalProcessor[Gemma4ProcessingInfo]):
825
image_size = images.get_image_size(item_idx)
826
# Resolve the effective max_soft_tokens by merging
827
# per-prompt kwargs with the config-level defaults,
818
- # consistent with how _call_hf_processor resolves it.
828
+ # consistent with how _apply_hf_processor_main resolves it.
829
# Without this merge, a missing per-prompt override
830
# would fall back to vision_cfg.default_output_length
831
# instead of the config's mm_processor_kwargs default.
vllm/model_executor/models/glm4_1v.py
+41
-31
@@ -1576,32 +1576,39 @@ class Glm4vMultiModalProcessor(BaseMultiModalProcessor[Glm4vProcessingInfo]):
1576
1577
return prepared_data, prepared_kwargs
1578
1579
- def _call_hf_processor(
1579
+ def _apply_hf_processor_main(
1580
self,
1581
- prompt: str,
1582
- mm_data: Mapping[str, object],
1583
- mm_kwargs: Mapping[str, object],
1581
+ mm_items: MultiModalDataItems,
1582
+ hf_processor_mm_kwargs: Mapping[str, object],
1583
) -> BatchFeature:
1585
- mm_data = dict(mm_data)
1584
+ valid_mm_items = mm_items.select(
1585
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
1586
+ )
1587
+ mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
1588
+
1589
if not mm_data:
1587
- tokenizer = self.info.get_tokenizer()
1588
- prompt_ids = tokenizer.encode(prompt, add_special_tokens=False)
1589
- return BatchFeature(dict(input_ids=[prompt_ids]), tensor_type="pt")
1590
+ return BatchFeature(dict(passthrough_data))
1591
1591
- processor = self.info.get_hf_processor(**mm_kwargs)
1592
+ prompt_text = self.dummy_inputs.get_dummy_text(mm_items.get_all_counts())
1593
+
1594
+ mm_data = dict(mm_data)
1595
+
1596
+ processor = self.info.get_hf_processor(**hf_processor_mm_kwargs)
1597
1598
use_direct_path = (
1599
not isinstance(processor, Glm4vProcessor) and TRANSFORMERS_WITH_GA
1600
)
1601
if use_direct_path:
1602
prepared_data, prepared_kwargs = self._get_direct_path_inputs(
1598
- mm_data, mm_kwargs
1603
+ mm_data, hf_processor_mm_kwargs
1604
)
1600
- return super()._call_hf_processor(
1601
- prompt=prompt,
1602
- mm_data=prepared_data,
1603
- mm_kwargs=prepared_kwargs,
1605
+ processed_data = self.info.ctx.call_hf_processor(
1606
+ self.info.get_hf_processor(**prepared_kwargs),
1607
+ dict(text=prompt_text, **prepared_data),
1608
+ prepared_kwargs,
1609
)
1610
+ processed_data.update(passthrough_data)
1611
+ return processed_data
1612
1613
if (
1614
"videos" in mm_data
@@ -1616,7 +1623,7 @@ class Glm4vMultiModalProcessor(BaseMultiModalProcessor[Glm4vProcessingInfo]):
1623
video_array, metadata = item
1624
1625
# don't update mm_kwargs inplace
1619
- video_mm_kwargs = dict(**mm_kwargs)
1626
+ video_mm_kwargs = dict(**hf_processor_mm_kwargs)
1627
video_mm_kwargs["do_sample_frames"] = metadata.get(
1628
"do_sample_frames", True
1629
)
@@ -1625,10 +1632,13 @@ class Glm4vMultiModalProcessor(BaseMultiModalProcessor[Glm4vProcessingInfo]):
1632
video_mm_data["videos"] = [[video_array]]
1633
video_mm_data["video_metadata"] = [[_to_video_metadata(metadata)]]
1634
1628
- video_outputs = super()._call_hf_processor(
1629
- prompt="<|begin_of_video|><|video|><|end_of_video|>",
1630
- mm_data=video_mm_data,
1631
- mm_kwargs=video_mm_kwargs,
1635
+ video_outputs = self.info.ctx.call_hf_processor(
1636
+ self.info.get_hf_processor(**video_mm_kwargs),
1637
+ dict(
1638
+ text="<|begin_of_video|><|video|><|end_of_video|>",
1639
+ **video_mm_data,
1640
+ ),
1641
+ video_mm_kwargs,
1642
)
1643
input_ids = video_outputs.pop("input_ids")
1644
if swap_video_frame_tokens:
@@ -1636,7 +1646,7 @@ class Glm4vMultiModalProcessor(BaseMultiModalProcessor[Glm4vProcessingInfo]):
1646
processor.video_token_id
1647
)
1648
video_placeholder = processor.tokenizer.batch_decode(input_ids)[0]
1639
- prompt = prompt.replace(
1649
+ prompt_text = prompt_text.replace(
1650
"<|begin_of_video|><|video|><|end_of_video|>",
1651
video_placeholder,
1652
1,
@@ -1652,20 +1662,20 @@ class Glm4vMultiModalProcessor(BaseMultiModalProcessor[Glm4vProcessingInfo]):
1662
video_outputs = dict()
1663
swap_video_frame_tokens = False
1664
1655
- processed_outputs = super()._call_hf_processor(
1656
- prompt=prompt,
1657
- mm_data=mm_data,
1658
- mm_kwargs=mm_kwargs,
1665
+ processed_data = self.info.ctx.call_hf_processor(
1666
+ self.info.get_hf_processor(**hf_processor_mm_kwargs),
1667
+ dict(text=prompt_text, **mm_data),
1668
+ hf_processor_mm_kwargs,
1669
)
1670
if swap_video_frame_tokens:
1661
- input_ids = processed_outputs["input_ids"]
1671
+ input_ids = processed_data["input_ids"]
1672
input_ids[input_ids == processor.video_token_id] = processor.image_token_id
1663
- processed_outputs["input_ids"] = input_ids
1664
- combined_outputs = dict(
1665
- processed_outputs,
1666
- **video_outputs,
1667
- )
1668
- return BatchFeature(combined_outputs)
1673
+ processed_data["input_ids"] = input_ids
1674
+
1675
+ processed_data.update(video_outputs)
1676
+ processed_data.update(passthrough_data)
1677
+
1678
+ return processed_data
1679
1680
def _get_mm_fields_config(
1681
self,
vllm/model_executor/models/glmasr.py
+37
-39
@@ -750,46 +750,40 @@ class GlmAsrMultiModalProcessor(BaseMultiModalProcessor["GlmAsrProcessingInfo"])
750
chunk_counts.append(min(n_chunks, max_windows))
751
return chunk_counts
752
753
- def _call_hf_processor(
753
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
754
+ return self.dummy_inputs.get_dummy_text(mm_counts)
755
+
756
+ def _preprocess_hf_mm_data(
757
self,
755
- prompt: str,
756
- mm_data: dict[str, object],
757
- mm_kwargs: Mapping[str, Any],
758
- ) -> BatchFeature:
759
- # Normalize input: handle deprecated key and list conversion.
758
+ mm_data: Mapping[str, object],
759
+ hf_processor_mm_kwargs: Mapping[str, object],
760
+ ) -> tuple[Mapping[str, object], Mapping[str, object]]:
761
+ mm_data = dict(mm_data)
762
if "audios" in mm_data:
763
mm_data["audio"] = mm_data.pop("audios")
764
763
- audio = mm_data.get("audio", [])
764
- audio_list = [audio] if audio and not isinstance(audio, list) else audio
765
-
766
- # Early return for text-only.
767
- if not audio_list:
768
- prompt_ids = self.info.get_tokenizer().encode(prompt)
769
- prompt_ids = self._apply_hf_processor_tokens_only(prompt_ids)
770
- return BatchFeature(dict(input_ids=[prompt_ids]), tensor_type="pt")
771
-
772
- # Handle sampling_rate
773
- feature_extractor = self.info.get_feature_extractor(**mm_kwargs)
774
- mm_kwargs = dict(
775
- **mm_kwargs,
765
+ feature_extractor = self.info.get_feature_extractor(**hf_processor_mm_kwargs)
766
+ hf_processor_mm_kwargs = dict(
767
+ **hf_processor_mm_kwargs,
768
sampling_rate=feature_extractor.sampling_rate,
769
)
770
779
- # Call parent method
780
- outputs = super()._call_hf_processor(
781
- prompt=prompt,
782
- mm_data=mm_data,
783
- mm_kwargs=mm_kwargs,
784
- )
771
+ return mm_data, hf_processor_mm_kwargs
772
786
- # Postprocess: rename mask and add chunk counts
773
+ def _postprocess_hf_mm_data(
774
+ self,
775
+ mm_data: Mapping[str, object],
776
+ hf_processor_mm_kwargs: Mapping[str, object],
777
+ processed_data: BatchFeature,
778
+ ) -> BatchFeature:
779
# Handle different key names from different transformers versions
788
- if "input_features_mask" in outputs:
789
- outputs["feature_attention_mask"] = outputs.pop("input_features_mask")
790
- elif "input_features_mask" not in outputs and "input_features" in outputs:
780
+ if "input_features_mask" in processed_data:
781
+ processed_data["feature_attention_mask"] = processed_data.pop(
782
+ "input_features_mask"
783
+ )
784
+ elif "input_features" in processed_data:
785
# If no mask is provided, create one from input_features
792
- input_features = outputs["input_features"]
786
+ input_features = processed_data["input_features"]
787
if isinstance(input_features, torch.Tensor):
788
# Create a mask of all ones matching the sequence length
789
mask = torch.ones(
@@ -797,18 +791,22 @@ class GlmAsrMultiModalProcessor(BaseMultiModalProcessor["GlmAsrProcessingInfo"])
791
input_features.shape[-1],
792
dtype=torch.long,
793
)
800
- outputs["feature_attention_mask"] = mask
794
+ processed_data["feature_attention_mask"] = mask
795
802
- # Get processor for chunk counts calculation
803
- processor = self.info.get_hf_processor(**mm_kwargs)
796
+ audio = mm_data.get("audio", [])
797
+ audio_list = [audio] if audio and not isinstance(audio, list) else audio
798
+ if audio_list:
799
+ processor = self.info.get_hf_processor(**hf_processor_mm_kwargs)
800
805
- # Override chunk counts calculation with GLM-ASR specific logic
806
- chunk_counts = self._calculate_chunk_counts(
807
- audio_list, processor.feature_extractor, processor
808
- )
809
- outputs["chunk_counts"] = torch.tensor(chunk_counts, dtype=torch.long)
801
+ # Override chunk counts calculation with GLM-ASR specific logic
802
+ chunk_counts = self._calculate_chunk_counts(
803
+ audio_list, processor.feature_extractor, processor
804
+ )
805
+ processed_data["chunk_counts"] = torch.tensor(
806
+ chunk_counts, dtype=torch.long
807
+ )
808
811
- return outputs
809
+ return processed_data
810
811
def _get_mm_fields_config(
812
self,
vllm/model_executor/models/granite_speech.py
+16
-12
@@ -180,12 +180,14 @@ class GraniteSpeechMultiModalProcessor(
180
)
181
]
182
183
- def _call_hf_processor(
183
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
184
+ return self.dummy_inputs.get_dummy_text(mm_counts)
185
+
186
+ def _preprocess_hf_mm_data(
187
self,
185
- prompt: str,
188
mm_data: Mapping[str, object],
187
- mm_kwargs: Mapping[str, object],
188
- ) -> BatchFeature:
189
+ hf_processor_mm_kwargs: Mapping[str, object],
190
+ ) -> tuple[Mapping[str, object], Mapping[str, object]]:
191
mm_data = dict(mm_data)
192
audios = mm_data.pop("audios", [])
193
@@ -193,21 +195,23 @@ class GraniteSpeechMultiModalProcessor(
195
# GraniteSpeechFeatureExtractor accepts "audio"
196
mm_data["audio"] = audios
197
196
- processed_outputs = super()._call_hf_processor(
197
- prompt=prompt,
198
- mm_data=mm_data,
199
- mm_kwargs=mm_kwargs,
200
- )
198
+ return mm_data, hf_processor_mm_kwargs
199
200
+ def _postprocess_hf_mm_data(
201
+ self,
202
+ mm_data: Mapping[str, object],
203
+ hf_processor_mm_kwargs: Mapping[str, object],
204
+ processed_data: BatchFeature,
205
+ ) -> BatchFeature:
206
if "audio" in mm_data:
207
# Calculate the number of audio tokens per entry in the batch;
208
# This is used to split the batch back out after padding.
209
audio_token_index = self.info.get_hf_config().audio_token_index
206
- processed_outputs["audio_embed_sizes"] = (
207
- processed_outputs["input_ids"] == audio_token_index
210
+ processed_data["audio_embed_sizes"] = (
211
+ processed_data["input_ids"] == audio_token_index
212
).sum(-1)
213
210
- return processed_outputs
214
+ return processed_data
215
216
217
class GraniteSpeechDummyInputsBuilder(
vllm/model_executor/models/h2ovl.py
+1
-1
@@ -135,7 +135,7 @@ class H2OVLMultiModalProcessor(BaseInternVLMultiModalProcessor[H2OVLProcessingIn
135
self,
136
inputs: ProcessorInputs,
137
timing_ctx: TimingContext,
138
- ) -> tuple[list[int], MultiModalProcessingInfo, bool]:
138
+ ) -> MultiModalProcessingInfo:
139
# The processor logic is different for len(images) <= 1 vs > 1
140
# Since the processing cache assumes that the processor output is
141
# invariant of how many images are passed per prompt, we only
vllm/model_executor/models/hunyuan_vision.py
-22
@@ -731,28 +731,6 @@ class HunYuanVLDummyInputsBuilder(BaseDummyInputsBuilder[HunYuanVLProcessingInfo
731
732
733
class HunYuanVLMultiModalProcessor(BaseMultiModalProcessor[HunYuanVLProcessingInfo]):
734
- def _call_hf_processor(
735
- self,
736
- prompt: str,
737
- mm_data: Mapping[str, object],
738
- mm_kwargs: Mapping[str, object],
739
- ) -> BatchFeature:
740
- hf_processor = self.info.get_hf_processor(**mm_kwargs)
741
- # HunYuanVLProcessor requires image placeholders wrapped with start/end tokens.
742
- if mm_data.get("images") is not None and prompt:
743
- img_tok = hf_processor.image_token
744
- wrapped = (
745
- f"{hf_processor.image_start_token}{img_tok}"
746
- f"{hf_processor.image_end_token}"
747
- )
748
- if img_tok in prompt and wrapped not in prompt:
749
- prompt = prompt.replace(img_tok, wrapped)
750
- return self.info.ctx.call_hf_processor(
751
- hf_processor,
752
- dict(text=prompt, **mm_data),
753
- mm_kwargs,
754
- )
755
-
734
def _get_prompt_updates(
735
self,
736
mm_items: MultiModalDataItems,
vllm/model_executor/models/hyperclovax_vision.py
+49
-55
@@ -190,77 +190,71 @@ class HCXVisionDummyInputsBuilder(BaseDummyInputsBuilder[HCXVisionProcessingInfo
190
191
192
class HCXVisionMultiModalProcessor(BaseMultiModalProcessor[HCXVisionProcessingInfo]):
193
- def _call_hf_processor(
193
+ def _apply_hf_processor_main(
194
self,
195
- prompt: str,
196
- mm_data: Mapping[str, object],
197
- mm_kwargs: Mapping[str, object],
195
+ mm_items: MultiModalDataItems,
196
+ hf_processor_mm_kwargs: Mapping[str, object],
197
) -> BatchFeature:
198
+ valid_mm_items = mm_items.select(
199
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
200
+ )
201
+ mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
202
+
203
+ if not mm_data:
204
+ return BatchFeature(dict(passthrough_data))
205
+
206
for video_idx, video_arr in enumerate(mm_data.get("videos", [])):
207
if video_arr.dtype != np.uint8:
208
mm_data["videos"][video_idx] = video_arr.astype(np.uint8)
209
203
- processed_outputs = self.info.ctx.call_hf_processor(
204
- hf_processor=self.info.get_hf_processor(**mm_kwargs),
210
+ images = mm_data.get("images")
211
+ videos = mm_data.get("videos")
212
+
213
+ # batchify input as a single item
214
+ processed_data = self.info.ctx.call_hf_processor(
215
+ hf_processor=self.info.get_hf_processor(**hf_processor_mm_kwargs),
216
data=dict(
206
- text=prompt,
207
- images=None,
208
- videos=None,
217
+ images=None if images is None else [images],
218
+ videos=None if videos is None else [videos],
219
),
210
- ) # text-only
211
-
212
- if len(mm_data) > 0:
213
- images = mm_data.get("images")
214
- videos = mm_data.get("videos")
215
-
216
- # batchify input as a single item
217
- _processed_outputs = self.info.ctx.call_hf_processor(
218
- hf_processor=self.info.get_hf_processor(**mm_kwargs),
219
- data=dict(
220
- text=None,
221
- images=None if images is None else [images],
222
- videos=None if videos is None else [videos],
223
- ),
224
- ) # mm-only
220
+ )
221
226
- for k, v in _processed_outputs.items():
227
- if isinstance(v, list) and len(v) > 0:
228
- assert len(v) == 1
229
- _processed_outputs[k] = v[0]
222
+ for k, v in processed_data.items():
223
+ if isinstance(v, list) and len(v) > 0:
224
+ assert len(v) == 1
225
+ processed_data[k] = v[0]
226
231
- if images:
232
- _processed_outputs["image_sizes_images"] = torch.tensor(
233
- _processed_outputs["image_sizes_images"]
234
- )
235
- _processed_outputs["vision_query_lengths_images"] = torch.tensor(
236
- _processed_outputs["vision_query_lengths_images"]
237
- )
227
+ if images:
228
+ processed_data["image_sizes_images"] = torch.tensor(
229
+ processed_data["image_sizes_images"]
230
+ )
231
+ processed_data["vision_query_lengths_images"] = torch.tensor(
232
+ processed_data["vision_query_lengths_images"]
233
+ )
234
239
- if videos:
240
- _idx_per_video = [
241
- 0,
242
- *accumulate(
243
- get_num_combined_frames(len(video)) for video in videos
244
- ),
235
+ if videos:
236
+ _idx_per_video = [
237
+ 0,
238
+ *accumulate(get_num_combined_frames(len(video)) for video in videos),
239
+ ]
240
+ processed_data["pixel_values_videos"] = [
241
+ processed_data["pixel_values_videos"][
242
+ _idx_per_video[i] : _idx_per_video[i + 1]
243
]
246
- _processed_outputs["pixel_values_videos"] = [
247
- _processed_outputs["pixel_values_videos"][
244
+ for i in range(len(videos))
245
+ ]
246
+ processed_data["vision_query_lengths_videos"] = [
247
+ torch.tensor(
248
+ processed_data["vision_query_lengths_videos"][
249
_idx_per_video[i] : _idx_per_video[i + 1]
250
]
250
- for i in range(len(videos))
251
- ]
252
- _processed_outputs["vision_query_lengths_videos"] = [
253
- torch.tensor(
254
- _processed_outputs["vision_query_lengths_videos"][
255
- _idx_per_video[i] : _idx_per_video[i + 1]
256
- ]
257
- )
258
- for i in range(len(videos))
259
- ]
251
+ )
252
+ for i in range(len(videos))
253
+ ]
254
261
- processed_outputs.update(_processed_outputs)
255
+ processed_data.update(passthrough_data)
256
263
- return processed_outputs
257
+ return processed_data
258
259
def _get_prompt_updates(
260
self,
vllm/model_executor/models/hyperclovax_vision_v2.py
+19
-9
@@ -261,34 +261,44 @@ class HCXVisionV2MultiModalProcessor(
261
):
262
"""Multimodal processor for HyperCLOVAX V2 (32B Think model)."""
263
264
- def _call_hf_processor(
264
+ def _apply_hf_processor_main(
265
self,
266
- prompt: str,
267
- mm_data: Mapping[str, object],
268
- mm_kwargs: Mapping[str, object],
266
+ mm_items: MultiModalDataItems,
267
+ hf_processor_mm_kwargs: Mapping[str, object],
268
) -> BatchFeature:
269
+ valid_mm_items = mm_items.select(
270
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
271
+ )
272
+ mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
273
+
274
+ if not mm_data:
275
+ return BatchFeature(dict(passthrough_data))
276
+
277
+ prompt_text = self.dummy_inputs.get_dummy_text(mm_items.get_all_counts())
278
+
279
images = mm_data.get("images")
280
videos = mm_data.get("videos")
281
282
# Get the HF processor
274
- hf_processor = self.info.get_hf_processor(**mm_kwargs)
283
+ hf_processor = self.info.get_hf_processor(**hf_processor_mm_kwargs)
284
285
# Build data dict for HF processor (images/videos only)
286
# NOTE: We pass the prompt as-is without token normalization.
287
# Token expansion is handled by vLLM via _get_prompt_updates.
288
data: dict[str, object] = dict(
280
- text=prompt,
289
+ text=prompt_text,
290
images=images,
291
videos=videos,
292
)
293
285
- processed_outputs = self.info.ctx.call_hf_processor(
294
+ processed_data = self.info.ctx.call_hf_processor(
295
hf_processor=hf_processor,
296
data=data,
288
- kwargs=mm_kwargs,
297
+ kwargs=hf_processor_mm_kwargs,
298
)
299
+ processed_data.update(passthrough_data)
300
291
- return processed_outputs
301
+ return processed_data
302
303
def _get_prompt_updates(
304
self,
vllm/model_executor/models/idefics3.py
+26
-19
@@ -239,51 +239,58 @@ class Idefics3DummyInputsBuilder(BaseDummyInputsBuilder[Idefics3ProcessingInfo])
239
240
241
class Idefics3MultiModalProcessor(BaseMultiModalProcessor[Idefics3ProcessingInfo]):
242
- def _call_hf_processor(
242
+ def _apply_hf_processor_main(
243
self,
244
- prompt: str,
245
- mm_data: Mapping[str, object],
246
- mm_kwargs: Mapping[str, object],
244
+ mm_items: MultiModalDataItems,
245
+ hf_processor_mm_kwargs: Mapping[str, object],
246
) -> BatchFeature:
248
- # Text-only input not supported in composite processor
249
- if not (images := mm_data.get("images", [])):
250
- prompt_ids = self.info.get_tokenizer().encode(prompt)
251
- prompt_ids = self._apply_hf_processor_tokens_only(prompt_ids)
252
- return BatchFeature(dict(input_ids=[prompt_ids]), tensor_type="pt")
247
+ valid_mm_items = mm_items.select(
248
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
249
+ )
250
+ mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
251
+
252
+ if not mm_data:
253
+ return BatchFeature(dict(passthrough_data))
254
255
image_processor = self.info.get_hf_processor().image_processor
256
if getattr(image_processor, "backend", "pil") == "pil":
256
- mm_kwargs = {"input_data_format": "channels_last", **mm_kwargs}
257
+ hf_processor_mm_kwargs = {
258
+ "input_data_format": "channels_last",
259
+ **hf_processor_mm_kwargs,
260
+ }
261
258
- processed_outputs = super()._call_hf_processor(
259
- prompt,
262
+ processed_data = self.info.ctx.call_hf_processor(
263
+ self.info.get_hf_processor(**hf_processor_mm_kwargs),
264
mm_data,
261
- mm_kwargs,
265
+ hf_processor_mm_kwargs,
266
)
267
268
+ images = mm_data.get("images", [])
269
mm_items = self.info.parse_mm_data({"image": images}, validate=False)
270
parsed_images = mm_items.get_items("image", ImageProcessorItems)
271
image_sizes = [
272
parsed_images.get_image_size(i) for i in range(len(parsed_images))
273
]
269
- hf_processor = self.info.get_hf_processor(**mm_kwargs)
274
+ hf_processor = self.info.get_hf_processor(**hf_processor_mm_kwargs)
275
276
num_patches = [
277
self.info.get_num_patches(
278
image_width=size.width,
279
image_height=size.height,
280
processor=hf_processor,
276
- mm_kwargs=mm_kwargs,
281
+ mm_kwargs=hf_processor_mm_kwargs,
282
)
283
for size in image_sizes
284
]
280
- processed_outputs["num_patches"] = torch.tensor(num_patches)
285
+ processed_data["num_patches"] = torch.tensor(num_patches)
286
287
# Remove the extra batch dimension
283
- processed_outputs["pixel_values"].squeeze_(0)
284
- processed_outputs["pixel_attention_mask"].squeeze_(0)
288
+ processed_data["pixel_values"].squeeze_(0)
289
+ processed_data["pixel_attention_mask"].squeeze_(0)
290
+
291
+ processed_data.update(passthrough_data)
292
286
- return processed_outputs
293
+ return processed_data
294
295
def _get_mm_fields_config(
296
self,
vllm/model_executor/models/interns1.py
+51
-25
@@ -332,19 +332,28 @@ class InternS1DummyInputsBuilder(BaseDummyInputsBuilder[InternS1ProcessingInfo])
332
class InternS1MultiModalProcessor(BaseMultiModalProcessor[InternS1ProcessingInfo]):
333
"""Basic image-only MultiModalProcessor for InternS1-style models."""
334
335
- def _call_hf_processor(
335
+ def _apply_hf_processor_main(
336
self,
337
- prompt: str,
338
- mm_data: Mapping[str, object],
339
- mm_kwargs: Mapping[str, object],
337
+ mm_items: MultiModalDataItems,
338
+ hf_processor_mm_kwargs: Mapping[str, object],
339
) -> BatchFeature:
340
+ valid_mm_items = mm_items.select(
341
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
342
+ )
343
+ mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
344
+
345
+ if not mm_data:
346
+ return BatchFeature(dict(passthrough_data))
347
+
348
+ prompt_text = self.dummy_inputs.get_dummy_text(mm_items.get_all_counts())
349
+
350
mm_data = dict(mm_data)
351
videos = mm_data.pop("videos", [])
352
images = mm_data.pop("images", [])
353
assert isinstance(videos, list)
354
assert isinstance(images, list)
355
347
- hf_processor = self.info.get_hf_processor(**mm_kwargs)
356
+ hf_processor = self.info.get_hf_processor(**hf_processor_mm_kwargs)
357
tokenizer = hf_processor.tokenizer
358
video_token_id = tokenizer.encode(
359
hf_processor.video_token, add_special_tokens=False
@@ -352,23 +361,29 @@ class InternS1MultiModalProcessor(BaseMultiModalProcessor[InternS1ProcessingInfo
361
assert len(video_token_id) == 1
362
video_token_id = video_token_id[0]
363
355
- prompt = re.sub(hf_processor.image_token, "<image_placeholder>", prompt)
356
- prompt = re.sub(hf_processor.video_token, "<video_placeholder>", prompt)
364
+ prompt_text = re.sub(
365
+ hf_processor.image_token, "<image_placeholder>", prompt_text
366
+ )
367
+ prompt_text = re.sub(
368
+ hf_processor.video_token, "<video_placeholder>", prompt_text
369
+ )
370
371
image_outputs = {}
372
if images:
373
image_pixel_values = []
374
for image in images:
362
- processed_outputs = super()._call_hf_processor(
363
- prompt=hf_processor.image_token,
364
- mm_data={"images": image},
365
- mm_kwargs=mm_kwargs,
375
+ processed_data = self.info.ctx.call_hf_processor(
376
+ self.info.get_hf_processor(**hf_processor_mm_kwargs),
377
+ dict(text=hf_processor.image_token, **{"images": image}),
378
+ hf_processor_mm_kwargs,
379
)
367
- image_pixel_values.append(processed_outputs.pop("pixel_values"))
380
+ image_pixel_values.append(processed_data.pop("pixel_values"))
381
369
- input_ids = processed_outputs.pop("input_ids")
382
+ input_ids = processed_data.pop("input_ids")
383
image_placeholder = tokenizer.batch_decode(input_ids)[0]
371
- prompt = prompt.replace("<image_placeholder>", image_placeholder, 1)
384
+ prompt_text = prompt_text.replace(
385
+ "<image_placeholder>", image_placeholder, 1
386
+ )
387
388
num_patches = [len(item) for item in image_pixel_values]
389
image_outputs = {
@@ -381,18 +396,20 @@ class InternS1MultiModalProcessor(BaseMultiModalProcessor[InternS1ProcessingInfo
396
if videos:
397
video_pixel_values = []
398
for video in videos:
384
- processed_outputs = super()._call_hf_processor(
385
- prompt=hf_processor.video_token,
386
- mm_data={"videos": video},
387
- mm_kwargs=mm_kwargs,
399
+ processed_data = self.info.ctx.call_hf_processor(
400
+ self.info.get_hf_processor(**hf_processor_mm_kwargs),
401
+ dict(text=hf_processor.video_token, **{"videos": video}),
402
+ hf_processor_mm_kwargs,
403
)
389
- video_pixel_values.append(processed_outputs.pop("pixel_values"))
404
+ video_pixel_values.append(processed_data.pop("pixel_values"))
405
391
- input_ids = processed_outputs.pop("input_ids")
406
+ input_ids = processed_data.pop("input_ids")
407
input_ids[input_ids == hf_processor.image_token_id] = video_token_id
408
409
video_placeholder = tokenizer.batch_decode(input_ids)[0]
395
- prompt = prompt.replace("<video_placeholder>", video_placeholder, 1)
410
+ prompt_text = prompt_text.replace(
411
+ "<video_placeholder>", video_placeholder, 1
412
+ )
413
414
num_frames = [len(item) for item in video_pixel_values]
415
video_outputs = {
@@ -401,11 +418,20 @@ class InternS1MultiModalProcessor(BaseMultiModalProcessor[InternS1ProcessingInfo
418
"video_token_id": torch.tensor(video_token_id),
419
}
420
404
- prompt = re.sub("<image_placeholder>", hf_processor.image_token, prompt)
405
- prompt = re.sub("<video_placeholder>", hf_processor.video_token, prompt)
406
- text_outputs = tokenizer(prompt, return_tensors="pt")
421
+ prompt_text = re.sub(
422
+ "<image_placeholder>", hf_processor.image_token, prompt_text
423
+ )
424
+ prompt_text = re.sub(
425
+ "<video_placeholder>", hf_processor.video_token, prompt_text
426
+ )
427
+ text_outputs = tokenizer(prompt_text, return_tensors="pt")
428
+
429
+ processed_data.update(text_outputs)
430
+ processed_data.update(image_outputs)
431
+ processed_data.update(video_outputs)
432
+ processed_data.update(passthrough_data)
433
408
- return BatchFeature({**text_outputs, **image_outputs, **video_outputs})
434
+ return processed_data
435
436
def _get_mm_fields_config(
437
self,
vllm/model_executor/models/internvl.py
+17
-19
@@ -220,27 +220,21 @@ class BaseInternVLDummyInputsBuilder(BaseDummyInputsBuilder[_I]):
220
class BaseInternVLMultiModalProcessor(BaseMultiModalProcessor[_I]):
221
"""Basic image-only MultiModalProcessor for InternVL-style models."""
222
223
- def _call_hf_processor(
223
+ def _postprocess_hf_mm_data(
224
self,
225
- prompt: str,
225
mm_data: Mapping[str, object],
227
- mm_kwargs: Mapping[str, object],
226
+ hf_processor_mm_kwargs: Mapping[str, object],
227
+ processed_data: BatchFeature,
228
) -> BatchFeature:
229
- processed_outputs = super()._call_hf_processor(
230
- prompt=prompt,
231
- mm_data=mm_data,
232
- mm_kwargs=mm_kwargs,
233
- )
234
-
235
- hf_processor = self.info.get_hf_processor(**mm_kwargs)
229
+ hf_processor = self.info.get_hf_processor(**hf_processor_mm_kwargs)
230
image_token_id = hf_processor.ctx_image_token_id
231
232
# Since there may be extra tokens in the feature placeholders,
233
# we need to pass the image token ID to the model to select the
234
# tokens to merge from the vision encoder outputs
241
- processed_outputs["image_token_id"] = torch.tensor(image_token_id)
235
+ processed_data["image_token_id"] = torch.tensor(image_token_id)
236
243
- return processed_outputs
237
+ return processed_data
238
239
def _get_image_fields_config(self, hf_inputs: BatchFeature):
240
image_num_patches = hf_inputs.get("image_num_patches", torch.empty(0))
@@ -455,19 +449,23 @@ class InternVLMultiModalProcessor(
449
):
450
"""InternVL MultiModalProcessor extended for video support"""
451
458
- def _call_hf_processor(
452
+ def _postprocess_hf_mm_data(
453
self,
460
- prompt: str,
454
mm_data: Mapping[str, object],
462
- mm_kwargs: Mapping[str, object],
455
+ hf_processor_mm_kwargs: Mapping[str, object],
456
+ processed_data: BatchFeature,
457
) -> BatchFeature:
464
- processed_outputs = super()._call_hf_processor(prompt, mm_data, mm_kwargs)
458
+ processed_data = super()._postprocess_hf_mm_data(
459
+ mm_data,
460
+ hf_processor_mm_kwargs,
461
+ processed_data,
462
+ )
463
466
- hf_processor = self.info.get_hf_processor(**mm_kwargs)
464
+ hf_processor = self.info.get_hf_processor(**hf_processor_mm_kwargs)
465
if (video_token_id := hf_processor.ctx_video_token_id) is not None:
468
- processed_outputs["video_token_id"] = torch.tensor(video_token_id)
466
+ processed_data["video_token_id"] = torch.tensor(video_token_id)
467
470
- return processed_outputs
468
+ return processed_data
469
470
def _get_video_fields_config(self, hf_inputs: BatchFeature):
471
video_num_patches = hf_inputs.get("video_num_patches", torch.empty(0))
vllm/model_executor/models/jina_vl.py
+19
-9
@@ -12,6 +12,7 @@ from vllm.logger import init_logger
12
from vllm.model_executor.layers.linear import ColumnParallelLinear, RowParallelLinear
13
from vllm.model_executor.layers.pooler import DispatchPooler
14
from vllm.multimodal import MULTIMODAL_REGISTRY
15
+from vllm.multimodal.parse import MultiModalDataItems
16
from vllm.sequence import IntermediateTensors
17
18
from .interfaces import SupportsCrossEncoding, SupportsMultiModal, SupportsScoreTemplate
@@ -54,19 +55,28 @@ class JinaVLScorer(nn.Module):
55
56
57
class JinaVLMultiModalProcessor(Qwen2VLMultiModalProcessor):
57
- def _call_hf_processor(
58
+ def _apply_hf_processor_main(
59
self,
59
- prompt: str,
60
- mm_data: Mapping[str, object],
61
- mm_kwargs: Mapping[str, object],
60
+ mm_items: MultiModalDataItems,
61
+ hf_processor_mm_kwargs: Mapping[str, object],
62
) -> BatchFeature:
63
- # NOTE: We should reverse the order of the mm_data because the
64
- # query prompt is placed after the document prompt in the score
65
- # template for JinaVLForRanking model, but in mm_data they are
66
- # stored in the opposite order (query first, then document).
63
+ valid_mm_items = mm_items.select(
64
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
65
+ )
66
+ mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
67
+
68
+ if not mm_data:
69
+ return BatchFeature(dict(passthrough_data))
70
+
71
for _, value in mm_data.items():
72
value.reverse()
69
- return super()._call_hf_processor(prompt, mm_data, mm_kwargs)
73
+ processed_data = self.info.ctx.call_hf_processor(
74
+ self.info.get_hf_processor(**hf_processor_mm_kwargs),
75
+ mm_data,
76
+ hf_processor_mm_kwargs,
77
+ )
78
+ processed_data.update(passthrough_data)
79
+ return processed_data
80
81
82
@MULTIMODAL_REGISTRY.register_processor(
vllm/model_executor/models/kanana_v.py
+15
-9
@@ -461,17 +461,21 @@ class KananaVMultiModalProcessor(BaseMultiModalProcessor[KananaVProcessingInfo])
461
def media_token_id(self) -> int:
462
return self.info.get_hf_config().text_config.eos_token_id + 1
463
464
- def _call_hf_processor(
464
+ def _apply_hf_processor_main(
465
self,
466
- prompt: str,
467
- mm_data: Mapping[str, object],
468
- mm_kwargs: Mapping[str, object],
466
+ mm_items: MultiModalDataItems,
467
+ hf_processor_mm_kwargs: Mapping[str, object],
468
) -> BatchFeature:
469
"""Run the underlying HF processor on text and image data."""
471
- # Text-only input is handled as a special case here.
470
+ valid_mm_items = mm_items.select(
471
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
472
+ )
473
+ mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
474
+
475
if not mm_data or not mm_data.get("images", []):
473
- prompt_ids = self.info.get_tokenizer().encode(prompt)
474
- return BatchFeature(dict(input_ids=[prompt_ids]), tensor_type="pt")
476
+ return BatchFeature(dict(passthrough_data))
477
+
478
+ prompt_text = self.dummy_inputs.get_dummy_text(mm_items.get_all_counts())
479
480
# Images
481
image_inputs = mm_data.get("images", [])
@@ -493,7 +497,7 @@ class KananaVMultiModalProcessor(BaseMultiModalProcessor[KananaVProcessingInfo])
497
498
tokenizer = self.info.get_tokenizer()
499
media_token = tokenizer.convert_ids_to_tokens([self.media_token_id])[0]
496
- prompt_replaced = prompt.replace("<image>", media_token)
500
+ prompt_replaced = prompt_text.replace("<image>", media_token)
501
input_ids = tokenizer.encode(prompt_replaced)
502
input_ids = torch.tensor(input_ids)
503
@@ -527,7 +531,9 @@ class KananaVMultiModalProcessor(BaseMultiModalProcessor[KananaVProcessingInfo])
531
image_token_thw=torch.tensor(image_meta["image_token_thw"]),
532
pixel_sizes=torch.tensor(pixel_sizes),
533
)
530
- return BatchFeature(combined_outputs, tensor_type="pt")
534
+ processed_data = BatchFeature(combined_outputs, tensor_type="pt")
535
+ processed_data.update(passthrough_data)
536
+ return processed_data
537
538
def _get_prompt_updates(
539
self,
vllm/model_executor/models/keye.py
+2
-9
@@ -1162,15 +1162,8 @@ class KeyeDummyInputsBuilder(KeyeBaseDummyInputsBuilder[KeyeProcessingInfo]):
1162
1163
1164
class KeyeMultiModalProcessor(BaseMultiModalProcessor[KeyeProcessingInfo]):
1165
- def _call_hf_processor(
1166
- self,
1167
- prompt: str,
1168
- mm_data: Mapping[str, object],
1169
- mm_kwargs: Mapping[str, object],
1170
- ) -> BatchFeature:
1171
- # Override to use the text path instead of token path to use the
1172
- # video-specific logic in processing_keye.py
1173
- return super()._call_hf_processor(prompt, mm_data, mm_kwargs)
1165
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
1166
+ return self.dummy_inputs.get_dummy_text(mm_counts)
1167
1168
def _get_prompt_updates(
1169
self,
vllm/model_executor/models/keye_vl1_5.py
+2
-9
@@ -378,15 +378,8 @@ class KeyeVL1_5ProcessingInfo(KeyeProcessingInfo):
378
379
380
class KeyeVL1_5MultiModalProcessor(BaseMultiModalProcessor[KeyeVL1_5ProcessingInfo]):
381
- def _call_hf_processor(
382
- self,
383
- prompt: str,
384
- mm_data: Mapping[str, object],
385
- mm_kwargs: Mapping[str, object],
386
- ) -> BatchFeature:
387
- # Override to use the text path instead of token path to use the
388
- # video-specific logic in processing_keye.py
389
- return super()._call_hf_processor(prompt, mm_data, mm_kwargs)
381
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
382
+ return self.dummy_inputs.get_dummy_text(mm_counts)
383
384
def _get_prompt_updates(
385
self,
vllm/model_executor/models/kimi_audio.py
+18
-8
@@ -37,6 +37,7 @@ from vllm.multimodal.parse import (
37
DictEmbeddingItems,
38
ModalityData,
39
ModalityDataItems,
40
+ MultiModalDataItems,
41
MultiModalDataParser,
42
)
43
from vllm.multimodal.processing import (
@@ -226,13 +227,20 @@ class KimiAudioMultiModalDataParser(MultiModalDataParser):
227
class KimiAudioMultiModalProcessor(BaseMultiModalProcessor[KimiAudioProcessingInfo]):
228
"""vLLM multi-modal processor wrapper for Kimi-Audio."""
229
229
- def _call_hf_processor(
230
+ def _apply_hf_processor_main(
231
self,
231
- prompt: str,
232
- mm_data: Mapping[str, object],
233
- mm_kwargs: Mapping[str, object],
232
+ mm_items: MultiModalDataItems,
233
+ hf_processor_mm_kwargs: Mapping[str, object],
234
) -> BatchFeature:
235
"""Call the HuggingFace processor."""
236
+ valid_mm_items = mm_items.select(
237
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
238
+ )
239
+ mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
240
+
241
+ if not mm_data:
242
+ return BatchFeature(dict(passthrough_data))
243
+
244
# Convert mm_data format: {'audios': [...]} -> {'audio': ...}
245
mm_data = dict(mm_data)
246
audios = mm_data.pop("audios", [])
@@ -252,11 +260,13 @@ class KimiAudioMultiModalProcessor(BaseMultiModalProcessor[KimiAudioProcessingIn
260
mm_data["audio"] = audio_arrays
261
262
# Use the context's call_hf_processor for proper handling
255
- return self.info.ctx.call_hf_processor(
256
- self.info.get_hf_processor(**mm_kwargs),
257
- dict(text=prompt, **mm_data),
258
- mm_kwargs,
263
+ processed_data = self.info.ctx.call_hf_processor(
264
+ self.info.get_hf_processor(**hf_processor_mm_kwargs),
265
+ mm_data,
266
+ hf_processor_mm_kwargs,
267
)
268
+ processed_data.update(passthrough_data)
269
+ return processed_data
270
271
def _get_mm_fields_config(
272
self,
vllm/model_executor/models/kimi_k25.py
-10
@@ -258,16 +258,6 @@ class KimiK25MultiModalProcessor(BaseMultiModalProcessor[KimiK25ProcessingInfo])
258
grid_thws=MultiModalFieldConfig.batched("vision_chunk", keep_on_cpu=True),
259
)
260
261
- def _call_hf_processor(
262
- self,
263
- prompt: str,
264
- mm_data: Mapping[str, object],
265
- mm_kwargs: Mapping[str, object],
266
- ) -> BatchFeature:
267
- # Override to use the text path instead of token path because vision chunk
268
- # is not considered
269
- return super()._call_hf_processor(prompt, mm_data, mm_kwargs)
270
-
261
def _get_prompt_updates(
262
self,
263
mm_items: MultiModalDataItems,
vllm/model_executor/models/kimi_vl.py
+3
@@ -237,6 +237,9 @@ class KimiVLDummyInputsBuilder(BaseDummyInputsBuilder[KimiVLProcessingInfo]):
237
238
239
class KimiVLMultiModalProcessor(BaseMultiModalProcessor[KimiVLProcessingInfo]):
240
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
241
+ return self.dummy_inputs.get_dummy_text(mm_counts)
242
+
243
def _get_mm_fields_config(
244
self,
245
hf_inputs: BatchFeature,
vllm/model_executor/models/lfm2_vl.py
+13
-20
@@ -393,45 +393,38 @@ class Lfm2VLDummyInputsBuilder(BaseDummyInputsBuilder[Lfm2VLProcessingInfo]):
393
394
395
class Lfm2VLMultiModalProcessor(BaseMultiModalProcessor[Lfm2VLProcessingInfo]):
396
- def _call_hf_processor(
396
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
397
+ return self.dummy_inputs.get_dummy_text(mm_counts)
398
+
399
+ def _postprocess_hf_mm_data(
400
self,
398
- prompt: str,
401
mm_data: Mapping[str, object],
400
- mm_kwargs: Mapping[str, object],
402
+ hf_processor_mm_kwargs: Mapping[str, object],
403
+ processed_data: BatchFeature,
404
) -> BatchFeature:
402
- # Text-only input not supported in composite processor
403
- if not (images := mm_data.get("images", [])):
404
- prompt_ids = self.info.get_tokenizer().encode(
405
- prompt, add_special_tokens=False
406
- )
407
- prompt_ids = self._apply_hf_processor_tokens_only(prompt_ids)
408
- return BatchFeature(dict(input_ids=[prompt_ids]), tensor_type="pt")
409
-
410
- processed_outputs = super()._call_hf_processor(
411
- prompt,
412
- mm_data,
413
- mm_kwargs,
414
- )
405
+ if not mm_data:
406
+ return processed_data
407
408
+ images = mm_data.get("images", [])
409
mm_items = self.info.parse_mm_data({"image": images}, validate=False)
410
parsed_images = mm_items.get_items("image", ImageProcessorItems)
411
image_sizes = [
412
parsed_images.get_image_size(i) for i in range(len(parsed_images))
413
]
421
- hf_processor = self.info.get_hf_processor(**mm_kwargs)
414
+ hf_processor = self.info.get_hf_processor(**hf_processor_mm_kwargs)
415
416
num_patches = [
417
self.info.get_num_patches(
418
image_width=size.width,
419
image_height=size.height,
420
processor=hf_processor,
428
- mm_kwargs=mm_kwargs,
421
+ mm_kwargs=hf_processor_mm_kwargs,
422
)
423
for size in image_sizes
424
]
432
- processed_outputs["num_patches"] = torch.tensor(num_patches)
425
+ processed_data["num_patches"] = torch.tensor(num_patches)
426
434
- return processed_outputs
427
+ return processed_data
428
429
def _get_mm_fields_config(
430
self,
vllm/model_executor/models/lightonocr.py
+18
-19
@@ -39,20 +39,19 @@ _I = TypeVar("_I", bound=Mistral3ProcessingInfo)
39
40
41
class LightOnOCRMultiModalProcessor(BaseMultiModalProcessor[Mistral3ProcessingInfo]):
42
- def _call_hf_processor(
42
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
43
+ return self.dummy_inputs.get_dummy_text(mm_counts)
44
+
45
+ def _postprocess_hf_mm_data(
46
self,
44
- prompt: str,
47
mm_data: Mapping[str, object],
46
- mm_kwargs: Mapping[str, object],
48
+ hf_processor_mm_kwargs: Mapping[str, object],
49
+ processed_data: BatchFeature,
50
) -> BatchFeature:
48
- processed_outputs = super()._call_hf_processor(
49
- prompt=prompt,
50
- mm_data=mm_data,
51
- mm_kwargs=mm_kwargs,
52
- )
51
+ if not mm_data:
52
+ return processed_data
53
54
- # NOTE: LightOnOCR does not use break/end tokens, so we remove them here.
55
- input_ids = processed_outputs.get("input_ids")
54
+ input_ids = processed_data.get("input_ids")
55
if input_ids is not None:
56
processor = self.info.get_hf_processor()
57
tokenizer = self.info.get_tokenizer()
@@ -67,22 +66,22 @@ class LightOnOCRMultiModalProcessor(BaseMultiModalProcessor[Mistral3ProcessingIn
66
torch.tensor([break_id, end_id]),
67
)
68
70
- processed_outputs["input_ids"] = input_ids[keep_mask].unsqueeze(0)
71
- if "attention_mask" in processed_outputs:
72
- processed_outputs["attention_mask"] = processed_outputs[
73
- "attention_mask"
74
- ][keep_mask].unsqueeze(0)
69
+ processed_data["input_ids"] = input_ids[keep_mask].unsqueeze(0)
70
+ if "attention_mask" in processed_data:
71
+ processed_data["attention_mask"] = processed_data["attention_mask"][
72
+ keep_mask
73
+ ].unsqueeze(0)
74
75
# un-pad pixel_values per-image so caches remain independent.
77
- pixel_values = processed_outputs.get("pixel_values")
76
+ pixel_values = processed_data.get("pixel_values")
77
if pixel_values is not None:
79
- image_sizes = processed_outputs["image_sizes"]
78
+ image_sizes = processed_data["image_sizes"]
79
assert len(pixel_values) == len(image_sizes)
81
- processed_outputs["pixel_values"] = [
80
+ processed_data["pixel_values"] = [
81
p[:, :h, :w] for p, (h, w) in zip(pixel_values, image_sizes)
82
]
83
85
- return processed_outputs
84
+ return processed_data
85
86
def _get_mm_fields_config(
87
self,
vllm/model_executor/models/llava.py
+12
-12
@@ -321,30 +321,30 @@ class PixtralHFProcessingInfo(BaseLlavaProcessingInfo):
321
322
323
class PixtralHFMultiModalProcessor(BaseMultiModalProcessor[PixtralHFProcessingInfo]):
324
- def _call_hf_processor(
324
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
325
+ return self.dummy_inputs.get_dummy_text(mm_counts)
326
+
327
+ def _postprocess_hf_mm_data(
328
self,
326
- prompt: str,
329
mm_data: Mapping[str, object],
328
- mm_kwargs: Mapping[str, object],
330
+ hf_processor_mm_kwargs: Mapping[str, object],
331
+ processed_data: BatchFeature,
332
) -> BatchFeature:
330
- processed_outputs = super()._call_hf_processor(
331
- prompt=prompt,
332
- mm_data=mm_data,
333
- mm_kwargs=mm_kwargs,
334
- )
333
+ if not mm_data:
334
+ return processed_data
335
336
- pixel_values = processed_outputs.get("pixel_values")
336
+ pixel_values = processed_data.get("pixel_values")
337
if pixel_values is not None:
338
# Avoid padding since we need the output for each image to be
339
# independent of other images for the cache to work correctly
340
- image_sizes = processed_outputs["image_sizes"]
340
+ image_sizes = processed_data["image_sizes"]
341
assert len(pixel_values) == len(image_sizes)
342
343
- processed_outputs["pixel_values"] = [
343
+ processed_data["pixel_values"] = [
344
p[:, :h, :w] for p, (h, w) in zip(pixel_values, image_sizes)
345
]
346
347
- return processed_outputs
347
+ return processed_data
348
349
def _get_mm_fields_config(
350
self,
vllm/model_executor/models/llava_next.py
+3
@@ -211,6 +211,9 @@ class BaseLlavaNextMultiModalProcessor(BaseLlavaMultiModalProcessor[_I]):
211
212
213
class LlavaNextMultiModalProcessor(BaseLlavaNextMultiModalProcessor[_I]):
214
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
215
+ return self.dummy_inputs.get_dummy_text(mm_counts)
216
+
217
def _get_mm_fields_config(
218
self,
219
hf_inputs: BatchFeature,
vllm/model_executor/models/llava_onevision.py
+31
-24
@@ -321,22 +321,33 @@ class LlavaOnevisionMultiModalProcessor(
321
pixel_values_videos=MultiModalFieldConfig.batched("video"),
322
)
323
324
- def _call_hf_processor(
324
+ def _apply_hf_processor_main(
325
self,
326
- prompt: str,
327
- mm_data: Mapping[str, object],
328
- mm_kwargs: Mapping[str, object],
326
+ mm_items: MultiModalDataItems,
327
+ hf_processor_mm_kwargs: Mapping[str, object],
328
) -> BatchFeature:
329
+ valid_mm_items = mm_items.select(
330
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
331
+ )
332
+ mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
333
+
334
+ if not mm_data:
335
+ return BatchFeature(dict(passthrough_data))
336
+
337
+ prompt_text = self.dummy_inputs.get_dummy_text(mm_items.get_all_counts())
338
+
339
mm_data = dict(mm_data)
340
videos = mm_data.pop("videos", [])
341
assert isinstance(videos, list)
342
343
if not videos:
335
- return super()._call_hf_processor(
336
- prompt=prompt,
337
- mm_data=mm_data,
338
- mm_kwargs=mm_kwargs,
344
+ processed_data = self.info.ctx.call_hf_processor(
345
+ self.info.get_hf_processor(**hf_processor_mm_kwargs),
346
+ dict(text=prompt_text, **mm_data),
347
+ hf_processor_mm_kwargs,
348
)
349
+ processed_data.update(passthrough_data)
350
+ return processed_data
351
352
# LLaVA-OneVision processor doesn't support multiple videos
353
# with different sizes when converting back to tensors
@@ -346,19 +357,13 @@ class LlavaOnevisionMultiModalProcessor(
357
image_token = processor.image_token
358
video_token = processor.video_token
359
349
- text_outputs = super()._call_hf_processor(
350
- prompt=prompt,
351
- mm_data={},
352
- mm_kwargs=mm_kwargs,
353
- )
354
-
360
images = mm_data.pop("images", [])
361
assert isinstance(images, list)
362
if images:
358
- processor_outputs = super()._call_hf_processor(
359
- prompt=image_token * len(images),
360
- mm_data={"images": images},
361
- mm_kwargs=mm_kwargs,
363
+ processor_outputs = self.info.ctx.call_hf_processor(
364
+ self.info.get_hf_processor(**hf_processor_mm_kwargs),
365
+ dict(text=image_token * len(images), **{"images": images}),
366
+ hf_processor_mm_kwargs,
367
)
368
image_outputs = {
369
k: v
@@ -370,10 +375,10 @@ class LlavaOnevisionMultiModalProcessor(
375
376
pixel_values_videos = []
377
for video in videos:
373
- item_outputs = super()._call_hf_processor(
374
- prompt=video_token,
375
- mm_data={"videos": video},
376
- mm_kwargs=mm_kwargs,
378
+ item_outputs = self.info.ctx.call_hf_processor(
379
+ self.info.get_hf_processor(**hf_processor_mm_kwargs),
380
+ dict(text=video_token, **{"videos": video}),
381
+ hf_processor_mm_kwargs,
382
)
383
384
pixel_values_videos.append(item_outputs["pixel_values_videos"][0])
@@ -381,11 +386,13 @@ class LlavaOnevisionMultiModalProcessor(
386
video_outputs = {"pixel_values_videos": pixel_values_videos}
387
388
combined_outputs = dict(
384
- text_outputs,
389
+ input_ids=prompt_text,
390
**image_outputs,
391
**video_outputs,
392
)
388
- return BatchFeature(combined_outputs)
393
+ processed_data = BatchFeature(combined_outputs)
394
+ processed_data.update(passthrough_data)
395
+ return processed_data
396
397
def _get_prompt_updates(
398
self,
vllm/model_executor/models/llava_onevision2.py
+48
-33
@@ -286,7 +286,7 @@ def _validate_video_sources(paths, model_config) -> list[str]:
286
# participates in the standard decode-stage pipeline.
287
#
288
# * codec backend: NOT a loader. OV2's codec path needs the video path string
289
-# to survive into ``_call_hf_processor``, where the HF processor builds the
289
+# to survive into ``_apply_hf_processor_main``, where the HF processor builds the
290
# codec canvas + smart_resize + patchify
291
# (pixel_values/image_grid_thw/patch_positions). That transform is
292
# path-level and inseparable; it cannot be reconstructed from pre-decoded
@@ -302,7 +302,7 @@ def prepare_codec_video_input(video_path: str) -> tuple:
302
303
Returns ``(dummy_ndarray, metadata)`` where the ndarray satisfies the
304
parser's 4-D shape check and the metadata carries the actual path to
305
- our ``_call_hf_processor``. Use as::
305
+ our ``_apply_hf_processor_main``. Use as::
306
307
multi_modal_data = {"video": prepare_codec_video_input("foo.mp4")}
308
@@ -1283,7 +1283,7 @@ class LlavaOnevision2ProcessingInfo(BaseProcessingInfo):
1283
# ``video_needs_metadata=True`` makes the parser preserve both the
1284
# ``(frames, metadata)`` tuples from the frame backend and the
1285
# ``(dummy, {marker: path})`` tuples from prepare_codec_video_input;
1286
- # both are dispatched by metadata content in ``_call_hf_processor``.
1286
+ # both are dispatched by metadata content in ``_apply_hf_processor_main``.
1287
return LlavaOnevision2MultiModalDataParser(
1288
self.get_hf_config().vision_config.spatial_merge_size,
1289
video_needs_metadata=True,
@@ -1547,18 +1547,23 @@ class LlavaOnevision2MultiModalDataParser(MultiModalDataParser):
1547
class LlavaOnevision2MultiModalProcessor(
1548
BaseMultiModalProcessor[LlavaOnevision2ProcessingInfo]
1549
):
1550
- def _call_hf_processor(
1550
+ def _apply_hf_processor_main(
1551
self,
1552
- prompt: str,
1553
- mm_data: Mapping[str, object],
1554
- mm_kwargs: Mapping[str, object],
1552
+ mm_items: MultiModalDataItems,
1553
+ hf_processor_mm_kwargs: Mapping[str, object],
1554
) -> BatchFeature:
1556
- # The wrapped OV2 processor is a bare custom class without the standard
1557
- # ProcessorMixin ``_merge_kwargs`` machinery, so vLLM's default path
1558
- # fails; overriding this method routes the base class to call us
1559
- # directly.
1560
- hf_processor = self.info.get_hf_processor(**mm_kwargs)
1561
- merged_kwargs = self.info.ctx.get_merged_mm_kwargs(mm_kwargs)
1555
+ valid_mm_items = mm_items.select(
1556
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
1557
+ )
1558
+ mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
1559
+
1560
+ if not mm_data:
1561
+ return BatchFeature(dict(passthrough_data))
1562
+
1563
+ prompt_text = self.dummy_inputs.get_dummy_text(mm_items.get_all_counts())
1564
+
1565
+ hf_processor = self.info.get_hf_processor(**hf_processor_mm_kwargs)
1566
+ merged_kwargs = self.info.ctx.get_merged_mm_kwargs(hf_processor_mm_kwargs)
1567
merged_kwargs.setdefault("return_tensors", "pt")
1568
call_kwargs = {
1569
k: v
@@ -1627,21 +1632,25 @@ class LlavaOnevision2MultiModalProcessor(
1632
if len(codec_video_paths) > 1
1633
else codec_video_paths[0]
1634
)
1630
- # Route through the base ``_call_hf_processor`` so float-tensor
1635
+ # Route through ``ctx.call_hf_processor`` so float-tensor
1636
# dtype postprocessing is applied automatically; inject
1637
# ``video_backend="codec"`` via mm_kwargs so the wrapped processor
1638
# dispatches to its codec branch.
1634
- output = super()._call_hf_processor(
1635
- prompt=prompt,
1636
- mm_data=mm_data,
1637
- mm_kwargs={**mm_kwargs, "video_backend": "codec"},
1639
+ output = self.info.ctx.call_hf_processor(
1640
+ self.info.get_hf_processor(
1641
+ **{**hf_processor_mm_kwargs, "video_backend": "codec"}
1642
+ ),
1643
+ dict(text=prompt_text, **mm_data),
1644
+ {**hf_processor_mm_kwargs, "video_backend": "codec"},
1645
)
1646
data = dict(output)
1640
- return BatchFeature(
1647
+ processed_data = BatchFeature(
1648
self._rename_codec_outputs_to_video(
1649
data, codec_video_paths, hf_processor
1650
)
1651
)
1652
+ processed_data.update(passthrough_data)
1653
+ return processed_data
1654
1655
# ---- Frame backend (registered LlavaOnevision2VideoBackend) ------
1656
# Every non-codec video reaches here as a ``(frames_ndarray, metadata)``
@@ -1657,7 +1666,9 @@ class LlavaOnevision2MultiModalProcessor(
1666
# local-file gating is enforced by the connector before decoding.
1667
if videos_present:
1668
timestamp_decimals = int(
1660
- mm_kwargs.get("timestamp_decimals", _DEFAULT_TIMESTAMP_DECIMALS)
1669
+ hf_processor_mm_kwargs.get(
1670
+ "timestamp_decimals", _DEFAULT_TIMESTAMP_DECIMALS
1671
+ )
1672
)
1673
1674
per_video_frames: list[list[Image.Image]] = []
@@ -1671,7 +1682,7 @@ class LlavaOnevision2MultiModalProcessor(
1682
# ``<{t} seconds><|vision_start|><|image_pad|><|vision_end|>``
1683
# blocks (matches the OV2 hf-chat reference exactly).
1684
new_prompt = _expand_video_markers_in_prompt(
1674
- prompt,
1685
+ prompt_text,
1686
per_video_timestamps,
1687
timestamp_decimals=timestamp_decimals,
1688
)
@@ -1700,7 +1711,7 @@ class LlavaOnevision2MultiModalProcessor(
1711
row_is_video: list[bool] = []
1712
vid_idx = 0
1713
img_idx = 0
1703
- for marker in marker_pattern.finditer(prompt):
1714
+ for marker in marker_pattern.finditer(prompt_text):
1715
if marker.lastgroup == "video":
1716
frames = per_video_frames[vid_idx]
1717
vid_idx += 1
@@ -1723,15 +1734,15 @@ class LlavaOnevision2MultiModalProcessor(
1734
merged_mm_data.pop("videos", None)
1735
merged_mm_data["images"] = flat_frames
1736
1726
- # Route through the base ``_call_hf_processor`` (applies float-tensor
1737
+ # Route through ``ctx.call_hf_processor`` (applies float-tensor
1738
# dtype postprocessing automatically). The wrapped processor's image
1739
# branch ignores video/codec-only kwargs and does not forward extra
1740
# **kwargs to the image processor, so passing the full merged kwarg
1741
# set here is a no-op beyond return_tensors/padding.
1731
- output = super()._call_hf_processor(
1732
- prompt=new_prompt,
1733
- mm_data=merged_mm_data,
1734
- mm_kwargs=mm_kwargs,
1742
+ output = self.info.ctx.call_hf_processor(
1743
+ self.info.get_hf_processor(**hf_processor_mm_kwargs),
1744
+ dict(text=new_prompt, **merged_mm_data),
1745
+ hf_processor_mm_kwargs,
1746
)
1747
data = dict(output)
1748
@@ -1788,17 +1799,21 @@ class LlavaOnevision2MultiModalProcessor(
1799
(len(per_video_timestamps),), dtype=torch.long
1800
)
1801
1791
- return BatchFeature(data)
1802
+ processed_data = BatchFeature(data)
1803
+ processed_data.update(passthrough_data)
1804
+ return processed_data
1805
1806
# ---- Image-only / text-only call --------------------------------
1794
- # No videos present: delegate to the base ``_call_hf_processor``, which
1807
+ # No videos present: delegate to ``ctx.call_hf_processor``, which
1808
# runs the wrapped processor over the (possibly empty) image set and
1809
# applies float-tensor dtype postprocessing automatically.
1797
- return super()._call_hf_processor(
1798
- prompt=prompt,
1799
- mm_data=mm_data,
1800
- mm_kwargs=mm_kwargs,
1810
+ processed_data = self.info.ctx.call_hf_processor(
1811
+ self.info.get_hf_processor(**hf_processor_mm_kwargs),
1812
+ dict(text=prompt_text, **mm_data),
1813
+ hf_processor_mm_kwargs,
1814
)
1815
+ processed_data.update(passthrough_data)
1816
+ return processed_data
1817
1818
def _rename_codec_outputs_to_video(
1819
self,
vllm/model_executor/models/midashenglm.py
+9
-20
@@ -26,7 +26,7 @@
26
import collections
27
import collections.abc
28
from collections.abc import Callable, Iterable, Mapping, Sequence
29
-from typing import Annotated, Any, TypeAlias, cast
29
+from typing import Annotated, TypeAlias, cast
30
31
import numpy as np
32
import torch
@@ -583,15 +583,17 @@ class MiDashengLMDummyInputsBuilder(BaseDummyInputsBuilder[MiDashengLMProcessing
583
class MiDashengLMMultiModalProcessor(
584
BaseMultiModalProcessor[MiDashengLMProcessingInfo]
585
):
586
- def _call_hf_processor(
586
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
587
+ return self.dummy_inputs.get_dummy_text(mm_counts)
588
+
589
+ def _preprocess_hf_mm_data(
590
self,
588
- prompt: str,
591
mm_data: Mapping[str, object],
590
- mm_kwargs: Mapping[str, Any],
591
- ) -> BatchFeature:
592
+ hf_processor_mm_kwargs: Mapping[str, object],
593
+ ) -> tuple[Mapping[str, object], Mapping[str, object]]:
594
+ mm_data = dict(mm_data)
595
audios = mm_data.pop("audios", [])
596
594
- # + Padding
597
min_audio_len = self.info.get_min_audio_len()
598
processed_audios = [
599
np.pad(
@@ -608,20 +610,7 @@ class MiDashengLMMultiModalProcessor(
610
if processed_audios:
611
mm_data["audio"] = processed_audios
612
611
- if not mm_data.get("audio", []):
612
- prompt_ids = self.info.get_tokenizer().encode(prompt)
613
- prompt_ids = self._apply_hf_processor_tokens_only(prompt_ids)
614
- return BatchFeature(dict(input_ids=[prompt_ids]), tensor_type="pt")
615
-
616
- mm_kwargs = dict(
617
- **mm_kwargs,
618
- )
619
-
620
- return super()._call_hf_processor(
621
- prompt=prompt,
622
- mm_data=mm_data,
623
- mm_kwargs=mm_kwargs,
624
- )
613
+ return mm_data, hf_processor_mm_kwargs
614
615
def _get_mm_fields_config(
616
self,
vllm/model_executor/models/mimo_v2_omni.py
+18
-5
@@ -912,16 +912,23 @@ class MiMoV2OmniMultiModalProcessor(BaseMultiModalProcessor[MiMoV2OmniProcessing
912
fields["va_audio_features"] = MultiModalFieldConfig.batched("va_audio")
913
return fields
914
915
- def _call_hf_processor(
915
+ def _apply_hf_processor_main(
916
self,
917
- prompt: str,
918
- mm_data: Mapping[str, object],
919
- mm_kwargs: Mapping[str, object],
917
+ mm_items: MultiModalDataItems,
918
+ hf_processor_mm_kwargs: Mapping[str, object],
919
) -> BatchFeature:
920
"""Convert numpy video arrays to (TCHW, timestamps) tuples for MiMo.
921
Also remap 'audios' → 'audio' since MiMoOmniProcessor.__call__ uses
922
the singular form.
923
"""
924
+ valid_mm_items = mm_items.select(
925
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
926
+ )
927
+ mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
928
+
929
+ if not mm_data:
930
+ return BatchFeature(dict(passthrough_data))
931
+
932
# Remap audios → audio (MiMoOmniProcessor uses singular param name)
933
if "audios" in mm_data:
934
mm_data = {**mm_data, "audio": mm_data["audios"]}
@@ -999,7 +1006,13 @@ class MiMoV2OmniMultiModalProcessor(BaseMultiModalProcessor[MiMoV2OmniProcessing
1006
1007
mm_data = {**mm_data, "videos": converted}
1008
1002
- return super()._call_hf_processor(prompt, mm_data, mm_kwargs)
1009
+ processed_data = self.info.ctx.call_hf_processor(
1010
+ self.info.get_hf_processor(**hf_processor_mm_kwargs),
1011
+ mm_data,
1012
+ hf_processor_mm_kwargs,
1013
+ )
1014
+ processed_data.update(passthrough_data)
1015
+ return processed_data
1016
1017
def _get_prompt_updates(
1018
self,
vllm/model_executor/models/minicpmo.py
+1
-1
@@ -448,7 +448,7 @@ class MiniCPMOMultiModalProcessor(MiniCPMVMultiModalProcessor[MiniCPMOProcessing
448
if isinstance(parsed_audios, MiniCPMOAudioEmbeddingItems):
449
audio_inputs = {}
450
else:
451
- audio_inputs = self._base_call_hf_processor(
451
+ audio_inputs = self._call_hf_processor_on_prompts(
452
prompts=[self.info.audio_pattern] * len(parsed_audios),
453
mm_data={"audios": [[audio] for audio in parsed_audios]},
454
mm_kwargs={**mm_kwargs, "chunk_input": True},
vllm/model_executor/models/minicpmv.py
+26
-18
@@ -881,7 +881,7 @@ class MiniCPMVMultiModalProcessor(BaseMultiModalProcessor[_I]):
881
if isinstance(parsed_images, MiniCPMVImageEmbeddingItems):
882
image_inputs = {}
883
else:
884
- image_inputs = self._base_call_hf_processor(
884
+ image_inputs = self._call_hf_processor_on_prompts(
885
prompts=[self.info.image_pattern] * len(parsed_images),
886
mm_data={"images": [[image] for image in parsed_images]},
887
mm_kwargs=mm_kwargs,
@@ -906,7 +906,7 @@ class MiniCPMVMultiModalProcessor(BaseMultiModalProcessor[_I]):
906
if isinstance(parsed_videos, MiniCPMVVideoEmbeddingItems):
907
video_inputs = {}
908
else:
909
- video_inputs = self._base_call_hf_processor(
909
+ video_inputs = self._call_hf_processor_on_prompts(
910
prompts=[
911
self.info.image_pattern * len(video) for video in parsed_videos
912
],
@@ -983,7 +983,7 @@ class MiniCPMVMultiModalProcessor(BaseMultiModalProcessor[_I]):
983
984
return new_token_ids, placeholders
985
986
- def _base_call_hf_processor(
986
+ def _call_hf_processor_on_prompts(
987
self,
988
prompts: list[str],
989
mm_data: Mapping[str, Sequence[object]],
@@ -993,19 +993,19 @@ class MiniCPMVMultiModalProcessor(BaseMultiModalProcessor[_I]):
993
) -> dict[str, NestedTensors]:
994
# This processor supports zipping prompt and mm_data together
995
if self.info.get_model_version() in {(2, 6), (4, 0), (4, 5), (4, 6)}:
996
- inputs = super()._call_hf_processor(
997
- prompt=prompts, # type: ignore
998
- mm_data=mm_data,
999
- mm_kwargs=mm_kwargs,
996
+ inputs = self.info.ctx.call_hf_processor(
997
+ self.info.get_hf_processor(**mm_kwargs),
998
+ dict(text=prompts, **mm_data),
999
+ mm_kwargs,
1000
)
1001
else:
1002
inputs = defaultdict[str, list[torch.Tensor]](list)
1003
1004
for i, prompt in enumerate(prompts):
1005
- inputs_one = super()._call_hf_processor(
1006
- prompt=prompt,
1007
- mm_data={k: v[i] for k, v in mm_data.items()},
1008
- mm_kwargs=mm_kwargs,
1005
+ inputs_one = self.info.ctx.call_hf_processor(
1006
+ self.info.get_hf_processor(**mm_kwargs),
1007
+ dict(text=prompt, **{k: v[i] for k, v in mm_data.items()}),
1008
+ mm_kwargs,
1009
)
1010
1011
for k, v in inputs_one.items():
@@ -1014,23 +1014,31 @@ class MiniCPMVMultiModalProcessor(BaseMultiModalProcessor[_I]):
1014
1015
return {k: inputs[k] for k in out_keys}
1016
1017
- def _call_hf_processor(
1017
+ def _apply_hf_processor_main(
1018
self,
1019
- prompt: str,
1020
- mm_data: Mapping[str, object],
1021
- mm_kwargs: Mapping[str, object],
1019
+ mm_items: MultiModalDataItems,
1020
+ hf_processor_mm_kwargs: Mapping[str, object],
1021
) -> BatchFeature:
1022
+ valid_mm_items = mm_items.select(
1023
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
1024
+ )
1025
+ mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
1026
+
1027
+ prompt_text = self.dummy_inputs.get_dummy_text(mm_items.get_all_counts())
1028
+
1029
tokenizer = self.info.get_tokenizer()
1030
1025
- input_ids = torch.tensor([tokenizer.encode(prompt)])
1026
- mm_inputs = self.process_mm_inputs(mm_data, mm_kwargs)
1031
+ input_ids = torch.tensor([tokenizer.encode(prompt_text)])
1032
+ mm_inputs = self.process_mm_inputs(mm_data, hf_processor_mm_kwargs)
1033
1028
- return BatchFeature(
1034
+ processed_data = BatchFeature(
1035
{
1036
"input_ids": input_ids,
1037
**mm_inputs,
1038
}
1039
)
1040
+ processed_data.update(passthrough_data)
1041
+ return processed_data
1042
1043
def _get_prompt_updates(
1044
self,
vllm/model_executor/models/mistral3.py
+12
-12
@@ -233,30 +233,30 @@ class Mistral3DummyInputsBuilder(BaseDummyInputsBuilder[Mistral3ProcessingInfo])
233
234
235
class Mistral3MultiModalProcessor(BaseMultiModalProcessor[Mistral3ProcessingInfo]):
236
- def _call_hf_processor(
236
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
237
+ return self.dummy_inputs.get_dummy_text(mm_counts)
238
+
239
+ def _postprocess_hf_mm_data(
240
self,
238
- prompt: str,
241
mm_data: Mapping[str, object],
240
- mm_kwargs: Mapping[str, object],
242
+ hf_processor_mm_kwargs: Mapping[str, object],
243
+ processed_data: BatchFeature,
244
) -> BatchFeature:
242
- processed_outputs = super()._call_hf_processor(
243
- prompt=prompt,
244
- mm_data=mm_data,
245
- mm_kwargs=mm_kwargs,
246
- )
245
+ if not mm_data:
246
+ return processed_data
247
248
- pixel_values = processed_outputs.get("pixel_values")
248
+ pixel_values = processed_data.get("pixel_values")
249
if pixel_values is not None:
250
# Avoid padding since we need the output for each image to be
251
# independent of other images for the cache to work correctly
252
- image_sizes = processed_outputs["image_sizes"]
252
+ image_sizes = processed_data["image_sizes"]
253
assert len(pixel_values) == len(image_sizes)
254
255
- processed_outputs["pixel_values"] = [
255
+ processed_data["pixel_values"] = [
256
p[:, :h, :w] for p, (h, w) in zip(pixel_values, image_sizes)
257
]
258
259
- return processed_outputs
259
+ return processed_data
260
261
def _get_mm_fields_config(
262
self,
vllm/model_executor/models/mllama4.py
+13
-13
@@ -589,23 +589,23 @@ class Mllama4ProcessingInfo(BaseProcessingInfo):
589
590
591
class Mllama4MultiModalProcessor(BaseMultiModalProcessor[Mllama4ProcessingInfo]):
592
- def _call_hf_processor(
592
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
593
+ return self.dummy_inputs.get_dummy_text(mm_counts)
594
+
595
+ def _postprocess_hf_mm_data(
596
self,
594
- prompt: str,
597
mm_data: Mapping[str, object],
596
- mm_kwargs: Mapping[str, object],
598
+ hf_processor_mm_kwargs: Mapping[str, object],
599
+ processed_data: BatchFeature,
600
) -> BatchFeature:
598
- processed_outputs = super()._call_hf_processor(
599
- prompt=prompt,
600
- mm_data=mm_data,
601
- mm_kwargs=mm_kwargs,
602
- )
601
+ if not mm_data:
602
+ return processed_data
603
604
- processor = self.info.get_hf_processor(**mm_kwargs)
604
+ processor = self.info.get_hf_processor(**hf_processor_mm_kwargs)
605
image_processor = processor.image_processor
606
vision_config = self.info.get_hf_config().vision_config
607
608
- if processed_outputs.get("pixel_values") is not None:
608
+ if processed_data.get("pixel_values") is not None:
609
assert "images" in mm_data, (
610
"images expected to be in mm_data when pixel_values is present"
611
)
@@ -636,10 +636,10 @@ class Mllama4MultiModalProcessor(BaseMultiModalProcessor[Mllama4ProcessingInfo])
636
1 if r_h * r_w == 1 else 1 + r_h * r_w for (r_h, r_w) in aspect_ratios
637
]
638
639
- processed_outputs["aspect_ratios"] = torch.tensor(aspect_ratios)
640
- processed_outputs["patches_per_image"] = torch.tensor(patches_per_image)
639
+ processed_data["aspect_ratios"] = torch.tensor(aspect_ratios)
640
+ processed_data["patches_per_image"] = torch.tensor(patches_per_image)
641
642
- return processed_outputs
642
+ return processed_data
643
644
def _get_mm_fields_config(
645
self,
vllm/model_executor/models/molmo.py
+42
-39
@@ -1123,17 +1123,44 @@ class MolmoDummyInputsBuilder(BaseDummyInputsBuilder[MolmoProcessingInfo]):
1123
1124
1125
class MolmoMultiModalProcessor(BaseMultiModalProcessor[MolmoProcessingInfo]):
1126
- def _call_hf_processor(
1126
+ def _postprocess_prompt(self, prompt: list[int]) -> list[int]:
1127
+ processor = self.info.get_hf_processor()
1128
+
1129
+ # The chat template is already applied to the prompt tokens
1130
+ # Use message_format="none" to avoid applying it again
1131
+ # Prepend an empty space if `always_start_with_space` is True
1132
+ tokens = processor.get_tokens_input(
1133
+ self.info.get_tokenizer().decode(prompt),
1134
+ message_format="none",
1135
+ always_start_with_space=True,
1136
+ )
1137
+
1138
+ # Prepend a BOS token id to the tokens
1139
+ return self.info.ctx.call_hf_processor(
1140
+ processor.process,
1141
+ dict(tokens=tokens),
1142
+ )["input_ids"].tolist()
1143
+
1144
+ def _apply_hf_processor_main(
1145
self,
1128
- prompt: str,
1129
- mm_data: Mapping[str, object],
1130
- mm_kwargs: Mapping[str, object],
1146
+ mm_items: MultiModalDataItems,
1147
+ hf_processor_mm_kwargs: Mapping[str, object],
1148
) -> BatchFeature:
1132
- hf_processor = self.info.get_hf_processor(**mm_kwargs)
1133
- processed_outputs = self.info.ctx.call_hf_processor(
1149
+ mm_counts = mm_items.get_all_counts()
1150
+
1151
+ valid_mm_items = mm_items.select({k for k, c in mm_counts.items() if c > 0})
1152
+ processor_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
1153
+
1154
+ if not processor_data:
1155
+ return BatchFeature(dict(passthrough_data))
1156
+
1157
+ hf_processor = self.info.get_hf_processor(**hf_processor_mm_kwargs)
1158
+ prompt_text = self.dummy_inputs.get_dummy_text(mm_counts)
1159
+
1160
+ processed_data = self.info.ctx.call_hf_processor(
1161
hf_processor.process,
1135
- dict(text=prompt, **mm_data),
1136
- mm_kwargs,
1162
+ dict(text=prompt_text, **processor_data),
1163
+ hf_processor_mm_kwargs,
1164
)
1165
1166
tokenizer = hf_processor.tokenizer
@@ -1141,17 +1168,16 @@ class MolmoMultiModalProcessor(BaseMultiModalProcessor[MolmoProcessingInfo]):
1168
1169
image_processor = hf_processor.image_processor
1170
1144
- input_ids: torch.Tensor = processed_outputs.pop("input_ids")
1145
- processed_outputs["input_ids"] = input_ids.unsqueeze(0)
1171
+ processed_data.pop("input_ids")
1172
1147
- if (images := mm_data.get("images")) is not None:
1173
+ if (images := processor_data.get("images")) is not None:
1174
mm_items = self.info.parse_mm_data({"image": images}, validate=False)
1175
parsed_images = mm_items.get_items("image", ImageProcessorItems)
1176
image_sizes = [
1177
parsed_images.get_image_size(i) for i in range(len(parsed_images))
1178
]
1179
1154
- feat_is_patch = processed_outputs["image_input_idx"] >= 0
1180
+ feat_is_patch = processed_data["image_input_idx"] >= 0
1181
1182
tilings = [
1183
self.info.select_tiling(
@@ -1165,35 +1191,12 @@ class MolmoMultiModalProcessor(BaseMultiModalProcessor[MolmoProcessingInfo]):
1191
num_crops = torch.tensor(tilings).prod(-1) + 1
1192
assert num_crops.sum() == len(feat_is_patch)
1193
1168
- processed_outputs["num_crops"] = num_crops
1169
- processed_outputs["img_patch_id"] = image_patch_id
1170
-
1171
- return processed_outputs
1172
-
1173
- def _apply_hf_processor_tokens_only(
1174
- self,
1175
- prompt_tokens: list[int],
1176
- ) -> list[int]:
1177
- processor = self.info.get_hf_processor()
1194
+ processed_data["num_crops"] = num_crops
1195
+ processed_data["img_patch_id"] = image_patch_id
1196
1179
- # The chat template is already applied to the prompt tokens
1180
- # Use message_format="none" to avoid applying it again
1181
- # Prepend an empty space if `always_start_with_space` is True
1182
- tokens = processor.get_tokens_input(
1183
- self.info.get_tokenizer().decode(prompt_tokens),
1184
- message_format="none",
1185
- always_start_with_space=True,
1186
- )
1187
-
1188
- # Prepend a BOS token id to the tokens
1189
- processed_data = self.info.ctx.call_hf_processor(
1190
- processor.process,
1191
- dict(tokens=tokens),
1192
- )
1193
- prompt_ids = processed_data.pop("input_ids").tolist()
1194
- print(prompt_ids, len(prompt_ids))
1197
+ processed_data.update(passthrough_data)
1198
1196
- return prompt_ids
1199
+ return processed_data
1200
1201
def _get_mm_fields_config(
1202
self,
vllm/model_executor/models/molmo2.py
+41
-32
@@ -1929,30 +1929,33 @@ class Molmo2DummyInputsBuilder(BaseDummyInputsBuilder[Molmo2ProcessingInfo]):
1929
1930
1931
class Molmo2MultiModalProcessor(BaseMultiModalProcessor[Molmo2ProcessingInfo]):
1932
- def _apply_hf_processor_tokens_only(
1933
- self,
1934
- prompt_tokens: list[int],
1935
- ) -> list[int]:
1932
+ def _postprocess_prompt(self, prompt: list[int]) -> list[int]:
1933
processor = self.info.get_hf_processor()
1934
tokenizer = processor.tokenizer
1935
bos_token_id = tokenizer.bos_token_id or tokenizer.eos_token_id
1936
1940
- if len(prompt_tokens) == 0 or prompt_tokens[0] != bos_token_id:
1937
+ if len(prompt) == 0 or prompt[0] != bos_token_id:
1938
# Prepend the bos token to the prompt tokens
1942
- prompt_tokens = [bos_token_id] + prompt_tokens
1939
+ prompt = [bos_token_id] + prompt
1940
1944
- return prompt_tokens
1941
+ return prompt
1942
1946
- def _call_hf_processor(
1943
+ def _apply_hf_processor_main(
1944
self,
1948
- prompt: str,
1949
- mm_data: Mapping[str, object],
1950
- mm_kwargs: Mapping[str, object],
1945
+ mm_items: MultiModalDataItems,
1946
+ hf_processor_mm_kwargs: Mapping[str, object],
1947
) -> BatchFeature:
1952
- mm_data = dict(mm_data)
1948
+ mm_counts = mm_items.get_all_counts()
1949
+
1950
+ valid_mm_items = mm_items.select({k for k, c in mm_counts.items() if c > 0})
1951
+ processor_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
1952
+
1953
+ prompt_text = self.dummy_inputs.get_dummy_text(mm_counts)
1954
+
1955
+ mm_data = dict(processor_data)
1956
1957
hf_config = self.info.get_hf_config()
1955
- hf_processor = self.info.get_hf_processor(**mm_kwargs)
1958
+ hf_processor = self.info.get_hf_processor(**hf_processor_mm_kwargs)
1959
1960
def patched_call(text=None, images=None, videos=None, **kwargs) -> BatchFeature:
1961
res = hf_processor(text=text, images=images, videos=videos, **kwargs)
@@ -1984,14 +1987,16 @@ class Molmo2MultiModalProcessor(BaseMultiModalProcessor[Molmo2ProcessingInfo]):
1987
# NOTE: metadata.frames_indices indicates
1988
# the sampled frames indices of pre-sampled videos, which is
1989
# used to calculate the timestamps. Make sure that
1987
- # do_sample_frames in mm_kwargs is false for presampled videos.
1990
+ # do_sample_frames in hf_processor_mm_kwargs is false for
1991
+ # presampled videos.
1992
1989
- # NOTE: a copy of mm_kwargs is created to update do_sample_frames,
1990
- # otherwise mm_hash for the object will be incorrect.
1991
- video_mm_kwargs = dict(**mm_kwargs)
1993
+ # NOTE: a copy of hf_processor_mm_kwargs is created to update
1994
+ # do_sample_frames, otherwise mm_hash for the object will be
1995
+ # incorrect.
1996
+ video_mm_kwargs = dict(**hf_processor_mm_kwargs)
1997
if "do_sample_frames" not in video_mm_kwargs:
1998
# molmo_utils already has "do_sample_frames" in
1994
- # mm_kwargs, don't overwrite it.
1999
+ # hf_processor_mm_kwargs, don't overwrite it.
2000
video_mm_kwargs["do_sample_frames"] = metadata.get(
2001
"do_sample_frames", False
2002
)
@@ -2015,7 +2020,7 @@ class Molmo2MultiModalProcessor(BaseMultiModalProcessor[Molmo2ProcessingInfo]):
2020
input_ids = input_ids[:, 1:]
2021
2022
video_string = tokenizer.batch_decode(input_ids)[0]
2018
- prompt = prompt.replace(VIDEO_PROMPT, video_string, 1)
2023
+ prompt_text = prompt_text.replace(VIDEO_PROMPT, video_string, 1)
2024
2025
video_grids = video_outputs.pop("video_grids")
2026
assert video_grids[:, 0].sum() == len(
@@ -2054,10 +2059,10 @@ class Molmo2MultiModalProcessor(BaseMultiModalProcessor[Molmo2ProcessingInfo]):
2059
else:
2060
all_video_outputs = dict()
2061
2057
- processed_outputs = self.info.ctx.call_hf_processor(
2062
+ processed_data = self.info.ctx.call_hf_processor(
2063
patched_call,
2059
- dict(text=prompt, **mm_data),
2060
- mm_kwargs,
2064
+ dict(text=prompt_text, **mm_data),
2065
+ hf_processor_mm_kwargs,
2066
)
2067
2068
if (images := mm_data.get("images")) is not None:
@@ -2077,22 +2082,22 @@ class Molmo2MultiModalProcessor(BaseMultiModalProcessor[Molmo2ProcessingInfo]):
2082
for image_size in image_sizes
2083
]
2084
num_crops = torch.tensor(tilings).prod(-1) + 1
2080
- assert sum(num_crops) == len(processed_outputs["pixel_values"])
2081
- assert sum(num_crops) == processed_outputs["image_num_crops"].sum().item()
2085
+ assert sum(num_crops) == len(processed_data["pixel_values"])
2086
+ assert sum(num_crops) == processed_data["image_num_crops"].sum().item()
2087
2083
- image_grids = processed_outputs.pop("image_grids")
2088
+ image_grids = processed_data.pop("image_grids")
2089
image_num_pooled_patches = image_grids[:, :2].prod(dim=1) + image_grids[
2090
:, 2:
2091
].prod(dim=1)
2092
2088
- processed_outputs["image_num_pooled_patches"] = image_num_pooled_patches
2089
- n_patches = processed_outputs["pixel_values"].shape[1]
2090
- processed_outputs["image_num_patches"] = (
2091
- processed_outputs["image_num_crops"] * n_patches
2093
+ processed_data["image_num_pooled_patches"] = image_num_pooled_patches
2094
+ n_patches = processed_data["pixel_values"].shape[1]
2095
+ processed_data["image_num_patches"] = (
2096
+ processed_data["image_num_crops"] * n_patches
2097
)
2098
(
2094
- processed_outputs["image_tokens"],
2095
- processed_outputs["num_image_tokens"],
2099
+ processed_data["image_tokens"],
2100
+ processed_data["num_image_tokens"],
2101
) = build_flat_image_bool_length(
2102
image_grids,
2103
hf_config,
@@ -2101,7 +2106,11 @@ class Molmo2MultiModalProcessor(BaseMultiModalProcessor[Molmo2ProcessingInfo]):
2106
use_single_crop_start_token=hf_processor.use_single_crop_start_token,
2107
)
2108
2104
- return BatchFeature({**processed_outputs, **all_video_outputs})
2109
+ processed_data.update(all_video_outputs)
2110
+ processed_data.update(passthrough_data)
2111
+ processed_data.pop("input_ids")
2112
+
2113
+ return processed_data
2114
2115
def _get_mm_fields_config(
2116
self,
vllm/model_executor/models/moondream3.py
-10
@@ -942,16 +942,6 @@ class Moondream3MultiModalProcessor(BaseMultiModalProcessor[Moondream3Processing
942
image_placeholder: str = "<image>"
943
bos_image_placeholder: str = "<|endoftext|><image>"
944
945
- def _call_hf_processor(
946
- self,
947
- prompt: str,
948
- mm_data: Mapping[str, object],
949
- mm_kwargs: Mapping[str, object],
950
- ) -> BatchFeature:
951
- # Moondream3's processor handles images directly rather than exposing a
952
- # separate `image_processor`, so keep the cache path on text+MM calls.
953
- return super()._call_hf_processor(prompt, mm_data, mm_kwargs)
954
-
945
@cached_property
946
def bos_image_placeholder_tokens(self) -> list[int]:
947
tokenizer = self.info.get_tokenizer()
vllm/model_executor/models/moss_audio.py
+17
-8
@@ -1335,23 +1335,32 @@ class MossAudioDummyInputsBuilder(BaseDummyInputsBuilder[MossAudioProcessingInfo
1335
1336
1337
class MossAudioMultiModalProcessor(BaseMultiModalProcessor[MossAudioProcessingInfo]):
1338
- def _call_hf_processor(
1338
+ def _apply_hf_processor_main(
1339
self,
1340
- prompt: str,
1341
- mm_data: Mapping[str, object],
1342
- mm_kwargs: Mapping[str, object],
1340
+ mm_items: MultiModalDataItems,
1341
+ hf_processor_mm_kwargs: Mapping[str, object],
1342
) -> BatchFeature:
1343
+ valid_mm_items = mm_items.select(
1344
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
1345
+ )
1346
+ mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
1347
+
1348
+ if not mm_data:
1349
+ return BatchFeature(dict(passthrough_data))
1350
+
1351
mm_data = dict(mm_data)
1352
audios = mm_data.pop("audios", [])
1353
if audios:
1354
mm_data["audio"] = audios
1348
- mm_kwargs = dict(mm_kwargs)
1349
- processor_kwargs = _filter_moss_audio_processor_config(mm_kwargs)
1350
- return self.info.ctx.call_hf_processor(
1355
+ hf_processor_mm_kwargs = dict(hf_processor_mm_kwargs)
1356
+ processor_kwargs = _filter_moss_audio_processor_config(hf_processor_mm_kwargs)
1357
+ processed_data = self.info.ctx.call_hf_processor(
1358
self.info.get_hf_processor(**processor_kwargs),
1352
- dict(text=prompt, **mm_data),
1359
+ mm_data,
1360
{},
1361
)
1362
+ processed_data.update(passthrough_data)
1363
+ return processed_data
1364
1365
def _get_mm_fields_config(
1366
self,
vllm/model_executor/models/moss_transcribe_diarize.py
+19
-14
@@ -461,24 +461,29 @@ class MossTranscribeDiarizeDummyInputsBuilder(
461
class MossTranscribeDiarizeMultiModalProcessor(
462
BaseMultiModalProcessor[MossTranscribeDiarizeProcessingInfo]
463
):
464
- def _call_hf_processor(
464
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
465
+ return self.dummy_inputs.get_dummy_text(mm_counts)
466
+
467
+ def _preprocess_hf_mm_data(
468
self,
466
- prompt: str,
469
mm_data: Mapping[str, object],
468
- mm_kwargs: Mapping[str, object],
470
+ hf_processor_mm_kwargs: Mapping[str, object],
471
+ ) -> tuple[Mapping[str, object], Mapping[str, object]]:
472
+ audios = _get_audios_from_mm_data(mm_data)
473
+
474
+ return dict(audio=audios), hf_processor_mm_kwargs
475
+
476
+ def _postprocess_hf_mm_data(
477
+ self,
478
+ mm_data: Mapping[str, object],
479
+ hf_processor_mm_kwargs: Mapping[str, object],
480
+ processed_data: BatchFeature,
481
) -> BatchFeature:
470
- tokenizer = self.info.get_tokenizer()
482
audios = _get_audios_from_mm_data(mm_data)
472
- if not audios:
473
- input_ids = tokenizer.encode(prompt, add_special_tokens=False)
474
- return BatchFeature({"input_ids": [input_ids]}, tensor_type="pt")
475
-
476
- processed = self.info.ctx.call_hf_processor(
477
- self.info.get_hf_processor(**mm_kwargs),
478
- dict(text=prompt, audio=audios),
479
- mm_kwargs,
480
- )
481
- return _add_vllm_audio_metadata(processed, len(audios))
483
+ if audios:
484
+ processed_data = _add_vllm_audio_metadata(processed_data, len(audios))
485
+
486
+ return processed_data
487
488
def _get_mm_fields_config(
489
self,
vllm/model_executor/models/muse_glimmer.py
+15
-7
@@ -262,13 +262,19 @@ class MuseGlimmerDummyInputsBuilder(BaseDummyInputsBuilder[MuseGlimmerProcessing
262
class MuseGlimmerMultiModalProcessor(
263
BaseMultiModalProcessor[MuseGlimmerProcessingInfo]
264
):
265
- def _call_hf_processor(
265
+ def _apply_hf_processor_main(
266
self,
267
- prompt: str,
268
- mm_data: Mapping[str, object],
269
- mm_kwargs: Mapping[str, object],
267
+ mm_items: MultiModalDataItems,
268
+ hf_processor_mm_kwargs: Mapping[str, object],
269
) -> BatchFeature:
271
- processor = self.info.get_hf_processor(**mm_kwargs)
270
+ valid_mm_items = mm_items.select(
271
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
272
+ )
273
+ mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
274
+
275
+ prompt_text = self.dummy_inputs.get_dummy_text(mm_items.get_all_counts())
276
+
277
+ processor = self.info.get_hf_processor(**hf_processor_mm_kwargs)
278
tokenizer = processor.tokenizer
279
config = self.info.get_hf_config()
280
images = mm_data.get("images", ())
@@ -276,7 +282,7 @@ class MuseGlimmerMultiModalProcessor(
282
if not isinstance(images, Sequence) or not isinstance(videos, Sequence):
283
raise TypeError("MuseGlimmer multi-modal data must be a sequence")
284
279
- prompt_ids = tokenizer.encode(prompt, add_special_tokens=False)
285
+ prompt_ids = tokenizer.encode(prompt_text, add_special_tokens=False)
286
image_sentinel_id = tokenizer.convert_tokens_to_ids(IMAGE_TOKEN)
287
video_sentinel_id = tokenizer.convert_tokens_to_ids(VIDEO_TOKEN)
288
if prompt_ids.count(image_sentinel_id) != len(images):
@@ -354,7 +360,9 @@ class MuseGlimmerMultiModalProcessor(
360
video_pixel_values=video_pixels,
361
video_feature_sizes=torch.tensor(video_sizes),
362
)
357
- return BatchFeature(data=data, tensor_type=None)
363
+ processed_data = BatchFeature(data=data, tensor_type=None)
364
+ processed_data.update(passthrough_data)
365
+ return processed_data
366
367
def _get_mm_fields_config(
368
self,
vllm/model_executor/models/nano_nemotron_vl.py
+8
-19
@@ -361,18 +361,8 @@ class NanoNemotronVLProcessingInfo(BaseProcessingInfo):
361
class NanoNemotronVLMultiModalProcessor(
362
BaseMultiModalProcessor[NanoNemotronVLProcessingInfo]
363
):
364
- def _call_hf_processor(
365
- self,
366
- prompt: str,
367
- mm_data: Mapping[str, object],
368
- mm_kwargs: Mapping[str, object],
369
- ) -> BatchFeature:
370
- """
371
- Bypass `call_hf_processor_mm_only` by no-op overriding`_call_hf_processor`,
372
- so it chooses this path:
373
- `type(self)._call_hf_processor != BaseMultiModalProcessor._call_hf_processor`
374
- """
375
- return super()._call_hf_processor(prompt, mm_data, mm_kwargs)
364
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
365
+ return self.dummy_inputs.get_dummy_text(mm_counts)
366
367
def _get_image_fields_config(self, hf_inputs: BatchFeature):
368
if self.info.is_dynamic_tiler:
@@ -406,7 +396,9 @@ class NanoNemotronVLMultiModalProcessor(
396
)
397
398
def _get_audio_fields_config(self, hf_inputs: BatchFeature):
409
- audio_num_clips = torch.as_tensor(hf_inputs["audio_num_clips"])
399
+ audio_num_clips = torch.as_tensor(
400
+ hf_inputs.get("audio_num_clips", torch.empty(0))
401
+ )
402
403
return dict(
404
input_audio_features=MultiModalFieldConfig.flat_from_sizes(
@@ -733,11 +725,8 @@ class NanoNemotronVLMultiModalProcessor(
725
# Bypass the cached path: the HF processor must receive the
726
# prompt (with injected <so_embedding>) and the audio data
727
# together so it can perform audio-token replacement natively.
736
- (
737
- prompt_ids,
738
- mm_info,
739
- is_update_applied,
740
- ) = self._apply_hf_processor(inputs, timing_ctx)
728
+ mm_info = self._apply_hf_processor(inputs, timing_ctx)
729
+ prompt_ids = self._postprocess_prompt(inputs.prompt)
730
731
with timing_ctx.record("apply_prompt_updates"):
732
prompt_ids, mm_placeholders = self._maybe_apply_prompt_updates(
@@ -745,7 +734,7 @@ class NanoNemotronVLMultiModalProcessor(
734
prompt_ids=prompt_ids,
735
mm_kwargs=mm_info.kwargs,
736
mm_prompt_updates=mm_info.prompt_updates,
748
- is_update_applied=is_update_applied,
737
+ is_update_applied=self.hf_processor_applies_updates,
738
)
739
740
mm_placeholder_ranges = {
vllm/model_executor/models/nemotron_parse.py
-16
@@ -411,22 +411,6 @@ class NemotronParseMultiModalProcessor(
411
) -> list[int]:
412
return [0]
413
414
- def _call_hf_processor(
415
- self,
416
- prompt: str,
417
- mm_data: Mapping[str, object],
418
- mm_kwargs: Mapping[str, object],
419
- ) -> BatchFeature:
420
- if mm_data:
421
- processed_outputs = super()._call_hf_processor(prompt, mm_data, mm_kwargs)
422
- else:
423
- hf_processor = self.info.get_hf_processor()
424
- tokenizer = hf_processor.tokenizer
425
- processed_outputs = tokenizer(
426
- prompt, add_special_tokens=False, return_tensors="pt"
427
- )
428
- return processed_outputs
429
-
414
def _get_mm_fields_config(
415
self,
416
hf_inputs: BatchFeature,
vllm/model_executor/models/ovis.py
+10
-21
@@ -341,41 +341,30 @@ class OvisMultiModalProcessor(BaseMultiModalProcessor[OvisProcessingInfo]):
341
# -300 is image_atom token, filter them out
342
return [vte_vocab_size + x + 300 for x in image_indicators if x < -300]
343
344
- def _call_hf_processor(
344
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
345
+ return self.dummy_inputs.get_dummy_text(mm_counts)
346
+
347
+ def _postprocess_hf_mm_data(
348
self,
346
- prompt: str,
349
mm_data: Mapping[str, object],
348
- mm_kwargs: Mapping[str, object],
350
+ hf_processor_mm_kwargs: Mapping[str, object],
351
+ processed_data: BatchFeature,
352
) -> BatchFeature:
353
if not mm_data:
351
- # Avoid warning from HF logger for text-only input
352
- tokenizer = self.info.get_tokenizer()
353
- prompt_ids = tokenizer.encode(prompt, add_special_tokens=False)
354
- return BatchFeature(dict(input_ids=[prompt_ids]), tensor_type="pt")
355
-
356
- processed_outputs = super()._call_hf_processor(
357
- prompt=prompt,
358
- mm_data=mm_data,
359
- mm_kwargs=mm_kwargs,
360
- )
354
+ return processed_data
355
356
hf_processor = self.info.get_hf_processor()
357
image_indicators = [
358
hf_processor.construct_image_indicators(grid)
365
- for grid in processed_outputs["grids"]
359
+ for grid in processed_data["grids"]
360
]
361
indicator_tokens = [
362
self.image_indicators_to_visual_tokens(indicator)
363
for indicator in image_indicators
364
]
371
- processed_outputs["indicator_tokens"] = torch.tensor(indicator_tokens)
372
- return processed_outputs
365
+ processed_data["indicator_tokens"] = torch.tensor(indicator_tokens)
366
374
- def _apply_hf_processor_tokens_only(
375
- self,
376
- prompt_tokens: list[int],
377
- ) -> list[int]:
378
- return prompt_tokens
367
+ return processed_data
368
369
def _get_mm_fields_config(
370
self,
vllm/model_executor/models/ovis2_5.py
+13
-23
@@ -332,53 +332,43 @@ class Ovis2_5MultiModalProcessor(BaseMultiModalProcessor[Ovis2_5ProcessingInfo])
332
if x >= INDICATOR_IDS[0]
333
]
334
335
- def _call_hf_processor(
335
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
336
+ return self.dummy_inputs.get_dummy_text(mm_counts)
337
+
338
+ def _postprocess_hf_mm_data(
339
self,
337
- prompt: str,
340
mm_data: Mapping[str, object],
339
- mm_kwargs: Mapping[str, object],
341
+ hf_processor_mm_kwargs: Mapping[str, object],
342
+ processed_data: BatchFeature,
343
) -> BatchFeature:
344
if not mm_data:
342
- # Avoid warning from HF logger for text-only input
343
- tokenizer = self.info.get_tokenizer()
344
- prompt_ids = tokenizer.encode(prompt, add_special_tokens=False)
345
- return BatchFeature(dict(input_ids=[prompt_ids]), tensor_type="pt")
346
-
347
- processed_outputs = super()._call_hf_processor(
348
- prompt=prompt,
349
- mm_data=mm_data,
350
- mm_kwargs=mm_kwargs,
351
- )
345
+ return processed_data
346
+
347
hf_processor = self.info.get_hf_processor()
348
349
if "videos" in mm_data:
350
visual_indicators = [
351
hf_processor.construct_visual_indicators((1, 1, 1), True)
357
- for grid in processed_outputs["video_grids"]
352
+ for grid in processed_data["video_grids"]
353
]
354
indicator_tokens = [
355
self.visual_indicators_to_visual_tokens(indicator)
356
for indicator in visual_indicators
357
]
363
- processed_outputs["video_indicator_tokens"] = torch.tensor(indicator_tokens)
358
+ processed_data["video_indicator_tokens"] = torch.tensor(indicator_tokens)
359
if "images" in mm_data:
360
visual_indicators = [
361
hf_processor.construct_visual_indicators((1, 1, 1), False)
367
- for grid in processed_outputs["grids"]
362
+ for grid in processed_data["grids"]
363
]
364
indicator_tokens = [
365
self.visual_indicators_to_visual_tokens(indicator)
366
for indicator in visual_indicators
367
]
368
374
- processed_outputs["indicator_tokens"] = torch.tensor(indicator_tokens)
375
- return processed_outputs
369
+ processed_data["indicator_tokens"] = torch.tensor(indicator_tokens)
370
377
- def _apply_hf_processor_tokens_only(
378
- self,
379
- prompt_tokens: list[int],
380
- ) -> list[int]:
381
- return prompt_tokens
371
+ return processed_data
372
373
def _get_mm_fields_config(
374
self,
vllm/model_executor/models/paddleocr_vl.py
+29
-24
@@ -244,32 +244,37 @@ class PaddleOCRVLDummyInputsBuilder(BaseDummyInputsBuilder[PaddleOCRVLProcessing
244
class PaddleOCRVLMultiModalProcessor(
245
BaseMultiModalProcessor[PaddleOCRVLProcessingInfo]
246
):
247
- def _call_hf_processor(
247
+ def _apply_hf_processor_main(
248
self,
249
- prompt: str,
250
- mm_data: Mapping[str, object],
251
- mm_kwargs: Mapping[str, object],
249
+ mm_items: MultiModalDataItems,
250
+ hf_processor_mm_kwargs: Mapping[str, object],
251
) -> BatchFeature:
253
- if mm_data:
254
- final_mm_kwargs = dict(mm_kwargs or {})
255
- final_mm_kwargs.setdefault("images_kwargs", {})
256
- # vLLM use PIL.Image, always set channel_last
257
- final_mm_kwargs["input_data_format"] = ChannelDimension.LAST
258
- processed_outputs = self.info.ctx.call_hf_processor(
259
- self.info.get_hf_processor(**final_mm_kwargs),
260
- dict(text=prompt, **mm_data),
261
- mm_kwargs,
262
- )
263
- num_patches_per_image = processed_outputs["image_grid_thw"].prod(-1)
264
- processed_outputs["pixel_values"] = processed_outputs["pixel_values"].split(
265
- num_patches_per_image.tolist()
266
- )
267
- else:
268
- tokenizer = self.info.get_tokenizer()
269
- processed_outputs = tokenizer(
270
- prompt, add_special_tokens=True, return_tensors="pt"
271
- )
272
- return processed_outputs
252
+ valid_mm_items = mm_items.select(
253
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
254
+ )
255
+ mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
256
+
257
+ if not mm_data:
258
+ return BatchFeature(dict(passthrough_data))
259
+
260
+ prompt_text = self.dummy_inputs.get_dummy_text(mm_items.get_all_counts())
261
+
262
+ final_mm_kwargs = dict(hf_processor_mm_kwargs or {})
263
+ final_mm_kwargs.setdefault("images_kwargs", {})
264
+ # vLLM use PIL.Image, always set channel_last
265
+ final_mm_kwargs["input_data_format"] = ChannelDimension.LAST
266
+ processed_data = self.info.ctx.call_hf_processor(
267
+ self.info.get_hf_processor(**final_mm_kwargs),
268
+ dict(text=prompt_text, **mm_data),
269
+ hf_processor_mm_kwargs,
270
+ )
271
+ num_patches_per_image = processed_data["image_grid_thw"].prod(-1)
272
+ processed_data["pixel_values"] = processed_data["pixel_values"].split(
273
+ num_patches_per_image.tolist()
274
+ )
275
+ processed_data.update(passthrough_data)
276
+
277
+ return processed_data
278
279
def _get_mm_fields_config(
280
self,
vllm/model_executor/models/paligemma.py
-17
@@ -152,23 +152,6 @@ class PaliGemmaDummyInputsBuilder(BaseDummyInputsBuilder[PaliGemmaProcessingInfo
152
153
154
class PaliGemmaMultiModalProcessor(BaseMultiModalProcessor[PaliGemmaProcessingInfo]):
155
- def _call_hf_processor(
156
- self,
157
- prompt: str,
158
- mm_data: Mapping[str, object],
159
- mm_kwargs: Mapping[str, object],
160
- ) -> BatchFeature:
161
- tokenizer = self.info.get_tokenizer()
162
- if not mm_data:
163
- prompt_ids = tokenizer.encode(prompt, add_special_tokens=False)
164
- return BatchFeature(dict(input_ids=[prompt_ids]), tensor_type="pt")
165
-
166
- return super()._call_hf_processor(
167
- prompt=prompt,
168
- mm_data=mm_data,
169
- mm_kwargs=mm_kwargs,
170
- )
171
-
155
def _get_mm_fields_config(
156
self,
157
hf_inputs: BatchFeature,
vllm/model_executor/models/phi3v.py
+10
-10
@@ -396,19 +396,19 @@ class Phi3VDummyInputsBuilder(BaseDummyInputsBuilder[Phi3VProcessingInfo]):
396
397
398
class Phi3VMultiModalProcessor(BaseMultiModalProcessor[Phi3VProcessingInfo]):
399
- def _call_hf_processor(
399
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
400
+ return self.dummy_inputs.get_dummy_text(mm_counts)
401
+
402
+ def _postprocess_hf_mm_data(
403
self,
401
- prompt: str,
404
mm_data: Mapping[str, object],
403
- mm_kwargs: Mapping[str, object],
405
+ hf_processor_mm_kwargs: Mapping[str, object],
406
+ processed_data: BatchFeature,
407
) -> BatchFeature:
405
- processed_outputs = super()._call_hf_processor(
406
- prompt=prompt,
407
- mm_data=mm_data,
408
- mm_kwargs=mm_kwargs,
409
- )
408
+ if not mm_data:
409
+ return processed_data
410
411
- input_ids = processed_outputs["input_ids"]
411
+ input_ids = processed_data["input_ids"]
412
assert isinstance(input_ids, torch.Tensor)
413
414
# Phi3v processor has inserted -1, -2 etc as placeholder in prompt_ids,
@@ -416,7 +416,7 @@ class Phi3VMultiModalProcessor(BaseMultiModalProcessor[Phi3VProcessingInfo]):
416
# Therefore, we need to do an early replacement here
417
input_ids.masked_fill_(input_ids < 0, _IMAGE_TOKEN_ID)
418
419
- return processed_outputs
419
+ return processed_data
420
421
def _get_mm_fields_config(
422
self,
vllm/model_executor/models/phi4mm.py
+24
-15
@@ -868,43 +868,52 @@ class Phi4MMDummyInputsBuilder(BaseDummyInputsBuilder[Phi4MMProcessingInfo]):
868
869
870
class Phi4MMMultiModalProcessor(BaseMultiModalProcessor[Phi4MMProcessingInfo]):
871
- def _call_hf_processor(
871
+ def _apply_hf_processor_main(
872
self,
873
- prompt: str,
874
- mm_data: Mapping[str, object],
875
- mm_kwargs: Mapping[str, object],
873
+ mm_items: MultiModalDataItems,
874
+ hf_processor_mm_kwargs: Mapping[str, object],
875
) -> BatchFeature:
876
+ valid_mm_items = mm_items.select(
877
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
878
+ )
879
+ mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
880
+
881
if not mm_data:
878
- prompt_ids = self.info.get_tokenizer().encode(prompt)
879
- prompt_ids = self._apply_hf_processor_tokens_only(prompt_ids)
880
- return BatchFeature(dict(input_ids=[prompt_ids]), tensor_type="pt")
882
+ return BatchFeature(dict(passthrough_data))
883
882
- sr = self.info.get_feature_extractor(**mm_kwargs).sampling_rate
884
+ prompt_text = self.dummy_inputs.get_dummy_text(mm_items.get_all_counts())
885
+
886
+ sr = self.info.get_feature_extractor(**hf_processor_mm_kwargs).sampling_rate
887
if audio_data := mm_data.get("audios", []):
888
mm_data["audios"] = [(data, sr) for data in audio_data]
889
886
- processed_outputs = super()._call_hf_processor(prompt, mm_data, mm_kwargs)
890
+ processed_data = self.info.ctx.call_hf_processor(
891
+ self.info.get_hf_processor(**hf_processor_mm_kwargs),
892
+ dict(text=prompt_text, **mm_data),
893
+ hf_processor_mm_kwargs,
894
+ )
895
888
- hf_processor = self.info.get_hf_processor(**mm_kwargs)
896
+ hf_processor = self.info.get_hf_processor(**hf_processor_mm_kwargs)
897
num_img_tokens = [
898
self.info.get_num_image_tokens(
899
image_width=img_size[0],
900
image_height=img_size[1],
901
processor=hf_processor,
902
)
895
- for img_size in processed_outputs["image_sizes"]
903
+ for img_size in processed_data["image_sizes"]
904
]
897
- processed_outputs["num_img_tokens"] = num_img_tokens
905
+ processed_data["num_img_tokens"] = num_img_tokens
906
899
- audio_features = processed_outputs["input_audio_embeds"]
907
+ audio_features = processed_data["input_audio_embeds"]
908
feature_sizes = [
909
self.info.get_audio_num_frames(len(audio), sr) for audio in audio_data
910
]
903
- processed_outputs["input_audio_embeds"] = [
911
+ processed_data["input_audio_embeds"] = [
912
audio_features[idx, :size] for idx, size in enumerate(feature_sizes)
913
]
914
+ processed_data.update(passthrough_data)
915
907
- return processed_outputs
916
+ return processed_data
917
918
def _get_mm_fields_config(
919
self,
vllm/model_executor/models/phi4siglip.py
+18
-10
@@ -142,16 +142,22 @@ class Phi4SiglipDummyInputsBuilder(
142
class Phi4SiglipMultiModalProcessor(
143
BaseMultiModalProcessor[Phi4SiglipProcessingInfo],
144
):
145
- def _call_hf_processor(
145
+ def _apply_hf_processor_main(
146
self,
147
- prompt: str,
148
- mm_data: Mapping[str, object],
149
- mm_kwargs: Mapping[str, object],
147
+ mm_items: MultiModalDataItems,
148
+ hf_processor_mm_kwargs: Mapping[str, object],
149
) -> BatchFeature:
151
- processed = super()._call_hf_processor(
152
- prompt=prompt,
153
- mm_data=mm_data,
154
- mm_kwargs=mm_kwargs,
150
+ valid_mm_items = mm_items.select(
151
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
152
+ )
153
+ mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
154
+
155
+ prompt_text = self.dummy_inputs.get_dummy_text(mm_items.get_all_counts())
156
+
157
+ processed = self.info.ctx.call_hf_processor(
158
+ self.info.get_hf_processor(**hf_processor_mm_kwargs),
159
+ dict(text=prompt_text, **mm_data),
160
+ hf_processor_mm_kwargs,
161
)
162
163
# The HF processor's tokenizer_image_token() replaces the "<image>"
@@ -163,10 +169,12 @@ class Phi4SiglipMultiModalProcessor(
169
# NOTE: tokenizer.__call__() (not .encode()) must be used so that
170
# added/special tokens like <|user|>, <|end|> are kept as single IDs.
171
tokenizer = self.info.get_tokenizer()
166
- new_ids = tokenizer(prompt).input_ids
172
+ new_ids = tokenizer(prompt_text).input_ids
173
processed["input_ids"] = torch.tensor([new_ids])
174
169
- return processed
175
+ processed_data = processed
176
+ processed_data.update(passthrough_data)
177
+ return processed_data
178
179
def _get_mm_fields_config(
180
self,
vllm/model_executor/models/pixtral.py
+22
-20
@@ -51,12 +51,10 @@ from vllm.multimodal.processing import BaseDummyInputsBuilder
51
from vllm.multimodal.processing.processor import (
52
BaseMultiModalProcessor,
53
BaseProcessingInfo,
54
- MultiModalProcessingInfo,
54
ProcessorInputs,
55
PromptReplacement,
56
PromptUpdate,
57
PromptUpdateDetails,
59
- TimingContext,
58
)
59
from vllm.platforms import current_platform
60
from vllm.sequence import IntermediateTensors
@@ -213,6 +211,9 @@ class PixtralDummyInputsBuilder(BaseDummyInputsBuilder[PixtralProcessingInfo]):
211
212
213
class PixtralMultiModalProcessor(BaseMultiModalProcessor[PixtralProcessingInfo]):
214
+ # The tokens are already inserted by the chat template
215
+ hf_processor_applies_updates = True
216
+
217
def _get_mm_fields_config(
218
self,
219
hf_inputs: Mapping[str, NestedTensors],
@@ -220,24 +221,35 @@ class PixtralMultiModalProcessor(BaseMultiModalProcessor[PixtralProcessingInfo])
221
) -> Mapping[str, MultiModalFieldConfig]:
222
return dict(images=MultiModalFieldConfig.batched("image"))
223
223
- def _call_hf_processor(
224
+ def _apply_hf_processor_main(
225
self,
225
- prompt: str,
226
- mm_data: Mapping[str, object],
227
- mm_kwargs: Mapping[str, object],
226
+ mm_items: MultiModalDataItems,
227
+ hf_processor_mm_kwargs: Mapping[str, object],
228
) -> BatchFeature:
229
+ valid_mm_items = mm_items.select(
230
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
231
+ )
232
+ mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
233
+
234
+ if not mm_data:
235
+ return BatchFeature(dict(passthrough_data))
236
+
237
+ prompt_text = self.dummy_inputs.get_dummy_text(mm_items.get_all_counts())
238
+
239
outputs = self.info.ctx.call_hf_processor(
230
- self.info.get_hf_processor(**mm_kwargs),
231
- dict(text=prompt, **mm_data),
240
+ self.info.get_hf_processor(**hf_processor_mm_kwargs),
241
+ dict(text=prompt_text, **mm_data),
242
# Avoid padding issue
233
- dict(**mm_kwargs, return_tensors=None),
243
+ dict(**hf_processor_mm_kwargs, return_tensors=None),
244
)
245
246
# Missing batch dimension
247
if is_list_of(outputs["input_ids"], int):
248
outputs["input_ids"] = [outputs["input_ids"]]
249
240
- return outputs
250
+ processed_data = outputs
251
+ processed_data.update(passthrough_data)
252
+ return processed_data
253
254
def _get_prompt_updates(
255
self,
@@ -273,16 +285,6 @@ class PixtralMultiModalProcessor(BaseMultiModalProcessor[PixtralProcessingInfo])
285
),
286
]
287
276
- def _cached_apply_hf_processor(
277
- self,
278
- inputs: ProcessorInputs,
279
- timing_ctx: TimingContext,
280
- ) -> tuple[list[int], MultiModalProcessingInfo, bool]:
281
- prompt_ids, mm_info, _ = super()._cached_apply_hf_processor(inputs, timing_ctx)
282
-
283
- # NOTE: The tokens are already inserted by the chat template
284
- return prompt_ids, mm_info, True
285
-
288
289
@MULTIMODAL_REGISTRY.register_processor(
290
PixtralMultiModalProcessor,
vllm/model_executor/models/qwen2_5_omni_thinker.py
+80
-83
@@ -499,83 +499,6 @@ class Qwen2_5OmniThinkerDummyInputsBuilder(
499
class Qwen2_5OmniThinkerMultiModalProcessor(
500
BaseMultiModalProcessor[Qwen2_5OmniThinkerProcessingInfo]
501
):
502
- def _call_hf_processor(
503
- self,
504
- prompt: str,
505
- mm_data: Mapping[str, object],
506
- mm_kwargs: Mapping[str, object],
507
- ) -> BatchFeature:
508
- mm_data = dict(mm_data)
509
- audios = mm_data.pop("audios", [])
510
-
511
- # NOTE: WhisperFeatureExtractor cannot handle empty list of audios
512
- if audios:
513
- # NOTE: Qwen2.5-Omni processor accept "audio"
514
- mm_data["audio"] = audios
515
- mm_kwargs = dict(
516
- **mm_kwargs,
517
- )
518
- elif (
519
- mm_kwargs.get("use_audio_in_video")
520
- and "audio" not in mm_data
521
- and mm_data.get("videos")
522
- ):
523
- # A subclass (e.g. Qwen3-Omni) may have already popped "audios"
524
- # and populated mm_data["audio"] itself before delegating here,
525
- # in which case this isn't a genuine "no audio" case. Likewise,
526
- # mm_data can be empty on a multimodal-processor-cache hit, where
527
- # there's nothing to (re-)process this call and the real result
528
- # comes from the cache — not a genuine "no audio" case either.
529
- raise ValueError(
530
- "Video doesn't have audio track with `audio_in_video=True`"
531
- )
532
-
533
- merged = self.info.ctx.get_merged_mm_kwargs(mm_kwargs)
534
- if mm_data.get("videos") and (
535
- merged.keys() & {"size", "min_pixels", "max_pixels"}
536
- ):
537
- mm_kwargs = dict(mm_kwargs)
538
- video_size = dict(self.info.get_hf_processor().video_processor.size)
539
- size_override = merged.get("size")
540
- if size_override is not None:
541
- video_size = video_size | size_override
542
- min_pixels = merged.get("min_pixels")
543
- if min_pixels is not None:
544
- video_size["shortest_edge"] = min_pixels
545
- max_pixels = merged.get("max_pixels")
546
- if max_pixels is not None:
547
- video_size["longest_edge"] = max_pixels
548
- mm_kwargs["size"] = video_size
549
-
550
- hf_inputs = super()._call_hf_processor(
551
- prompt=prompt,
552
- mm_data=mm_data,
553
- mm_kwargs=mm_kwargs,
554
- )
555
-
556
- input_features = hf_inputs.pop("input_features", None)
557
- feature_attention_mask = hf_inputs.get("feature_attention_mask", None)
558
- if "input_audio_features" not in hf_inputs and input_features is not None:
559
- if feature_attention_mask is not None:
560
- input_features = input_features.permute(0, 2, 1)[
561
- feature_attention_mask.bool()
562
- ].permute(1, 0)
563
- hf_inputs["input_audio_features"] = input_features
564
- if (
565
- "audio_feature_lengths" not in hf_inputs
566
- and feature_attention_mask is not None
567
- ):
568
- hf_inputs["audio_feature_lengths"] = feature_attention_mask.sum(-1)
569
-
570
- video_second_per_grid = hf_inputs.get("video_second_per_grid", None)
571
- if video_second_per_grid is not None:
572
- hf_inputs["second_per_grid_ts"] = video_second_per_grid
573
-
574
- use_audio_in_video = mm_kwargs.get("use_audio_in_video", False)
575
- hf_inputs["use_audio_in_video"] = torch.tensor(use_audio_in_video)
576
-
577
- return hf_inputs
578
-
502
def _get_mm_fields_config(
503
self,
504
hf_inputs: BatchFeature,
@@ -885,7 +808,7 @@ class Qwen2_5OmniThinkerMultiModalProcessor(
808
),
809
]
810
888
- def _apply_hf_processor_mm_only(
811
+ def _apply_hf_processor_main(
812
self,
813
mm_items: MultiModalDataItems,
814
hf_processor_mm_kwargs: Mapping[str, object],
@@ -903,13 +826,87 @@ class Qwen2_5OmniThinkerMultiModalProcessor(
826
)
827
mm_counts["audio"] -= mm_counts["video"]
828
906
- _, mm_processed_data = self._apply_hf_processor_text_mm(
907
- prompt_text=self.dummy_inputs.get_dummy_text(mm_counts),
908
- mm_items=mm_items,
909
- hf_processor_mm_kwargs=hf_processor_mm_kwargs,
829
+ valid_mm_items = mm_items.select(
830
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
831
)
832
+ processor_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
833
+
834
+ prompt_text = self.dummy_inputs.get_dummy_text(mm_counts)
835
+
836
+ mm_data = dict(processor_data)
837
+ audios = mm_data.pop("audios", [])
838
+
839
+ # NOTE: WhisperFeatureExtractor cannot handle empty list of audios
840
+ if audios:
841
+ # NOTE: Qwen2.5-Omni processor accept "audio"
842
+ mm_data["audio"] = audios
843
+ hf_processor_mm_kwargs = dict(
844
+ **hf_processor_mm_kwargs,
845
+ )
846
+ elif (
847
+ hf_processor_mm_kwargs.get("use_audio_in_video")
848
+ and "audio" not in mm_data
849
+ and mm_data.get("videos")
850
+ ):
851
+ # A subclass (e.g. Qwen3-Omni) may have already popped "audios"
852
+ # and populated mm_data["audio"] itself before delegating here,
853
+ # in which case this isn't a genuine "no audio" case. Likewise,
854
+ # mm_data can be empty on a multimodal-processor-cache hit, where
855
+ # there's nothing to (re-)process this call and the real result
856
+ # comes from the cache — not a genuine "no audio" case either.
857
+ raise ValueError(
858
+ "Video doesn't have audio track with `audio_in_video=True`"
859
+ )
860
+
861
+ merged = self.info.ctx.get_merged_mm_kwargs(hf_processor_mm_kwargs)
862
+ if mm_data.get("videos") and (
863
+ merged.keys() & {"size", "min_pixels", "max_pixels"}
864
+ ):
865
+ hf_processor_mm_kwargs = dict(hf_processor_mm_kwargs)
866
+ video_size = dict(self.info.get_hf_processor().video_processor.size)
867
+ size_override = merged.get("size")
868
+ if size_override is not None:
869
+ video_size = video_size | size_override
870
+ min_pixels = merged.get("min_pixels")
871
+ if min_pixels is not None:
872
+ video_size["shortest_edge"] = min_pixels
873
+ max_pixels = merged.get("max_pixels")
874
+ if max_pixels is not None:
875
+ video_size["longest_edge"] = max_pixels
876
+ hf_processor_mm_kwargs["size"] = video_size
877
+
878
+ hf_inputs = self.info.ctx.call_hf_processor(
879
+ self.info.get_hf_processor(**hf_processor_mm_kwargs),
880
+ dict(text=prompt_text, **mm_data),
881
+ hf_processor_mm_kwargs,
882
+ )
883
+
884
+ input_features = hf_inputs.pop("input_features", None)
885
+ feature_attention_mask = hf_inputs.get("feature_attention_mask", None)
886
+ if "input_audio_features" not in hf_inputs and input_features is not None:
887
+ if feature_attention_mask is not None:
888
+ input_features = input_features.permute(0, 2, 1)[
889
+ feature_attention_mask.bool()
890
+ ].permute(1, 0)
891
+ hf_inputs["input_audio_features"] = input_features
892
+ if (
893
+ "audio_feature_lengths" not in hf_inputs
894
+ and feature_attention_mask is not None
895
+ ):
896
+ hf_inputs["audio_feature_lengths"] = feature_attention_mask.sum(-1)
897
+
898
+ video_second_per_grid = hf_inputs.get("video_second_per_grid", None)
899
+ if video_second_per_grid is not None:
900
+ hf_inputs["second_per_grid_ts"] = video_second_per_grid
901
+
902
+ use_audio_in_video = hf_processor_mm_kwargs.get("use_audio_in_video", False)
903
+ hf_inputs["use_audio_in_video"] = torch.tensor(use_audio_in_video)
904
+
905
+ processed_data = hf_inputs
906
+ processed_data.update(passthrough_data)
907
+ processed_data.pop("input_ids")
908
912
- return mm_processed_data
909
+ return processed_data
910
911
912
class Qwen2_5OmniConditionalGenerationMixin:
vllm/model_executor/models/qwen2_5_vl.py
-10
@@ -1154,16 +1154,6 @@ class Qwen2_5_VLMultiModalProcessor(Qwen2VLMultiModalProcessor):
1154
second_per_grid_ts=MultiModalFieldConfig.batched("video", keep_on_cpu=True),
1155
)
1156
1157
- def _call_hf_processor(
1158
- self,
1159
- prompt: str,
1160
- mm_data: Mapping[str, object],
1161
- mm_kwargs: Mapping[str, object],
1162
- ) -> BatchFeature:
1163
- # Override to use the text path instead of token path to use the
1164
- # video-specific logic in processing_qwen2_5_vl.py
1165
- return super()._call_hf_processor(prompt, mm_data, mm_kwargs)
1166
-
1157
def _get_prompt_updates(
1158
self,
1159
mm_items: MultiModalDataItems,
vllm/model_executor/models/qwen2_audio.py
+12
-21
@@ -235,36 +235,27 @@ class Qwen2AudioDummyInputsBuilder(BaseDummyInputsBuilder[Qwen2AudioProcessingIn
235
236
237
class Qwen2AudioMultiModalProcessor(BaseMultiModalProcessor[Qwen2AudioProcessingInfo]):
238
- def _call_hf_processor(
238
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
239
+ return self.dummy_inputs.get_dummy_text(mm_counts)
240
+
241
+ def _preprocess_hf_mm_data(
242
self,
240
- prompt: str,
243
mm_data: Mapping[str, object],
242
- mm_kwargs: Mapping[str, Any],
243
- ) -> BatchFeature:
244
- # NOTE - we rename audios -> audio in mm data because transformers has
245
- # deprecated audios for the qwen2audio processor and will remove
246
- # support for it in transformers 4.54.
244
+ hf_processor_mm_kwargs: Mapping[str, object],
245
+ ) -> tuple[Mapping[str, object], Mapping[str, object]]:
246
+ feature_extractor = self.info.get_feature_extractor(**hf_processor_mm_kwargs)
247
+
248
+ mm_data = dict(mm_data)
249
audios = mm_data.pop("audios", [])
250
if audios:
251
mm_data["audio"] = audios
252
251
- # Text-only input not supported in composite processor
252
- if not mm_data.get("audio", []):
253
- prompt_ids = self.info.get_tokenizer().encode(prompt)
254
- prompt_ids = self._apply_hf_processor_tokens_only(prompt_ids)
255
- return BatchFeature(dict(input_ids=[prompt_ids]), tensor_type="pt")
256
-
257
- feature_extractor = self.info.get_feature_extractor(**mm_kwargs)
258
- mm_kwargs = dict(
259
- **mm_kwargs,
253
+ hf_processor_mm_kwargs = dict(
254
+ **hf_processor_mm_kwargs,
255
sampling_rate=feature_extractor.sampling_rate,
256
)
257
263
- return super()._call_hf_processor(
264
- prompt=prompt,
265
- mm_data=mm_data,
266
- mm_kwargs=mm_kwargs,
267
- )
258
+ return mm_data, hf_processor_mm_kwargs
259
260
def _get_mm_fields_config(
261
self,
vllm/model_executor/models/qwen3_omni_moe_thinker.py
+71
-55
@@ -1195,79 +1195,95 @@ Qwen3OmniMoeThinkerDummyInputsBuilder = Qwen2_5OmniThinkerDummyInputsBuilder
1195
class Qwen3OmniMoeThinkerMultiModalProcessor(
1196
Qwen2_5OmniThinkerMultiModalProcessor,
1197
):
1198
- def _call_hf_processor(
1198
+ def _get_hf_mm_data(
1199
self,
1200
- prompt: str,
1201
- mm_data: Mapping[str, object],
1202
- mm_kwargs: Mapping[str, object],
1203
- ) -> BatchFeature:
1204
- mm_data = dict(mm_data)
1205
- audios = mm_data.pop("audios", [])
1206
-
1207
- def pad_to_hop_length(x: np.ndarray, hop_length: int) -> np.ndarray:
1208
- length = x.shape[-1]
1209
- if length % hop_length != 0:
1210
- pad_length = hop_length - (length % hop_length)
1211
- x = np.pad(x, (0, pad_length), mode="constant", constant_values=0)
1212
- return x
1213
-
1214
- # NOTE: WhisperFeatureExtractor cannot handle empty list of audios
1215
- feature_extractor = self.info.get_feature_extractor(**mm_kwargs)
1216
- hop_length = feature_extractor.hop_length
1200
+ mm_items: MultiModalDataItems,
1201
+ ) -> tuple[Mapping[str, object], Mapping[str, object]]:
1202
+ processor_data, passthrough_data = super()._get_hf_mm_data(mm_items)
1203
+
1204
+ audios = processor_data.get("audios")
1205
if audios:
1218
- # NOTE: Qwen3-Omni processor accept "audio"
1206
+ feature_extractor = self.info.get_feature_extractor()
1207
+ hop_length = feature_extractor.hop_length
1208
+
1209
+ def pad_to_hop_length(x: np.ndarray) -> np.ndarray:
1210
+ length = x.shape[-1]
1211
+ if length % hop_length != 0:
1212
+ pad_length = hop_length - (length % hop_length)
1213
+ x = np.pad(x, (0, pad_length), mode="constant", constant_values=0)
1214
+ return x
1215
+
1216
# To make sure the cache works with padding=True, we pre-padded
1220
- # the audio to multiple of hop_length.
1221
- mm_data["audio"] = [
1222
- pad_to_hop_length(audio, hop_length)
1217
+ # the audio to a multiple of hop_length.
1218
+ processor_data = dict(processor_data)
1219
+ processor_data["audios"] = [
1220
+ pad_to_hop_length(audio)
1221
if isinstance(audio, np.ndarray)
1224
- else (pad_to_hop_length(audio[0], hop_length), audio[1])
1222
+ else (pad_to_hop_length(audio[0]), audio[1])
1223
for audio in audios
1224
]
1225
1226
+ return processor_data, passthrough_data
1227
+
1228
+ def _apply_hf_processor_main(
1229
+ self,
1230
+ mm_items: MultiModalDataItems,
1231
+ hf_processor_mm_kwargs: Mapping[str, object],
1232
+ ) -> BatchFeature:
1233
+ if mm_items.get_all_counts().get("audio", 0):
1234
# TODO(Isotr0py): Remove this patch after upstream fix PR
1235
# released and Transformers version update:
1236
# https://github.com/huggingface/transformers/pull/41473
1231
- mm_kwargs = dict(mm_kwargs)
1232
- mm_kwargs["audio_kwargs"] = dict(mm_kwargs.get("audio_kwargs") or {})
1233
- mm_kwargs["text_kwargs"] = dict(mm_kwargs.get("text_kwargs") or {})
1234
- elif mm_kwargs.get("use_audio_in_video") and mm_data.get("videos"):
1235
- # mm_data can be empty on a multimodal-processor-cache hit, where
1236
- # there's nothing to (re-)process this call and the real result
1237
- # comes from the cache — not a genuine "no audio" case.
1238
- raise ValueError(
1239
- "Video doesn't have audio track with `audio_in_video=True`"
1237
+ hf_processor_mm_kwargs = dict(hf_processor_mm_kwargs)
1238
+ hf_processor_mm_kwargs["audio_kwargs"] = dict(
1239
+ hf_processor_mm_kwargs.get("audio_kwargs") or {}
1240
+ )
1241
+ hf_processor_mm_kwargs["text_kwargs"] = dict(
1242
+ hf_processor_mm_kwargs.get("text_kwargs") or {}
1243
)
1244
1242
- hf_inputs = super()._call_hf_processor(
1243
- prompt=prompt,
1244
- mm_data=mm_data,
1245
- mm_kwargs=mm_kwargs,
1245
+ processed_data = super()._apply_hf_processor_main(
1246
+ mm_items,
1247
+ hf_processor_mm_kwargs,
1248
)
1249
1250
if (
1249
- "audio_feature_lengths" in hf_inputs
1250
- and "feature_attention_mask" in hf_inputs
1251
- and (audios := mm_data.get("audio", []))
1251
+ "audio_feature_lengths" in processed_data
1252
+ and "feature_attention_mask" in processed_data
1253
):
1253
- audio_num_frames = []
1254
- for _, audio in enumerate(audios):
1255
- audio_length = len(audio[0]) if isinstance(audio, tuple) else len(audio)
1256
- num_frame = (
1257
- (audio_length // hop_length)
1258
- if audio_length % hop_length == 0
1259
- else (audio_length // hop_length - 1)
1254
+ valid_mm_items = mm_items.select(
1255
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
1256
+ )
1257
+ processor_data, _ = self._get_hf_mm_data(valid_mm_items)
1258
+ audios = processor_data.get("audios", [])
1259
+ if audios:
1260
+ feature_extractor = self.info.get_feature_extractor(
1261
+ **hf_processor_mm_kwargs
1262
)
1261
- if mm_kwargs.get("truncation", False):
1262
- num_frame = min(
1263
- num_frame, feature_extractor.n_samples // hop_length
1263
+ hop_length = feature_extractor.hop_length
1264
+
1265
+ audio_num_frames = []
1266
+ for audio in audios:
1267
+ audio_length = (
1268
+ len(audio[0]) if isinstance(audio, tuple) else len(audio)
1269
)
1265
- audio_num_frames.append(num_frame)
1266
- hf_inputs["feature_attention_mask"] = [
1267
- torch.ones(num_frame) for num_frame in audio_num_frames
1268
- ]
1269
- hf_inputs["audio_feature_lengths"] = torch.tensor(audio_num_frames)
1270
- return hf_inputs
1270
+ num_frame = (
1271
+ (audio_length // hop_length)
1272
+ if audio_length % hop_length == 0
1273
+ else (audio_length // hop_length - 1)
1274
+ )
1275
+ if hf_processor_mm_kwargs.get("truncation", False):
1276
+ num_frame = min(
1277
+ num_frame, feature_extractor.n_samples // hop_length
1278
+ )
1279
+ audio_num_frames.append(num_frame)
1280
+
1281
+ processed_data["feature_attention_mask"] = [
1282
+ torch.ones(num_frame) for num_frame in audio_num_frames
1283
+ ]
1284
+ processed_data["audio_feature_lengths"] = torch.tensor(audio_num_frames)
1285
+
1286
+ return processed_data
1287
1288
def _maybe_apply_prompt_updates(
1289
self,
vllm/model_executor/models/qwen3_vl.py
+35
-22
@@ -1280,12 +1280,21 @@ class Qwen3VLMultiModalProcessor(BaseMultiModalProcessor[Qwen3VLProcessingInfo])
1280
proc_impl = getattr(type(hf_processor), "replace_video_token", None)
1281
return proc_impl is not None and proc_impl is not mixin_impl
1282
1283
- def _call_hf_processor(
1283
+ def _apply_hf_processor_main(
1284
self,
1285
- prompt: str,
1286
- mm_data: Mapping[str, object],
1287
- mm_kwargs: Mapping[str, object],
1285
+ mm_items: MultiModalDataItems,
1286
+ hf_processor_mm_kwargs: Mapping[str, object],
1287
) -> BatchFeature:
1288
+ valid_mm_items = mm_items.select(
1289
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
1290
+ )
1291
+ mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
1292
+
1293
+ if not mm_data:
1294
+ return BatchFeature(dict(passthrough_data))
1295
+
1296
+ prompt_text = self.dummy_inputs.get_dummy_text(mm_items.get_all_counts())
1297
+
1298
mm_data = dict(mm_data)
1299
1300
# Separate video processing from image processing. Because the videos
@@ -1314,8 +1323,8 @@ class Qwen3VLMultiModalProcessor(BaseMultiModalProcessor[Qwen3VLProcessingInfo])
1323
1324
# NOTE: a copy of is created to update do_sample_frames,
1325
# otherwise mm_hash for the object will be incorrect.
1317
- video_mm_kwargs = dict(**mm_kwargs)
1318
- merged = self.info.ctx.get_merged_mm_kwargs(mm_kwargs)
1326
+ video_mm_kwargs = dict(**hf_processor_mm_kwargs)
1327
+ merged = self.info.ctx.get_merged_mm_kwargs(hf_processor_mm_kwargs)
1328
if merged.keys() & {"size", "min_pixels", "max_pixels"}:
1329
video_size = dict(self.info.get_video_processor().size)
1330
size_override = merged.get("size")
@@ -1365,10 +1374,13 @@ class Qwen3VLMultiModalProcessor(BaseMultiModalProcessor[Qwen3VLProcessingInfo])
1374
if "num_frames" in video_mm_kwargs and "fps" not in video_mm_kwargs:
1375
video_mm_kwargs["fps"] = None
1376
1368
- video_outputs = super()._call_hf_processor(
1369
- prompt="<|vision_start|><|video_pad|><|vision_end|>",
1370
- mm_data=video_mm_data,
1371
- mm_kwargs=video_mm_kwargs,
1377
+ video_outputs = self.info.ctx.call_hf_processor(
1378
+ self.info.get_hf_processor(**video_mm_kwargs),
1379
+ dict(
1380
+ text="<|vision_start|><|video_pad|><|vision_end|>",
1381
+ **video_mm_data,
1382
+ ),
1383
+ video_mm_kwargs,
1384
)
1385
1386
# Discard HF output input_ids — we use get_video_repl below
@@ -1424,12 +1436,14 @@ class Qwen3VLMultiModalProcessor(BaseMultiModalProcessor[Qwen3VLProcessingInfo])
1436
# fps/num_frames are video-only kwargs already consumed by the loop;
1437
# exclude them so the text/image processor call below never gets a list.
1438
non_video_mm_kwargs = {
1427
- k: v for k, v in mm_kwargs.items() if k not in ("fps", "num_frames")
1439
+ k: v
1440
+ for k, v in hf_processor_mm_kwargs.items()
1441
+ if k not in ("fps", "num_frames")
1442
}
1429
- processed_outputs = super()._call_hf_processor(
1430
- prompt=prompt,
1431
- mm_data=mm_data,
1432
- mm_kwargs=non_video_mm_kwargs,
1443
+ processed_data = self.info.ctx.call_hf_processor(
1444
+ self.info.get_hf_processor(**non_video_mm_kwargs),
1445
+ dict(text=prompt_text, **mm_data),
1446
+ non_video_mm_kwargs,
1447
)
1448
1449
# Replace each placeholder with pre-computed video tokens.
@@ -1443,20 +1457,19 @@ class Qwen3VLMultiModalProcessor(BaseMultiModalProcessor[Qwen3VLProcessingInfo])
1457
hf_config.video_token_id,
1458
hf_config.vision_end_token_id,
1459
]
1446
- input_ids = processed_outputs.pop("input_ids")
1460
+ input_ids = processed_data.pop("input_ids")
1461
if not isinstance(input_ids, list):
1462
input_ids = input_ids.tolist()
1463
(prompt_ids,) = input_ids
1464
expanded_ids = _replace_video_token_placeholders(
1465
prompt_ids, video_target, video_input_ids_lst
1466
)
1453
- processed_outputs["input_ids"] = [expanded_ids]
1467
+ processed_data["input_ids"] = [expanded_ids]
1468
1455
- combined_outputs = dict(
1456
- processed_outputs,
1457
- **video_outputs,
1458
- )
1459
- return BatchFeature(combined_outputs)
1469
+ processed_data.update(video_outputs)
1470
+ processed_data.update(passthrough_data)
1471
+
1472
+ return processed_data
1473
1474
def _get_mm_fields_config(
1475
self,
vllm/model_executor/models/transformers/multimodal.py
+78
-53
@@ -345,28 +345,6 @@ class _MultiModalProcessorBase(BaseMultiModalProcessor[MultiModalProcessingInfo]
345
346
return mm_fields
347
348
- def _call_hf_processor(
349
- self,
350
- prompt: str,
351
- mm_data: Mapping[str, object],
352
- mm_kwargs: Mapping[str, object],
353
- ) -> "BatchFeature":
354
- """Run the HF processor and unpad inputs."""
355
- try:
356
- hf_inputs = super()._call_hf_processor(prompt, mm_data, mm_kwargs)
357
- except ValueError:
358
- if any(mm_data.values()):
359
- raise
360
- # Some processors reject a prompt holding placeholders with
361
- # no data to go with them, so tokenize it without them
362
- prompt_ids = self.info.get_tokenizer().encode(prompt)
363
- hf_inputs = transformers.BatchFeature(
364
- dict(input_ids=[prompt_ids]), tensor_type="pt"
365
- )
366
- self._unpad_images(hf_inputs)
367
- self._unpad_audios(hf_inputs, mm_data, mm_kwargs)
368
- return hf_inputs
369
-
348
def _unpad_audios(
349
self,
350
hf_inputs: "BatchFeature",
@@ -450,19 +428,6 @@ class LegacyMultiModalProcessor(_MultiModalProcessorBase):
428
deriving them from updates."""
429
return []
430
453
- def _call_hf_processor(
454
- self,
455
- prompt: str,
456
- mm_data: Mapping[str, object],
457
- mm_kwargs: Mapping[str, object],
458
- ) -> "BatchFeature":
459
- """Ask which modality owns each token of the expanded prompt, which is the
460
- only marking of the tokens an expansion adds around an item."""
461
- if any(mm_data.values()):
462
- mm_data = {**mm_data, "return_mm_token_type_ids": True}
463
-
464
- return super()._call_hf_processor(prompt, mm_data, mm_kwargs)
465
-
431
def _get_mm_token_ids(self, modality: str) -> list[int]:
432
"""Token ids marking where `modality` sits in the prompt, which for some
433
processors differ from the placeholder written into it.
@@ -612,24 +577,55 @@ class LegacyMultiModalProcessor(_MultiModalProcessorBase):
577
578
with timing_ctx.record("apply_hf_processor"):
579
hf_processor = self.info.get_hf_processor(**hf_processor_mm_kwargs)
615
- prompt = hf_processor.decode(prompt)
580
+ prompt_text = hf_processor.decode(prompt)
581
582
# Bypass cached processor and always apply to the full set of mm inputs
583
# NOTE: we can't just set caching=False because base class method
584
# transforms outputs to `MultiModalKwargs` which is not going to
585
# work for Transformers. The vision path has logic tied to
586
# `mm_tokens_per_modality` in _apply_vision()
622
- prompt_ids, processed_data = self._apply_hf_processor_text_mm(
623
- prompt_text=prompt,
624
- mm_items=mm_items,
625
- hf_processor_mm_kwargs={
626
- **hf_processor_mm_kwargs,
627
- # HF processors only accept text, and the decoded string already
628
- # contains any special tokens, so don't let them be added again
629
- # (`_call_hf_processor` disables `add_special_tokens`).
630
- "add_special_tokens": False,
631
- },
587
+ hf_processor_mm_kwargs = {
588
+ **hf_processor_mm_kwargs,
589
+ # HF processors only accept text, and the decoded string already
590
+ # contains any special tokens, so don't let them be added again
591
+ # (the HF processor call disables `add_special_tokens`).
592
+ "add_special_tokens": False,
593
+ }
594
+
595
+ valid_mm_items = mm_items.select(
596
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
597
)
598
+ processor_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
599
+
600
+ # Ask which modality owns each token of the expanded prompt, which
601
+ # is the only marking of the tokens an expansion adds around an item
602
+ if any(processor_data.values()):
603
+ processor_data = {**processor_data, "return_mm_token_type_ids": True}
604
+
605
+ try:
606
+ processed_data = self.info.ctx.call_hf_processor(
607
+ self.info.get_hf_processor(**hf_processor_mm_kwargs),
608
+ dict(text=prompt_text, **processor_data),
609
+ hf_processor_mm_kwargs,
610
+ )
611
+ except ValueError:
612
+ if any(processor_data.values()):
613
+ raise
614
+ # Some processors reject a prompt holding placeholders with
615
+ # no data to go with them, so tokenize it without them
616
+ prompt_ids = self.info.get_tokenizer().encode(prompt_text)
617
+ processed_data = transformers.BatchFeature(
618
+ dict(input_ids=[prompt_ids]), tensor_type="pt"
619
+ )
620
+ self._unpad_images(processed_data)
621
+ self._unpad_audios(processed_data, processor_data, hf_processor_mm_kwargs)
622
+ processed_data.update(passthrough_data)
623
+
624
+ input_ids = processed_data.pop("input_ids")
625
+ if not isinstance(input_ids, list):
626
+ input_ids = input_ids.tolist()
627
+
628
+ (prompt_ids,) = input_ids
629
630
# Use overrides if provided; fallback to data-dependent hashing.
631
with timing_ctx.record("get_mm_hashes"):
@@ -773,17 +769,41 @@ class OffsetsMultiModalProcessor(_MultiModalProcessorBase):
769
except (AttributeError, KeyError, TypeError):
770
return None
771
776
- def _call_hf_processor(
772
+ def _apply_hf_processor_main(
773
self,
778
- prompt: str,
779
- mm_data: Mapping[str, object],
780
- mm_kwargs: Mapping[str, object],
774
+ mm_items: MultiModalDataItems,
775
+ hf_processor_mm_kwargs: Mapping[str, object],
776
) -> "BatchFeature":
782
- """Ask for the replacement each placeholder expands to, and record it as
783
- per-item fields: its token ids, and the tokens or patches behind them."""
777
+ mm_counts = mm_items.get_all_counts()
778
+
779
+ valid_mm_items = mm_items.select({k for k, c in mm_counts.items() if c > 0})
780
+ mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
781
+
782
+ prompt_text = self.dummy_inputs.get_dummy_text(mm_counts)
783
+
784
+ # Ask for the replacement each placeholder expands to, and record it as
785
+ # per-item fields: its token ids, and the tokens or patches behind them
786
if has_mm_data := any(mm_data.values()):
787
mm_data = {**mm_data, "return_text_replacement_offsets": True}
786
- hf_inputs = super()._call_hf_processor(prompt, mm_data, mm_kwargs)
788
+
789
+ try:
790
+ hf_inputs = self.info.ctx.call_hf_processor(
791
+ self.info.get_hf_processor(**hf_processor_mm_kwargs),
792
+ dict(text=prompt_text, **mm_data),
793
+ hf_processor_mm_kwargs,
794
+ )
795
+ except ValueError:
796
+ if any(mm_data.values()):
797
+ raise
798
+ # Some processors reject a prompt holding placeholders with
799
+ # no data to go with them, so tokenize it without them
800
+ tokenizer = self.info.get_tokenizer()
801
+ hf_inputs = transformers.BatchFeature(
802
+ dict(input_ids=[tokenizer.encode(prompt_text)]), tensor_type="pt"
803
+ )
804
+ self._unpad_images(hf_inputs)
805
+ self._unpad_audios(hf_inputs, mm_data, hf_processor_mm_kwargs)
806
+
807
# Drop the inputs the model would reject
808
hf_inputs.pop("mm_token_type_ids", None)
809
hf_inputs.pop("token_type_ids", None)
@@ -802,6 +822,8 @@ class OffsetsMultiModalProcessor(_MultiModalProcessorBase):
822
"so it can be fixed, and install `transformers<5.15.0` in the "
823
"meantime to locate placeholders in the expanded prompt instead."
824
)
825
+ hf_inputs.update(passthrough_data)
826
+ hf_inputs.pop("input_ids", None)
827
return hf_inputs
828
829
tokenizer = self.info.get_tokenizer()
@@ -829,6 +851,9 @@ class OffsetsMultiModalProcessor(_MultiModalProcessorBase):
851
counts.append(int(ids.eq(_get_embed_token_id(ids)).sum()))
852
hf_inputs["num_audio_tokens"] = torch.tensor(counts)
853
854
+ hf_inputs.update(passthrough_data)
855
+ hf_inputs.pop("input_ids")
856
+
857
return hf_inputs
858
859
vllm/model_executor/models/ultravox.py
+67
-24
@@ -190,41 +190,84 @@ class UltravoxDummyInputsBuilder(BaseDummyInputsBuilder[UltravoxProcessingInfo])
190
191
192
class UltravoxMultiModalProcessor(BaseMultiModalProcessor[UltravoxProcessingInfo]):
193
- def _call_hf_processor(
193
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
194
+ return self.dummy_inputs.get_dummy_text(mm_counts)
195
+
196
+ def _preprocess_hf_mm_data(
197
self,
195
- prompt: str,
198
mm_data: Mapping[str, object],
197
- mm_kwargs: Mapping[str, object],
198
- ) -> BatchFeature:
199
- # Text-only input not supported in composite processor
200
- if not mm_data.get("audios", []):
201
- prompt_ids = self.info.get_tokenizer().encode(
202
- prompt, add_special_tokens=False
203
- )
204
- prompt_ids = self._apply_hf_processor_tokens_only(prompt_ids)
205
- return BatchFeature(dict(input_ids=[prompt_ids]), tensor_type="pt")
206
-
207
- mm_data = dict(mm_data)
208
- audios = mm_data.pop("audios", [])
199
+ hf_processor_mm_kwargs: Mapping[str, object],
200
+ ) -> tuple[Mapping[str, object], Mapping[str, object]]:
201
+ audios = mm_data.get("audios", [])
202
assert isinstance(audios, list)
203
211
- feature_extractor = self.info.get_feature_extractor(**mm_kwargs)
212
- mm_kwargs = dict(
213
- **mm_kwargs,
204
+ feature_extractor = self.info.get_feature_extractor(**hf_processor_mm_kwargs)
205
+ hf_processor_mm_kwargs = dict(
206
+ **hf_processor_mm_kwargs,
207
sampling_rate=feature_extractor.sampling_rate,
208
include_audio_num_chunks=True,
209
)
210
218
- item_processor_data = dict(**mm_data, audios=audios)
211
+ return mm_data, hf_processor_mm_kwargs
212
220
- output = super()._call_hf_processor(
221
- prompt=prompt,
222
- mm_data=item_processor_data,
223
- mm_kwargs=mm_kwargs,
213
+ def _apply_hf_processor_main(
214
+ self,
215
+ mm_items: MultiModalDataItems,
216
+ hf_processor_mm_kwargs: Mapping[str, object],
217
+ ) -> BatchFeature:
218
+ valid_mm_items = mm_items.select(
219
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
220
)
225
- output["audio_features"] = output.pop("audio_values")
221
+ processor_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
222
+
223
+ if processor_data:
224
+ processor_data, hf_processor_mm_kwargs = self._preprocess_hf_mm_data(
225
+ processor_data,
226
+ hf_processor_mm_kwargs,
227
+ )
228
+
229
+ prompt_text = self._get_hf_processor_text(mm_items.get_all_counts())
230
+ if prompt_text is not None:
231
+ processor_data = dict(text=prompt_text, **processor_data)
232
+
233
+ hf_processor = self.info.get_hf_processor(**hf_processor_mm_kwargs)
234
+
235
+ def patched_call(**kwargs) -> BatchFeature:
236
+ # The remote UltravoxProcessor does not support the
237
+ # `truncation` kwarg injected by `call_hf_processor`: it
238
+ # passes `truncation` to its audio processor by itself while
239
+ # also forwarding `**kwargs` there, causing a duplicate
240
+ # keyword argument error. Its hardcoded `truncation=False`
241
+ # already matches vLLM's intended behavior.
242
+ kwargs.pop("truncation", None)
243
+
244
+ return hf_processor(**kwargs)
245
+
246
+ processed_data = self.info.ctx.call_hf_processor(
247
+ patched_call,
248
+ processor_data,
249
+ hf_processor_mm_kwargs,
250
+ )
251
+ processed_data.update(passthrough_data)
252
+ else:
253
+ processed_data = BatchFeature(dict(passthrough_data))
254
+
255
+ return self._postprocess_hf_mm_data(
256
+ processor_data,
257
+ hf_processor_mm_kwargs,
258
+ processed_data,
259
+ )
260
+
261
+ def _postprocess_hf_mm_data(
262
+ self,
263
+ mm_data: Mapping[str, object],
264
+ hf_processor_mm_kwargs: Mapping[str, object],
265
+ processed_data: BatchFeature,
266
+ ) -> BatchFeature:
267
+ if "audio_values" in processed_data:
268
+ processed_data["audio_features"] = processed_data.pop("audio_values")
269
227
- return output
270
+ return processed_data
271
272
def _get_mm_fields_config(
273
self,
vllm/model_executor/models/unlimited_ocr.py
+1
-1
@@ -225,7 +225,7 @@ class UnlimitedOCRMultiModalProcessor(DeepseekOCRMultiModalProcessor):
225
self,
226
inputs: ProcessorInputs,
227
timing_ctx: TimingContext,
228
- ) -> tuple[list[int], MultiModalProcessingInfo, bool]:
228
+ ) -> MultiModalProcessingInfo:
229
# The processor logic differs for single-image (crop) vs multi-image
230
# (no crop) requests. The processing cache assumes per-item output is
231
# invariant of how many images are passed per prompt, so we only cache
vllm/model_executor/models/voxtral.py
+23
-25
@@ -47,12 +47,10 @@ from vllm.multimodal.processing import BaseDummyInputsBuilder
47
from vllm.multimodal.processing.processor import (
48
BaseMultiModalProcessor,
49
BaseProcessingInfo,
50
- MultiModalProcessingInfo,
50
PlaceholderFeaturesInfo,
51
ProcessorInputs,
52
PromptReplacement,
53
PromptUpdate,
55
- TimingContext,
54
)
55
from vllm.sequence import IntermediateTensors
56
from vllm.tokenizers import cached_tokenizer_from_config
@@ -204,6 +202,9 @@ class VoxtralDummyInputsBuilder(BaseDummyInputsBuilder[VoxtralProcessingInfo]):
202
203
204
class VoxtralMultiModalProcessor(BaseMultiModalProcessor[VoxtralProcessingInfo]):
205
+ # The tokens are already inserted by the chat template
206
+ hf_processor_applies_updates = True
207
+
208
def _get_mm_fields_config(
209
self,
210
hf_inputs: Mapping[str, NestedTensors],
@@ -220,12 +221,14 @@ class VoxtralMultiModalProcessor(BaseMultiModalProcessor[VoxtralProcessingInfo])
221
# skip validation here
222
pass
223
223
- def _call_hf_processor(
224
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
225
+ return self.dummy_inputs.get_dummy_text(mm_counts)
226
+
227
+ def _preprocess_hf_mm_data(
228
self,
225
- prompt: str,
229
mm_data: Mapping[str, object],
227
- mm_kwargs: Mapping[str, object],
228
- ) -> BatchFeature:
230
+ hf_processor_mm_kwargs: Mapping[str, object],
231
+ ) -> tuple[Mapping[str, object], Mapping[str, object]]:
232
mm_data = dict(mm_data)
233
audios = mm_data.pop("audios", [])
234
@@ -233,18 +236,23 @@ class VoxtralMultiModalProcessor(BaseMultiModalProcessor[VoxtralProcessingInfo])
236
# MistralCommonVoxtralProcessor accepts "audio"
237
mm_data["audio"] = audios
238
236
- outputs = self.info.ctx.call_hf_processor(
237
- self.info.get_hf_processor(**mm_kwargs),
238
- dict(text=prompt, **mm_data),
239
- # Avoid padding issue
240
- dict(**mm_kwargs, return_tensors=None),
241
- )
239
+ # Avoid padding issue
240
+ hf_processor_mm_kwargs = dict(**hf_processor_mm_kwargs, return_tensors=None)
241
+
242
+ return mm_data, hf_processor_mm_kwargs
243
244
+ def _postprocess_hf_mm_data(
245
+ self,
246
+ mm_data: Mapping[str, object],
247
+ hf_processor_mm_kwargs: Mapping[str, object],
248
+ processed_data: BatchFeature,
249
+ ) -> BatchFeature:
250
# Missing batch dimension
244
- if is_list_of(outputs["input_ids"], int):
245
- outputs["input_ids"] = [outputs["input_ids"]]
251
+ input_ids = processed_data.get("input_ids")
252
+ if is_list_of(input_ids, int):
253
+ processed_data["input_ids"] = [input_ids]
254
247
- return outputs
255
+ return processed_data
256
257
def _get_prompt_updates(
258
self,
@@ -287,16 +295,6 @@ class VoxtralMultiModalProcessor(BaseMultiModalProcessor[VoxtralProcessingInfo])
295
),
296
]
297
290
- def _cached_apply_hf_processor(
291
- self,
292
- inputs: ProcessorInputs,
293
- timing_ctx: TimingContext,
294
- ) -> tuple[list[int], MultiModalProcessingInfo, bool]:
295
- prompt_ids, mm_info, _ = super()._cached_apply_hf_processor(inputs, timing_ctx)
296
-
297
- # NOTE: The tokens are already inserted by the chat template
298
- return prompt_ids, mm_info, True
299
-
298
299
@MULTIMODAL_REGISTRY.register_processor(
300
VoxtralMultiModalProcessor,
vllm/model_executor/models/whisper.py
+27
-18
@@ -756,27 +756,36 @@ class WhisperMultiModalProcessor(EncDecMultiModalProcessor[WhisperProcessingInfo
756
# for encoder profiling.
757
return [0]
758
759
- def _call_hf_processor(
759
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
760
+ return self.dummy_inputs.get_dummy_text(mm_counts)
761
+
762
+ def _preprocess_hf_mm_data(
763
self,
761
- prompt: str,
764
mm_data: Mapping[str, object],
763
- mm_kwargs: Mapping[str, object],
764
- ) -> BatchFeature:
765
- if mm_data:
766
- feature_extractor = self.info.get_feature_extractor(**mm_kwargs)
767
- mm_data = dict(audio=mm_data.pop("audios"))
768
- mm_kwargs = dict(
769
- **mm_kwargs,
770
- sampling_rate=feature_extractor.sampling_rate,
771
- )
772
- processed_outputs = super()._call_hf_processor(
773
- prompt=prompt,
774
- mm_data=mm_data,
775
- mm_kwargs=mm_kwargs,
765
+ hf_processor_mm_kwargs: Mapping[str, object],
766
+ ) -> tuple[Mapping[str, object], Mapping[str, object]]:
767
+ feature_extractor = self.info.get_feature_extractor(**hf_processor_mm_kwargs)
768
+
769
+ mm_data = dict(mm_data)
770
+ mm_data["audio"] = mm_data.pop("audios")
771
+
772
+ hf_processor_mm_kwargs = dict(
773
+ **hf_processor_mm_kwargs,
774
+ sampling_rate=feature_extractor.sampling_rate,
775
)
777
- if "labels" in processed_outputs:
778
- processed_outputs["input_ids"] = processed_outputs.pop("labels")
779
- return processed_outputs
776
+
777
+ return mm_data, hf_processor_mm_kwargs
778
+
779
+ def _postprocess_hf_mm_data(
780
+ self,
781
+ mm_data: Mapping[str, object],
782
+ hf_processor_mm_kwargs: Mapping[str, object],
783
+ processed_data: BatchFeature,
784
+ ) -> BatchFeature:
785
+ if "labels" in processed_data:
786
+ processed_data["input_ids"] = processed_data.pop("labels")
787
+
788
+ return processed_data
789
790
def _get_mm_fields_config(
791
self,
vllm/models/dots3_note/common/processor.py
+2
-7
@@ -649,13 +649,8 @@ class Dots3NoteMultiModalProcessor(BaseMultiModalProcessor[Dots3NoteProcessingIn
649
processor_data["videos"] = raw_videos
650
return processor_data, passthrough_data
651
652
- def _call_hf_processor(
653
- self,
654
- prompt: str,
655
- mm_data: Mapping[str, object],
656
- mm_kwargs: Mapping[str, object],
657
- ) -> BatchFeature:
658
- return super()._call_hf_processor(prompt, mm_data, mm_kwargs)
652
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
653
+ return self.dummy_inputs.get_dummy_text(mm_counts)
654
655
def _get_mm_fields_config(
656
self,
vllm/models/inkling/common/mm_preprocess.py
+18
-14
@@ -22,7 +22,7 @@ from vllm.multimodal.inputs import (
22
MultiModalFieldConfig,
23
MultiModalKwargsItems,
24
)
25
-from vllm.multimodal.parse import MultiModalDataParser
25
+from vllm.multimodal.parse import MultiModalDataItems, MultiModalDataParser
26
from vllm.multimodal.processing import (
27
BaseDummyInputsBuilder,
28
BaseMultiModalProcessor,
@@ -135,8 +135,8 @@ class InklingDummyInputsBuilder(BaseDummyInputsBuilder[InklingProcessingInfo]):
135
num_images = mm_counts.get("image", 0)
136
num_audios = mm_counts.get("audio", 0)
137
# Use spellings the renderer would emit; tokenization is bypassed in
138
- # _call_hf_processor (we build input_ids directly), so the exact text
139
- # only needs to be a stable per-item marker.
138
+ # _apply_hf_processor_main (we build input_ids directly), so the exact
139
+ # text only needs to be a stable per-item marker.
140
return ("<|content_image|>" * num_images) + (
141
"<|content_audio_input|>" * num_audios
142
)
@@ -179,17 +179,19 @@ class InklingDummyInputsBuilder(BaseDummyInputsBuilder[InklingProcessingInfo]):
179
180
181
class InklingMultiModalProcessor(BaseMultiModalProcessor[InklingProcessingInfo]):
182
- def _call_hf_processor(
182
+ def _apply_hf_processor_main(
183
self,
184
- prompt: str,
185
- mm_data: Mapping[str, object],
186
- mm_kwargs: Mapping[str, object],
184
+ mm_items: MultiModalDataItems,
185
+ hf_processor_mm_kwargs: Mapping[str, object],
186
) -> BatchFeature:
188
- # Inkling is not a standard HF processor (no fused text+mm call), so we run
189
- # the vendored extractors ourselves and tokenize the text separately.
190
- # The MM placeholders in `prompt` are expanded later by the prompt
191
- # updates, so here we emit ONE placeholder id per media item.
192
- processor = self.info.get_hf_processor(**mm_kwargs)
187
+ valid_mm_items = mm_items.select(
188
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
189
+ )
190
+ mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
191
+
192
+ prompt_text = self.dummy_inputs.get_dummy_text(mm_items.get_all_counts())
193
+
194
+ processor = self.info.get_hf_processor(**hf_processor_mm_kwargs)
195
tokenizer = self.info.get_tokenizer()
196
197
images = mm_data.get("images") or []
@@ -200,7 +202,7 @@ class InklingMultiModalProcessor(BaseMultiModalProcessor[InklingProcessingInfo])
202
audios = list(cast(Iterable[Any], audios))
203
204
prompt_ids = self._tokenize_with_placeholders(
203
- prompt, tokenizer, len(images), len(audios)
205
+ prompt_text, tokenizer, len(images), len(audios)
206
)
207
208
data: dict[str, Any] = {"input_ids": [prompt_ids]}
@@ -232,7 +234,9 @@ class InklingMultiModalProcessor(BaseMultiModalProcessor[InklingProcessingInfo])
234
data["input_audio_features"] = input_audio_features
235
data["num_audio_tokens"] = torch.tensor(num_audio_tokens, dtype=torch.int64)
236
235
- return BatchFeature(data=data, tensor_type=None)
237
+ processed_data = BatchFeature(data=data, tensor_type=None)
238
+ processed_data.update(passthrough_data)
239
+ return processed_data
240
241
def _tokenize_with_placeholders(
242
self,
vllm/models/kimi_k3/common/mm_preprocess.py
-13
@@ -231,19 +231,6 @@ class KimiK3DummyInputsBuilder(BaseDummyInputsBuilder[KimiK3ProcessingInfo]):
231
class KimiK3MultiModalProcessor(BaseMultiModalProcessor[KimiK3ProcessingInfo]):
232
"""Image-only multi-modal processor for Kimi-K3."""
233
234
- def _call_hf_processor(
235
- self,
236
- prompt: str,
237
- mm_data: Mapping[str, object],
238
- mm_kwargs: Mapping[str, object],
239
- ) -> BatchFeature:
240
- # Override so the base always routes through the text+mm path
241
- # (`KimiK3Processor.__call__`). Otherwise the mm-only fast path calls
242
- # the checkpoint image processor directly with bare PIL images, but it
243
- # requires `{"type": "image", "image": PIL}` media dicts that only our
244
- # wrapper builds.
245
- return super()._call_hf_processor(prompt, mm_data, mm_kwargs)
246
-
234
def _get_mm_fields_config(
235
self,
236
hf_inputs: BatchFeature,
vllm/models/minimax_m3/common/mm_preprocess.py
+19
-8
@@ -306,12 +306,21 @@ class MiniMaxM3VLDummyInputsBuilder(BaseDummyInputsBuilder[MiniMaxM3VLProcessing
306
class MiniMaxM3VLMultiModalProcessor(
307
BaseMultiModalProcessor[MiniMaxM3VLProcessingInfo]
308
):
309
- def _call_hf_processor(
309
+ def _apply_hf_processor_main(
310
self,
311
- prompt: str,
312
- mm_data: Mapping[str, object],
313
- mm_kwargs: Mapping[str, object],
311
+ mm_items: MultiModalDataItems,
312
+ hf_processor_mm_kwargs: Mapping[str, object],
313
) -> BatchFeature:
314
+ valid_mm_items = mm_items.select(
315
+ {k for k, c in mm_items.get_all_counts().items() if c > 0}
316
+ )
317
+ mm_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
318
+
319
+ if not mm_data:
320
+ return BatchFeature(dict(passthrough_data))
321
+
322
+ prompt_text = self.dummy_inputs.get_dummy_text(mm_items.get_all_counts())
323
+
324
mm_data = dict(mm_data)
325
# With ``video_needs_metadata=True`` each video arrives as a
326
# ``(frames, metadata)`` tuple. Split the frames back out and forward the
@@ -340,15 +349,17 @@ class MiniMaxM3VLMultiModalProcessor(
349
350
# Override the video processor's default do_resize=False (set for a
351
# pre-resized pipeline) to True for vLLM's raw-frame inputs.
343
- merged = dict(do_resize=True, **mm_kwargs)
344
- data = dict(text=prompt, **mm_data)
352
+ merged = dict(do_resize=True, **hf_processor_mm_kwargs)
353
+ data = dict(text=prompt_text, **mm_data)
354
if video_metadata is not None:
355
data["video_metadata"] = video_metadata
347
- return self.info.ctx.call_hf_processor(
348
- self.info.get_hf_processor(**mm_kwargs),
356
+ processed_data = self.info.ctx.call_hf_processor(
357
+ self.info.get_hf_processor(**hf_processor_mm_kwargs),
358
data,
359
merged,
360
)
361
+ processed_data.update(passthrough_data)
362
+ return processed_data
363
364
def _get_mm_fields_config(
365
self,
vllm/multimodal/processing/context.py
+6
-3
@@ -262,9 +262,6 @@ class InputProcessingContext:
262
hf_processor: Callable[..., BatchFeature] | ProcessorMixin,
263
data: Mapping[str, object],
264
kwargs: Mapping[str, object] = {},
265
- *,
266
- num_tries: int = 1,
267
- max_tries: int = 5,
265
) -> BatchFeature:
266
"""
267
Call `hf_processor` on the prompt `data`
@@ -274,6 +271,12 @@ class InputProcessingContext:
271
272
merged_kwargs = self.get_merged_mm_kwargs(kwargs)
273
274
+ # vLLM needs the full untruncated sequence to keep multi-modal
275
+ # placeholder tokens aligned; note that the text inputs in
276
+ # call_hf_processor are just dummy text, not the original prompt.
277
+ # The original prompt is already tokenized by the renderer.
278
+ merged_kwargs.setdefault("truncation", False)
279
+
280
allowed_kwargs = get_allowed_kwarg_only_overrides(
281
hf_processor,
282
merged_kwargs,
vllm/multimodal/processing/dummy_inputs.py
+1
-4
@@ -91,10 +91,7 @@ class BaseDummyInputsBuilder(ABC, Generic[_I]):
91
# prompt tokens.
92
dummy_prompt = []
93
else:
94
- dummy_prompt = tokenizer.encode(
95
- dummy_text,
96
- **self.info.default_tok_params.get_encode_kwargs(),
97
- )
94
+ dummy_prompt = tokenizer.encode(dummy_text, truncation=False)
95
96
return ProcessorInputs(
97
prompt=dummy_prompt,
vllm/multimodal/processing/processor.py
+83
-121
@@ -5,9 +5,10 @@ from collections import defaultdict, deque
5
from collections.abc import Callable, Generator, ItemsView, Iterable, Mapping, Sequence
6
from dataclasses import dataclass, field, replace
7
from enum import Enum
8
-from functools import lru_cache, partial
8
+from functools import lru_cache
9
from typing import (
10
TYPE_CHECKING,
11
+ ClassVar,
12
Generic,
13
NamedTuple,
14
Protocol,
@@ -28,7 +29,6 @@ from vllm.inputs import (
29
)
30
from vllm.logger import init_logger
31
from vllm.tokenizers import TokenizerLike
31
-from vllm.transformers_utils.processor import call_hf_processor_mm_only
32
from vllm.utils.collection_utils import flatten_2d_lists, full_groupby
33
34
from ..inputs import (
@@ -1177,6 +1177,12 @@ class BaseMultiModalProcessor(ABC, Generic[_I]):
1177
Not to be confused with `transformers.ProcessorMixin`.
1178
"""
1179
1180
+ hf_processor_applies_updates: ClassVar[bool] = False
1181
+ """
1182
+ Only set to True if the tokenizer or chat template
1183
+ (which are run before MM processing) inserts placeholder tokens.
1184
+ """
1185
+
1186
def __init__(
1187
self,
1188
info: _I,
@@ -1302,131 +1308,100 @@ class BaseMultiModalProcessor(ABC, Generic[_I]):
1308
1309
return processor_data, passthrough_data
1310
1305
- def _call_hf_processor(
1306
- self,
1307
- prompt: str,
1308
- # Not to be confused with `mm_data` in `self.apply`.
1309
- # This refers to the data to be passed to HF processor.
1310
- mm_data: Mapping[str, object],
1311
- mm_kwargs: Mapping[str, object],
1312
- ) -> BatchFeature:
1311
+ def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str | None:
1312
"""
1314
- Call the HF processor on the prompt text and
1315
- associated multi-modal data.
1313
+ Get the text to pass to the HF processor alongside the multi-modal
1314
+ data.
1315
+
1316
+ By default, no text is passed. If the HF processor requires that
1317
+ text and multi-modal items correspond to each other, you should
1318
+ override this method to return dummy text generated by
1319
+ [`DummyInputsBuilder`][vllm.multimodal.processing.BaseDummyInputsBuilder].
1320
"""
1317
- return self.info.ctx.call_hf_processor(
1318
- self.info.get_hf_processor(**mm_kwargs),
1319
- dict(text=prompt, **mm_data),
1320
- mm_kwargs,
1321
- )
1321
+ return None
1322
1323
- def _apply_hf_processor_text_mm(
1323
+ def _preprocess_hf_mm_data(
1324
self,
1325
- prompt_text: str,
1326
- mm_items: MultiModalDataItems,
1325
+ mm_data: Mapping[str, object],
1326
hf_processor_mm_kwargs: Mapping[str, object],
1328
- ) -> tuple[list[int], BatchFeature]:
1327
+ ) -> tuple[Mapping[str, object], Mapping[str, object]]:
1328
"""
1330
- Apply the HF processor on the prompt text and multi-modal data
1331
- together.
1329
+ Pre-process the multi-modal data and HF processor keyword arguments
1330
+ before they are passed to the HF processor.
1331
+
1332
+ By default, both are returned as-is. If the HF processor expects the
1333
+ multi-modal data under different keys than those provided by the
1334
+ multi-modal items (e.g. `audio` instead of `audios`), or requires
1335
+ additional keyword arguments (e.g. `sampling_rate`), you should
1336
+ override this method.
1337
"""
1333
- valid_mm_items = mm_items.select(
1334
- {k for k, c in mm_items.get_all_counts().items() if c > 0}
1335
- )
1336
- processor_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
1337
-
1338
- processed_data = self._call_hf_processor(
1339
- prompt=prompt_text,
1340
- mm_data=processor_data,
1341
- mm_kwargs=hf_processor_mm_kwargs,
1342
- )
1343
- processed_data.update(passthrough_data)
1344
-
1345
- input_ids = processed_data.pop("input_ids")
1346
- if not isinstance(input_ids, list):
1347
- input_ids = input_ids.tolist()
1348
-
1349
- (prompt_ids,) = input_ids
1338
+ return mm_data, hf_processor_mm_kwargs
1339
1351
- return prompt_ids, processed_data
1352
-
1353
- def _apply_hf_processor_tokens_only(
1340
+ def _postprocess_hf_mm_data(
1341
self,
1355
- prompt_tokens: list[int],
1356
- ) -> list[int]:
1342
+ mm_data: Mapping[str, object],
1343
+ hf_processor_mm_kwargs: Mapping[str, object],
1344
+ processed_data: BatchFeature,
1345
+ ) -> BatchFeature:
1346
"""
1358
- Apply the HF processor on the prompt tokens only.
1347
+ Post-process the output of the HF processor.
1348
1360
- Most HF processors accept prompt text but not prompt tokens.
1361
- If the HF processor adds or removes tokens that are not related to
1362
- multi-modal data, you should override this method so it is consistent
1363
- with the tokenization of the corresponding text.
1349
+ By default, the output is returned as-is. If you need to modify the
1350
+ output of the HF processor before it is converted into multi-modal
1351
+ keyword arguments, you should override this method.
1352
"""
1365
- return prompt_tokens
1353
+ return processed_data
1354
1367
- def _apply_hf_processor_mm_only(
1355
+ def _apply_hf_processor_main(
1356
self,
1357
mm_items: MultiModalDataItems,
1358
hf_processor_mm_kwargs: Mapping[str, object],
1359
) -> BatchFeature:
1360
"""
1373
- Apply the HF processor on the multi-modal data only.
1374
-
1375
- Since HF processor requires that text and multi-modal items
1376
- correspond to each other, we generate dummy text using
1377
- [`DummyInputsBuilder`][vllm.multimodal.processing.BaseDummyInputsBuilder]
1378
- to go along with the multi-modal data.
1361
+ Apply the HF processor on the multi-modal data.
1362
"""
1380
- # Custom logic based on text inputs
1381
- if type(self)._call_hf_processor != BaseMultiModalProcessor._call_hf_processor:
1382
- mm_counts = mm_items.get_all_counts()
1383
-
1384
- _, mm_processed_data = self._apply_hf_processor_text_mm(
1385
- prompt_text=self.dummy_inputs.get_dummy_text(mm_counts),
1386
- mm_items=mm_items,
1387
- hf_processor_mm_kwargs=hf_processor_mm_kwargs,
1388
- )
1389
-
1390
- return mm_processed_data
1391
-
1363
valid_mm_items = mm_items.select(
1364
{k for k, c in mm_items.get_all_counts().items() if c > 0}
1365
)
1366
processor_data, passthrough_data = self._get_hf_mm_data(valid_mm_items)
1367
1397
- processed_data = self.info.ctx.call_hf_processor(
1398
- partial(
1399
- call_hf_processor_mm_only,
1368
+ if processor_data:
1369
+ processor_data, hf_processor_mm_kwargs = self._preprocess_hf_mm_data(
1370
+ processor_data, hf_processor_mm_kwargs
1371
+ )
1372
+
1373
+ prompt_text = self._get_hf_processor_text(mm_items.get_all_counts())
1374
+ if prompt_text is not None:
1375
+ processor_data = dict(text=prompt_text, **processor_data)
1376
+
1377
+ processed_data = self.info.ctx.call_hf_processor(
1378
self.info.get_hf_processor(**hf_processor_mm_kwargs),
1401
- ),
1379
+ processor_data,
1380
+ hf_processor_mm_kwargs,
1381
+ )
1382
+ processed_data.update(passthrough_data)
1383
+ else:
1384
+ from transformers.feature_extraction_utils import BatchFeature
1385
+
1386
+ processed_data = BatchFeature(dict(passthrough_data))
1387
+
1388
+ return self._postprocess_hf_mm_data(
1389
processor_data,
1390
hf_processor_mm_kwargs,
1391
+ processed_data,
1392
)
1405
- processed_data.update(passthrough_data)
1406
-
1407
- return processed_data
1393
1409
- def _apply_hf_processor_main(
1410
- self,
1411
- prompt: list[int],
1412
- mm_items: MultiModalDataItems,
1413
- hf_processor_mm_kwargs: Mapping[str, object],
1414
- ) -> tuple[list[int], BatchFeature, bool]:
1394
+ def _postprocess_prompt(self, prompt: list[int]) -> list[int]:
1395
"""
1416
- Apply the HF processor on the prompt tokens and multi-modal data.
1396
+ Post-process the prompt token IDs before locating or applying
1397
+ multi-modal prompt updates.
1398
1418
- In addition, return whether prompt updates have been applied
1419
- (`False` here, since for token prompts the updates are normally
1420
- applied separately via `_maybe_apply_prompt_updates`).
1399
+ By default, the prompt is returned as-is. If the HF processor (or
1400
+ chat template) applies additional transformations to the prompt
1401
+ that are not reflected in its multi-modal outputs, you should
1402
+ override this method to replicate them.
1403
"""
1422
- prompt_ids = self._apply_hf_processor_tokens_only(prompt)
1423
-
1424
- mm_processed_data = self._apply_hf_processor_mm_only(
1425
- mm_items=mm_items,
1426
- hf_processor_mm_kwargs=hf_processor_mm_kwargs,
1427
- )
1428
-
1429
- return prompt_ids, mm_processed_data, False
1404
+ return prompt
1405
1406
def _get_cache_missing_items(
1407
self,
@@ -1531,14 +1506,9 @@ class BaseMultiModalProcessor(ABC, Generic[_I]):
1506
self,
1507
inputs: ProcessorInputs,
1508
timing_ctx: TimingContext,
1534
- ) -> tuple[list[int], MultiModalProcessingInfo, bool]:
1509
+ ) -> MultiModalProcessingInfo:
1510
with timing_ctx.record("apply_hf_processor"):
1536
- (
1537
- prompt_ids,
1538
- mm_processed_data,
1539
- is_update_applied,
1540
- ) = self._apply_hf_processor_main(
1541
- prompt=inputs.prompt,
1511
+ mm_processed_data = self._apply_hf_processor_main(
1512
mm_items=inputs.mm_data_items,
1513
hf_processor_mm_kwargs=inputs.hf_processor_mm_kwargs,
1514
)
@@ -1569,13 +1539,13 @@ class BaseMultiModalProcessor(ABC, Generic[_I]):
1539
prompt_updates=mm_prompt_updates,
1540
)
1541
1572
- return prompt_ids, mm_info, is_update_applied
1542
+ return mm_info
1543
1544
def _cached_apply_hf_processor(
1545
self,
1546
inputs: ProcessorInputs,
1547
timing_ctx: TimingContext,
1578
- ) -> tuple[list[int], MultiModalProcessingInfo, bool]:
1548
+ ) -> MultiModalProcessingInfo:
1549
"""
1550
Apply the HF processor on the full prompt text,
1551
caching the results and reusing cached results.
@@ -1599,16 +1569,11 @@ class BaseMultiModalProcessor(ABC, Generic[_I]):
1569
mm_hashes=mm_hashes,
1570
)
1571
1602
- # NOTE: `prompt` does not correspond to `mm_missing_data_items`,
1572
+ # NOTE: The prompt does not correspond to `mm_missing_data_items`,
1573
# so we can't apply prompt updates until the new multimodal
1574
# items are combined with the cached multimodal items
1575
with timing_ctx.record("apply_hf_processor"):
1606
- (
1607
- prompt_ids,
1608
- mm_missing_processed_data,
1609
- is_update_applied,
1610
- ) = self._apply_hf_processor_main(
1611
- prompt=inputs.prompt,
1576
+ mm_missing_processed_data = self._apply_hf_processor_main(
1577
mm_items=mm_missing_data_items,
1578
hf_processor_mm_kwargs=inputs.hf_processor_mm_kwargs,
1579
)
@@ -1641,7 +1606,7 @@ class BaseMultiModalProcessor(ABC, Generic[_I]):
1606
prompt_updates=mm_prompt_updates,
1607
)
1608
1644
- return prompt_ids, mm_info, is_update_applied
1609
+ return mm_info
1610
1611
def _apply_token_matches(
1612
self,
@@ -1770,7 +1735,7 @@ class BaseMultiModalProcessor(ABC, Generic[_I]):
1735
"There is likely a problem with your "
1736
"implementation of merged multi-modal processor for this "
1737
"model (usually arising from an inconsistency between "
1773
- "`_call_hf_processor` and `_get_mm_fields_config`)."
1738
+ "`_apply_hf_processor_main` and `_get_mm_fields_config`)."
1739
)
1740
1741
def _validate_mm_updates(
@@ -1805,8 +1770,8 @@ class BaseMultiModalProcessor(ABC, Generic[_I]):
1770
f"Expected there to be {item_count} prompt placeholders "
1771
f"corresponding to {item_count} {modality} items, but "
1772
f"instead found {len(placeholders)} prompt placeholders! "
1808
- "Make sure the implementation of `_call_hf_processor` and "
1809
- "`_get_mm_fields_config` are consistent with each other."
1773
+ "Make sure the implementation of `_apply_hf_processor_main` "
1774
+ "and `_get_mm_fields_config` are consistent with each other."
1775
)
1776
1777
def _maybe_apply_prompt_updates(
@@ -1854,11 +1819,8 @@ class BaseMultiModalProcessor(ABC, Generic[_I]):
1819
3. Extract information about the placeholder tokens from the
1820
processed token IDs.
1821
"""
1857
- (
1858
- prompt_ids,
1859
- mm_info,
1860
- is_update_applied,
1861
- ) = self._cached_apply_hf_processor(inputs, timing_ctx)
1822
+ prompt_ids = self._postprocess_prompt(inputs.prompt)
1823
+ mm_info = self._cached_apply_hf_processor(inputs, timing_ctx)
1824
1825
with timing_ctx.record("apply_prompt_updates"):
1826
prompt_ids, mm_placeholders = self._maybe_apply_prompt_updates(
@@ -1866,7 +1828,7 @@ class BaseMultiModalProcessor(ABC, Generic[_I]):
1828
prompt_ids=prompt_ids,
1829
mm_kwargs=mm_info.kwargs,
1830
mm_prompt_updates=mm_info.prompt_updates,
1869
- is_update_applied=is_update_applied,
1831
+ is_update_applied=self.hf_processor_applies_updates,
1832
)
1833
1834
mm_placeholder_ranges = {
vllm/transformers_utils/processor.py
-44
@@ -11,17 +11,13 @@ from transformers import (
11
AutoImageProcessor,
12
AutoProcessor,
13
AutoVideoProcessor,
14
- BatchFeature,
14
processing_utils,
15
)
17
-from transformers.audio_utils import AudioInput
16
from transformers.feature_extraction_utils import FeatureExtractionMixin
17
from transformers.image_processing_utils import BaseImageProcessor
20
-from transformers.image_utils import ImageInput
18
from transformers.models.auto.video_processing_auto import VIDEO_PROCESSOR_MAPPING_NAMES
19
from transformers.processing_utils import ProcessorMixin
20
from transformers.video_processing_utils import BaseVideoProcessor
24
-from transformers.video_utils import VideoInput
21
from typing_extensions import TypeVar
22
23
from vllm.logger import init_logger
@@ -554,43 +550,3 @@ def cached_video_processor_from_config(
550
processor_cls_overrides=processor_cls, # type: ignore[arg-type]
551
**_merge_mm_kwargs(model_config, AutoVideoProcessor, **kwargs),
552
)
557
-
558
-
559
-def call_hf_processor_mm_only(
560
- processor: ProcessorMixin,
561
- images: ImageInput | None = None,
562
- videos: VideoInput | None = None,
563
- audio: AudioInput | None = None,
564
- **kwargs,
565
-) -> BatchFeature:
566
- output_kwargs = processor._merge_kwargs(
567
- get_processor_kwargs_type(processor),
568
- **kwargs,
569
- )
570
-
571
- if audio is not None and (
572
- feature_extractor := getattr(processor, "feature_extractor", None)
573
- ):
574
- audio_inputs = feature_extractor(audio, **output_kwargs["audio_kwargs"])
575
- audio_inputs["feature_attention_mask"] = audio_inputs.pop("attention_mask")
576
- else:
577
- audio_inputs = {}
578
-
579
- if images is not None and (
580
- image_processor := getattr(processor, "image_processor", None)
581
- ):
582
- images_inputs = image_processor(images=images, **output_kwargs["images_kwargs"])
583
- else:
584
- images_inputs = {}
585
-
586
- if videos is not None and (
587
- video_processor := getattr(processor, "video_processor", None)
588
- ):
589
- videos_inputs = video_processor(videos=videos, **output_kwargs["videos_kwargs"])
590
- else:
591
- videos_inputs = {}
592
-
593
- return BatchFeature(
594
- data={**audio_inputs, **images_inputs, **videos_inputs},
595
- tensor_type=kwargs.get("return_tensors"),
596
- )