A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 05:30 UTC
[Model] Always use Transformers backend for PaliGemma and Gemma3-MM (#26715)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Cyrus Leung committed
Oct 17, 2025 at 13:03 UTC
8c017b34908f8d4a877d862dd21b99aef7057c55
12 files changed
+54
-1219
docs/models/hardware_supported_models/tpu.md
+2
-2
@@ -16,8 +16,8 @@
16
| meta-llama/Llama-4-* | Llama4ForConditionalGeneration | ❌ |
17
| microsoft/Phi-3-mini-128k-instruct | Phi3ForCausalLM | 🟨 |
18
| microsoft/phi-4 | Phi3ForCausalLM | ❌ |
19
-| google/gemma-3-27b-it | Gemma3ForConditionalGeneration | 🟨 |
20
-| google/gemma-3-4b-it | Gemma3ForConditionalGeneration | ❌ |
19
+| google/gemma-3-27b-it | TransformersForMultimodalLM | 🟨 |
20
+| google/gemma-3-4b-it | TransformersForMultimodalLM | ❌ |
21
| deepseek-ai/DeepSeek-R1 | DeepseekV3ForCausalLM | ❌ |
22
| deepseek-ai/DeepSeek-V3 | DeepseekV3ForCausalLM | ❌ |
23
| RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8 | LlamaForCausalLM | ✅ |
docs/models/supported_models.md
+3
-20
@@ -650,7 +650,6 @@ These models primarily accept the [`LLM.generate`](./generative_models.md#llmgen
650
| `DeepseekVLV2ForCausalLM`<sup>^</sup> | DeepSeek-VL2 | T + I<sup>+</sup> | `deepseek-ai/deepseek-vl2-tiny`, `deepseek-ai/deepseek-vl2-small`, `deepseek-ai/deepseek-vl2`, etc. | | ✅︎ |
651
| `Ernie4_5_VLMoeForConditionalGeneration` | Ernie4.5-VL | T + I<sup>+</sup>/ V<sup>+</sup> | `baidu/ERNIE-4.5-VL-28B-A3B-PT`, `baidu/ERNIE-4.5-VL-424B-A47B-PT` | | ✅︎ |
652
| `FuyuForCausalLM` | Fuyu | T + I | `adept/fuyu-8b`, etc. | | ✅︎ |
653
-| `Gemma3ForConditionalGeneration` | Gemma 3 | T + I<sup>+</sup> | `google/gemma-3-4b-it`, `google/gemma-3-27b-it`, etc. | ✅︎ | ✅︎ |
653
| `Gemma3nForConditionalGeneration` | Gemma 3n | T + I + A | `google/gemma-3n-E2B-it`, `google/gemma-3n-E4B-it`, etc. | | |
654
| `GLM4VForCausalLM`<sup>^</sup> | GLM-4V | T + I | `zai-org/glm-4v-9b`, `zai-org/cogagent-9b-20241220`, etc. | ✅︎ | ✅︎ |
655
| `Glm4vForConditionalGeneration` | GLM-4.1V-Thinking | T + I<sup>E+</sup> + V<sup>E+</sup> | `zai-org/GLM-4.1V-9B-Thinking`, etc. | ✅︎ | ✅︎ |
@@ -679,7 +678,6 @@ These models primarily accept the [`LLM.generate`](./generative_models.md#llmgen
678
| `NVLM_D_Model` | NVLM-D 1.0 | T + I<sup>+</sup> | `nvidia/NVLM-D-72B`, etc. | | ✅︎ |
679
| `Ovis` | Ovis2, Ovis1.6 | T + I<sup>+</sup> | `AIDC-AI/Ovis2-1B`, `AIDC-AI/Ovis1.6-Llama3.2-3B`, etc. | | ✅︎ |
680
| `Ovis2_5` | Ovis2.5 | T + I<sup>+</sup> + V | `AIDC-AI/Ovis2.5-9B`, etc. | | |
682
-| `PaliGemmaForConditionalGeneration` | PaliGemma, PaliGemma 2 | T + I<sup>E</sup> | `google/paligemma-3b-pt-224`, `google/paligemma-3b-mix-224`, `google/paligemma2-3b-ft-docci-448`, etc. | | ✅︎ |
681
| `Phi3VForCausalLM` | Phi-3-Vision, Phi-3.5-Vision | T + I<sup>E+</sup> | `microsoft/Phi-3-vision-128k-instruct`, `microsoft/Phi-3.5-vision-instruct`, etc. | | ✅︎ |
682
| `Phi4MMForCausalLM` | Phi-4-multimodal | T + I<sup>+</sup> / T + A<sup>+</sup> / I<sup>+</sup> + A<sup>+</sup> | `microsoft/Phi-4-multimodal-instruct`, etc. | ✅︎ | ✅︎ |
683
| `Phi4MultimodalForCausalLM` | Phi-4-multimodal (HF Transformers) | T + I<sup>+</sup> / T + A<sup>+</sup> / I<sup>+</sup> + A<sup>+</sup> | `microsoft/Phi-4-multimodal-instruct` (with revision `refs/pr/70`), etc. | ✅︎ | ✅︎ |
@@ -704,6 +702,8 @@ Some models are supported only via the [Transformers backend](#transformers). Th
702
| Architecture | Models | Inputs | Example HF Models | [LoRA](../features/lora.md) | [PP](../serving/parallelism_scaling.md) |
703
|--------------|--------|--------|-------------------|-----------------------------|-----------------------------------------|
704
| `Emu3ForConditionalGeneration` | Emu3 | T + I | `BAAI/Emu3-Chat-hf` | ✅︎ | ✅︎ |
705
+| `Gemma3ForConditionalGeneration` | Gemma 3 | T + I<sup>+</sup> | `google/gemma-3-4b-it`, `google/gemma-3-27b-it`, etc. | ✅︎ | ✅︎ |
706
+| `PaliGemmaForConditionalGeneration` | PaliGemma, PaliGemma 2 | T + I<sup>E</sup> | `google/paligemma-3b-pt-224`, `google/paligemma-3b-mix-224`, `google/paligemma2-3b-ft-docci-448`, etc. | ✅︎ | ✅︎ |
707
708
<sup>^</sup> You need to set the architecture name via `--hf-overrides` to match the one in vLLM.
709
• For example, to use DeepSeek-VL2 series models:
@@ -712,21 +712,7 @@ Some models are supported only via the [Transformers backend](#transformers). Th
712
<sup>+</sup> Multiple items can be inputted per text prompt for this modality.
713
714
!!! warning
715
- Both V0 and V1 support `Gemma3ForConditionalGeneration` for text-only inputs.
716
- However, there are differences in how they handle text + image inputs:
717
-
718
- V0 correctly implements the model's attention pattern:
719
- - Uses bidirectional attention between the image tokens corresponding to the same image
720
- - Uses causal attention for other tokens
721
- - Implemented via (naive) PyTorch SDPA with masking tensors
722
- - Note: May use significant memory for long prompts with image
723
-
724
- V1 currently uses a simplified attention pattern:
725
- - Uses causal attention for all tokens, including image tokens
726
- - Generates reasonable outputs but does not match the original model's attention for text + image inputs, especially when `{"do_pan_and_scan": true}`
727
- - Will be updated in the future to support the correct behavior
728
-
729
- This limitation exists because the model's mixed attention pattern (bidirectional for images, causal otherwise) is not yet supported by vLLM's attention backends.
715
+ For `Gemma3ForConditionalGeneration`, `{"do_pan_and_scan": true}` is not supported in Transformers backend yet.
716
717
!!! note
718
`Gemma3nForConditionalGeneration` is only supported on V1 due to shared KV caching and it depends on `timm>=1.0.17` to make use of its
@@ -778,9 +764,6 @@ Some models are supported only via the [Transformers backend](#transformers). Th
764
The official `openbmb/MiniCPM-V-2` doesn't work yet, so we need to use a fork (`HwwwH/MiniCPM-V-2`) for now.
765
For more details, please see: <https://github.com/vllm-project/vllm/pull/4087#issuecomment-2250397630>
766
781
-!!! warning
782
- Our PaliGemma implementations have the same problem as Gemma 3 (see above) for both V0 and V1.
783
-
767
!!! note
768
For Qwen2.5-Omni and Qwen3-Omni, reading audio from video pre-processing (`--mm-processor-kwargs '{"use_audio_in_video": true}'`) is currently work in progress and not yet supported.
769
examples/offline_inference/vision_language.py
+2
-1
@@ -248,7 +248,8 @@ def run_gemma3(questions: list[str], modality: str) -> ModelRequestData:
248
model=model_name,
249
max_model_len=2048,
250
max_num_seqs=2,
251
- mm_processor_kwargs={"do_pan_and_scan": True},
251
+ # TODO: Support this in transformers backend
252
+ # mm_processor_kwargs={"do_pan_and_scan": True},
253
limit_mm_per_prompt={modality: 1},
254
)
255
tests/models/language/generation/test_gemma.py
+5
-11
@@ -3,7 +3,7 @@
3
import numpy as np
4
import pytest
5
6
-MODELS = ["google/gemma-2b", "google/gemma-2-2b", "google/gemma-3-4b-it"]
6
+MODELS = ["google/gemma-2b", "google/gemma-2-2b"]
7
8
9
@pytest.mark.parametrize("model", MODELS)
@@ -14,14 +14,8 @@ def test_dummy_loader(vllm_runner, monkeypatch, model: str) -> None:
14
model,
15
load_format="dummy",
16
) as llm:
17
- if model == "google/gemma-3-4b-it":
18
- normalizers = llm.llm.collective_rpc(
19
- lambda self: self.model_runner.model.language_model.model.normalizer.cpu().item() # noqa: E501
20
- )
21
- config = llm.llm.llm_engine.model_config.hf_config.text_config
22
- else:
23
- normalizers = llm.llm.collective_rpc(
24
- lambda self: self.model_runner.model.model.normalizer.cpu().item()
25
- )
26
- config = llm.llm.llm_engine.model_config.hf_config
17
+ normalizers = llm.apply_model(
18
+ lambda model: model.model.normalizer.cpu().item()
19
+ )
20
+ config = llm.llm.llm_engine.model_config.hf_config
21
assert np.allclose(normalizers, config.hidden_size**0.5, rtol=2e-3)
tests/models/multimodal/generation/test_common.py
+34
-40
@@ -113,25 +113,6 @@ VLM_TEST_SETTINGS = {
113
dtype="bfloat16" if current_platform.is_cpu() else "auto",
114
marks=[pytest.mark.core_model, pytest.mark.cpu_model],
115
),
116
- "paligemma": VLMTestInfo(
117
- models=["google/paligemma-3b-mix-224"],
118
- test_type=VLMTestType.IMAGE,
119
- prompt_formatter=identity,
120
- img_idx_to_prompt=lambda idx: "",
121
- # Paligemma uses its own sample prompts because the default one fails
122
- single_image_prompts=IMAGE_ASSETS.prompts(
123
- {
124
- "stop_sign": "caption es",
125
- "cherry_blossom": "What is in the picture?",
126
- }
127
- ),
128
- auto_cls=AutoModelForImageTextToText,
129
- vllm_output_post_proc=model_utils.paligemma_vllm_to_hf_output,
130
- dtype="bfloat16",
131
- marks=[
132
- pytest.mark.skip(reason="vLLM does not support PrefixLM attention mask")
133
- ],
134
- ),
116
"qwen2_5_vl": VLMTestInfo(
117
models=["Qwen/Qwen2.5-VL-3B-Instruct"],
118
test_type=(VLMTestType.IMAGE, VLMTestType.MULTI_IMAGE, VLMTestType.VIDEO),
@@ -196,14 +177,24 @@ VLM_TEST_SETTINGS = {
177
# Gemma3 has bidirectional mask on images
178
"gemma3-transformers": VLMTestInfo(
179
models=["google/gemma-3-4b-it"],
199
- test_type=VLMTestType.IMAGE,
200
- prompt_formatter=lambda vid_prompt: f"<'<bos><start_of_turn>user\n{vid_prompt}<start_of_image><end_of_turn>\n<start_of_turn>model\n", # noqa: E501
201
- max_model_len=4096,
180
+ test_type=(VLMTestType.IMAGE, VLMTestType.MULTI_IMAGE),
181
+ prompt_formatter=lambda img_prompt: f"<bos><start_of_turn>user\n{img_prompt}<end_of_turn>\n<start_of_turn>model\n", # noqa: E501
182
+ single_image_prompts=IMAGE_ASSETS.prompts(
183
+ {
184
+ "stop_sign": "<start_of_image>What's the content in the center of the image?", # noqa: E501
185
+ "cherry_blossom": "<start_of_image>What is the season?",
186
+ }
187
+ ),
188
+ multi_image_prompt="<start_of_image><start_of_image>Describe the two images in detail.", # noqa: E501
189
+ max_model_len=8192,
190
auto_cls=AutoModelForImageTextToText,
191
+ # TODO: Support `do_pan_and_scan` in transformers backend
192
+ # patch_hf_runner=model_utils.gemma3_patch_hf_runner,
193
vllm_output_post_proc=model_utils.gemma3_vllm_to_hf_output,
194
image_size_factors=[(0.25, 0.5, 1.0)],
195
vllm_runner_kwargs={
196
"model_impl": "transformers",
197
+ # "mm_processor_kwargs": {"do_pan_and_scan": True},
198
},
199
marks=[pytest.mark.core_model],
200
),
@@ -222,6 +213,27 @@ VLM_TEST_SETTINGS = {
213
},
214
marks=[pytest.mark.core_model],
215
),
216
+ # PaliGemma has PrefixLM attention
217
+ "paligemma-transformers": VLMTestInfo(
218
+ models=["google/paligemma-3b-mix-224"],
219
+ test_type=VLMTestType.IMAGE,
220
+ prompt_formatter=identity,
221
+ img_idx_to_prompt=lambda idx: "",
222
+ # PaliGemma uses its own sample prompts because the default one fails
223
+ single_image_prompts=IMAGE_ASSETS.prompts(
224
+ {
225
+ "stop_sign": "caption es",
226
+ "cherry_blossom": "What is in the picture?",
227
+ }
228
+ ),
229
+ auto_cls=AutoModelForImageTextToText,
230
+ vllm_output_post_proc=model_utils.paligemma_vllm_to_hf_output,
231
+ image_size_factors=[(0.25, 0.5, 1.0)],
232
+ vllm_runner_kwargs={
233
+ "model_impl": "transformers",
234
+ },
235
+ marks=[pytest.mark.core_model],
236
+ ),
237
# Pixel values from processor are not 4D or 5D arrays
238
"qwen2_5_vl-transformers": VLMTestInfo(
239
models=["Qwen/Qwen2.5-VL-3B-Instruct"],
@@ -348,24 +360,6 @@ VLM_TEST_SETTINGS = {
360
image_size_factors=[(), (0.25,), (0.25, 0.25, 0.25), (0.25, 0.2, 0.15)],
361
marks=[large_gpu_mark(min_gb=32)],
362
),
351
- "gemma3": VLMTestInfo(
352
- models=["google/gemma-3-4b-it"],
353
- test_type=(VLMTestType.IMAGE, VLMTestType.MULTI_IMAGE),
354
- prompt_formatter=lambda img_prompt: f"<bos><start_of_turn>user\n{img_prompt}<end_of_turn>\n<start_of_turn>model\n", # noqa: E501
355
- single_image_prompts=IMAGE_ASSETS.prompts(
356
- {
357
- "stop_sign": "<start_of_image>What's the content in the center of the image?", # noqa: E501
358
- "cherry_blossom": "<start_of_image>What is the season?",
359
- }
360
- ),
361
- multi_image_prompt="<start_of_image><start_of_image>Describe the two images in detail.", # noqa: E501
362
- max_model_len=4096,
363
- max_num_seqs=2,
364
- auto_cls=AutoModelForImageTextToText,
365
- vllm_runner_kwargs={"mm_processor_kwargs": {"do_pan_and_scan": True}},
366
- patch_hf_runner=model_utils.gemma3_patch_hf_runner,
367
- num_logprobs=10,
368
- ),
363
"glm4v": VLMTestInfo(
364
models=["zai-org/glm-4v-9b"],
365
test_type=VLMTestType.IMAGE,
tests/models/multimodal/generation/vlm_utils/model_utils.py
-10
@@ -328,16 +328,6 @@ def gemma3_patch_hf_runner(hf_model: HfRunner) -> HfRunner:
328
329
hf_model.processor = processor
330
331
- orig_generate = hf_model.model.generate
332
-
333
- def _generate(self, *args, **kwargs):
334
- # FIXME: https://github.com/huggingface/transformers/issues/38333
335
- kwargs["disable_compile"] = True
336
-
337
- return orig_generate(*args, **kwargs)
338
-
339
- hf_model.model.generate = types.MethodType(_generate, hf_model.model)
340
-
331
return hf_model
332
333
tests/models/multimodal/processing/test_common.py
-4
@@ -222,7 +222,6 @@ def _test_processing_correctness(
222
_ADD_SPECIAL_TOKENS_OVERRIDES = {
223
"ovis": False,
224
"ovis2_5": False,
225
- "paligemma": False,
225
"ultravox": False,
226
"whisper": False,
227
}
@@ -333,7 +332,6 @@ def _test_processing_correctness_one(
332
"deepseek-ai/deepseek-vl2-tiny",
333
"baidu/ERNIE-4.5-VL-28B-A3B-PT",
334
"adept/fuyu-8b",
336
- "google/gemma-3-4b-it",
335
"google/gemma-3n-E2B-it",
336
"zai-org/glm-4v-9b",
337
"zai-org/GLM-4.1V-9B-Thinking",
@@ -370,8 +368,6 @@ def _test_processing_correctness_one(
368
"AIDC-AI/Ovis1.6-Llama3.2-3B",
369
"AIDC-AI/Ovis2-1B",
370
"AIDC-AI/Ovis2.5-2B",
373
- "google/paligemma-3b-mix-224",
374
- "google/paligemma2-3b-ft-docci-448",
371
"microsoft/Phi-3.5-vision-instruct",
372
"microsoft/Phi-4-multimodal-instruct",
373
"mistralai/Pixtral-12B-2409",
tests/models/multimodal/processing/test_tensor_schema.py
-1
@@ -48,7 +48,6 @@ ARCH_NEEDS_EXTRAS = [
48
"Idefics3ForConditionalGeneration",
49
"LlavaForConditionalGeneration",
50
"MiniCPMV",
51
- "PaliGemmaForConditionalGeneration",
51
]
52
REPO_ID_TO_SKIP = {
53
"nm-testing/pixtral-12b-FP8-dynamic": "duplicated test",
vllm/model_executor/models/gemma3_mm.py
deleted
-710
@@ -1,710 +0,0 @@
1
-# SPDX-License-Identifier: Apache-2.0
2
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-import math
4
-from collections.abc import Iterable, Mapping, Sequence
5
-from typing import Annotated, Any, Literal
6
-
7
-import torch
8
-from torch import nn
9
-from transformers import BatchFeature, Gemma3Config, Gemma3Processor
10
-from transformers.models.gemma3.processing_gemma3 import Gemma3ProcessorKwargs
11
-
12
-import vllm.envs as envs
13
-from vllm.config import VllmConfig
14
-from vllm.config.multimodal import BaseDummyOptions
15
-from vllm.logger import init_logger
16
-from vllm.model_executor.layers.layernorm import GemmaRMSNorm
17
-from vllm.model_executor.models.module_mapping import MultiModelKeys
18
-from vllm.multimodal import MULTIMODAL_REGISTRY
19
-from vllm.multimodal.inputs import (
20
- MultiModalDataDict,
21
- MultiModalFieldConfig,
22
- MultiModalKwargsItems,
23
-)
24
-from vllm.multimodal.parse import ImageProcessorItems, ImageSize, MultiModalDataItems
25
-from vllm.multimodal.processing import (
26
- BaseMultiModalProcessor,
27
- BaseProcessingInfo,
28
- MultiModalPromptUpdates,
29
- MultiModalPromptUpdatesApplyResult,
30
- PlaceholderFeaturesInfo,
31
- PromptReplacement,
32
- PromptUpdate,
33
- PromptUpdateDetails,
34
- replace_token_matches,
35
-)
36
-from vllm.multimodal.profiling import BaseDummyInputsBuilder
37
-from vllm.sequence import IntermediateTensors
38
-from vllm.utils.tensor_schema import TensorSchema, TensorShape
39
-
40
-from .interfaces import (
41
- MultiModalEmbeddings,
42
- SupportsLoRA,
43
- SupportsMultiModal,
44
- SupportsPP,
45
-)
46
-from .siglip import SiglipVisionModel
47
-from .utils import (
48
- AutoWeightsLoader,
49
- WeightsMapper,
50
- init_vllm_registered_model,
51
- maybe_prefix,
52
-)
53
-
54
-logger = init_logger(__name__)
55
-
56
-
57
-class Gemma3ImagePixelInputs(TensorSchema):
58
- """
59
- Dimensions:
60
- - p: Number of patches total (over each image over each prompt in the
61
- batch)
62
- - c: Number of channels (3)
63
- - h: Height of each patch
64
- - w: Width of each patch
65
- - bn: Batch size * number of images
66
- """
67
-
68
- type: Literal["pixel_values"] = "pixel_values"
69
-
70
- pixel_values: Annotated[torch.Tensor, TensorShape("p", 3, "h", "w")]
71
-
72
- num_patches: Annotated[torch.Tensor, TensorShape("bn")]
73
-
74
-
75
-Gemma3ImageInputs = Gemma3ImagePixelInputs
76
-
77
-
78
-class Gemma3ProcessingInfo(BaseProcessingInfo):
79
- def get_hf_config(self):
80
- return self.ctx.get_hf_config(Gemma3Config)
81
-
82
- def get_hf_processor(self, **kwargs: object):
83
- return self.ctx.get_hf_processor(Gemma3Processor, **kwargs)
84
-
85
- def get_supported_mm_limits(self) -> Mapping[str, int | None]:
86
- return {"image": None}
87
-
88
- def _resolve_image_kwargs(
89
- self,
90
- processor: Gemma3Processor,
91
- keys: set[str],
92
- ) -> dict[str, Any]:
93
- image_processor = processor.image_processor
94
- kwargs = processor._merge_kwargs(
95
- Gemma3ProcessorKwargs,
96
- tokenizer_init_kwargs=processor.tokenizer.init_kwargs,
97
- )
98
-
99
- images_kwargs = kwargs["images_kwargs"]
100
-
101
- def _resolve_kw(key: str):
102
- val = getattr(image_processor, key)
103
- if val is None:
104
- val = images_kwargs[key]
105
-
106
- return val
107
-
108
- return {k: _resolve_kw(k) for k in keys}
109
-
110
- def get_num_crops(
111
- self,
112
- *,
113
- image_width: int,
114
- image_height: int,
115
- processor: Gemma3Processor | None,
116
- ) -> int:
117
- if processor is None:
118
- processor = self.get_hf_processor()
119
-
120
- images_kwargs = self._resolve_image_kwargs(
121
- processor,
122
- {
123
- "do_pan_and_scan",
124
- "pan_and_scan_min_crop_size",
125
- "pan_and_scan_max_num_crops",
126
- "pan_and_scan_min_ratio_to_activate",
127
- },
128
- )
129
-
130
- do_pan_and_scan = images_kwargs["do_pan_and_scan"]
131
- pan_and_scan_min_crop_size = images_kwargs["pan_and_scan_min_crop_size"]
132
- pan_and_scan_max_num_crops = images_kwargs["pan_and_scan_max_num_crops"]
133
- pan_and_scan_min_ratio_to_activate = images_kwargs[
134
- "pan_and_scan_min_ratio_to_activate"
135
- ]
136
-
137
- if not do_pan_and_scan:
138
- return 0
139
-
140
- if envs.VLLM_USE_V1:
141
- logger.warning_once(
142
- "`do_pan_and_scan=True` has suboptimal results on V1 "
143
- "because of the simplified attention pattern being used."
144
- )
145
-
146
- # Based on Gemma3ImageProcessor.pan_and_scan
147
- if image_width >= image_height:
148
- if image_width / image_height < pan_and_scan_min_ratio_to_activate:
149
- return 0
150
-
151
- num_crops_w = min(
152
- int(math.floor(image_width / pan_and_scan_min_crop_size)),
153
- int(math.floor(image_width / image_height + 0.5)),
154
- )
155
-
156
- num_crops_w = max(2, num_crops_w)
157
- num_crops_w = min(pan_and_scan_max_num_crops, num_crops_w)
158
- num_crops_h = 1
159
- else:
160
- if image_height / image_width < pan_and_scan_min_ratio_to_activate:
161
- return 0
162
-
163
- num_crops_h = min(
164
- int(math.floor(image_height / pan_and_scan_min_crop_size)),
165
- int(math.floor(image_height / image_width + 0.5)),
166
- )
167
-
168
- num_crops_h = max(2, num_crops_h)
169
- num_crops_h = min(pan_and_scan_max_num_crops, num_crops_h)
170
- num_crops_w = 1
171
-
172
- crop_size_w = int(math.ceil(image_width / num_crops_w))
173
- crop_size_h = int(math.ceil(image_height / num_crops_h))
174
-
175
- if min(crop_size_w, crop_size_h) < pan_and_scan_min_crop_size:
176
- return 0
177
-
178
- return num_crops_w * num_crops_h
179
-
180
- def get_image_repl(
181
- self,
182
- *,
183
- image_width: int,
184
- image_height: int,
185
- processor: Gemma3Processor | None,
186
- ) -> PromptUpdateDetails[str]:
187
- if processor is None:
188
- processor = self.get_hf_processor()
189
-
190
- boi_token = processor.boi_token
191
-
192
- num_crops = self.get_num_crops(
193
- image_width=image_width,
194
- image_height=image_height,
195
- processor=processor,
196
- )
197
-
198
- if num_crops == 0:
199
- image_text = boi_token
200
- else:
201
- crops_image_tokens = " ".join(boi_token for _ in range(num_crops))
202
- image_text = (
203
- f"Here is the original image {boi_token} and here are some "
204
- f"crops to help you see better {crops_image_tokens}"
205
- )
206
-
207
- repl_full = image_text.replace(boi_token, processor.full_image_sequence)
208
-
209
- tokenizer = processor.tokenizer
210
- vocab = tokenizer.get_vocab()
211
- image_token_id = vocab[tokenizer.image_token]
212
-
213
- return PromptUpdateDetails.select_token_id(repl_full, image_token_id)
214
-
215
- def get_num_image_tokens(
216
- self,
217
- *,
218
- image_width: int,
219
- image_height: int,
220
- processor: Gemma3Processor | None,
221
- ) -> int:
222
- if processor is None:
223
- processor = self.get_hf_processor()
224
-
225
- num_crops = self.get_num_crops(
226
- image_width=image_width,
227
- image_height=image_height,
228
- processor=processor,
229
- )
230
- image_seq_len = processor.image_seq_length
231
-
232
- return (num_crops + 1) * image_seq_len
233
-
234
- def get_image_size_with_most_features(self) -> ImageSize:
235
- processor = self.get_hf_processor()
236
-
237
- images_kwargs = self._resolve_image_kwargs(
238
- processor, {"pan_and_scan_max_num_crops"}
239
- )
240
- max_num_crops = images_kwargs["pan_and_scan_max_num_crops"]
241
-
242
- # Result in the max possible feature size (h:w = max_num_crops:1)
243
- return ImageSize(height=50 * max_num_crops, width=50)
244
-
245
-
246
-class Gemma3DummyInputsBuilder(BaseDummyInputsBuilder[Gemma3ProcessingInfo]):
247
- def get_dummy_text(self, mm_counts: Mapping[str, int]) -> str:
248
- num_images = mm_counts.get("image", 0)
249
-
250
- processor = self.info.get_hf_processor()
251
- image_token = processor.boi_token
252
-
253
- return image_token * num_images
254
-
255
- def get_dummy_mm_data(
256
- self,
257
- seq_len: int,
258
- mm_counts: Mapping[str, int],
259
- mm_options: Mapping[str, BaseDummyOptions] | None = None,
260
- ) -> MultiModalDataDict:
261
- num_images = mm_counts.get("image", 0)
262
-
263
- target_width, target_height = self.info.get_image_size_with_most_features()
264
-
265
- image_overrides = mm_options.get("image") if mm_options else None
266
-
267
- return {
268
- "image": self._get_dummy_images(
269
- width=target_width,
270
- height=target_height,
271
- num_images=num_images,
272
- overrides=image_overrides,
273
- )
274
- }
275
-
276
-
277
-class Gemma3MultiModalProcessor(BaseMultiModalProcessor[Gemma3ProcessingInfo]):
278
- def _call_hf_processor(
279
- self,
280
- prompt: str,
281
- mm_data: Mapping[str, object],
282
- mm_kwargs: Mapping[str, object],
283
- tok_kwargs: Mapping[str, object],
284
- ) -> BatchFeature:
285
- processed_outputs = super()._call_hf_processor(
286
- prompt,
287
- mm_data,
288
- mm_kwargs,
289
- tok_kwargs,
290
- )
291
-
292
- # HF processor pops the `num_crops` kwarg, which is needed by vLLM
293
- if (images := mm_data.get("images")) is not None:
294
- parsed_images = (
295
- self._get_data_parser()
296
- .parse_mm_data({"image": images})
297
- .get_items("image", ImageProcessorItems)
298
- )
299
- image_sizes = [
300
- parsed_images.get_image_size(i) for i in range(len(parsed_images))
301
- ]
302
- hf_processor = self.info.get_hf_processor(**mm_kwargs)
303
-
304
- num_crops = [
305
- self.info.get_num_crops(
306
- image_width=size.width,
307
- image_height=size.height,
308
- processor=hf_processor,
309
- )
310
- for size in image_sizes
311
- ]
312
- processed_outputs["num_patches"] = torch.tensor(num_crops) + 1
313
-
314
- return processed_outputs
315
-
316
- def _get_mm_fields_config(
317
- self,
318
- hf_inputs: BatchFeature,
319
- hf_processor_mm_kwargs: Mapping[str, object],
320
- ) -> Mapping[str, MultiModalFieldConfig]:
321
- num_patches = hf_inputs.get("num_patches", torch.empty(0))
322
-
323
- return dict(
324
- pixel_values=MultiModalFieldConfig.flat_from_sizes("image", num_patches),
325
- num_patches=MultiModalFieldConfig.batched("image"),
326
- )
327
-
328
- def _get_prompt_updates(
329
- self,
330
- mm_items: MultiModalDataItems,
331
- hf_processor_mm_kwargs: Mapping[str, Any],
332
- out_mm_kwargs: MultiModalKwargsItems,
333
- ) -> Sequence[PromptUpdate]:
334
- hf_processor = self.info.get_hf_processor(**hf_processor_mm_kwargs)
335
- image_token = hf_processor.boi_token
336
-
337
- def get_replacement_gemma3(item_idx: int):
338
- images = mm_items.get_items("image", ImageProcessorItems)
339
-
340
- image_size = images.get_image_size(item_idx)
341
- return self.info.get_image_repl(
342
- image_width=image_size.width,
343
- image_height=image_size.height,
344
- processor=hf_processor,
345
- )
346
-
347
- return [
348
- PromptReplacement(
349
- modality="image",
350
- target=image_token,
351
- replacement=get_replacement_gemma3,
352
- )
353
- ]
354
-
355
- def _apply_token_matches(
356
- self,
357
- prompt: list[int],
358
- mm_prompt_updates: MultiModalPromptUpdates,
359
- ) -> tuple[list[int], MultiModalPromptUpdatesApplyResult]:
360
- token_ids, res = super()._apply_token_matches(prompt, mm_prompt_updates)
361
-
362
- # "\n\n\n" and "\n\n\n\n" are single tokens
363
- # Since our replacement can insert "\n\n" next to "\n"
364
- # tokens, we have to combine them to be consistent with
365
- # the output of the tokenizer
366
- tokenizer = self.info.get_tokenizer()
367
- vocab = tokenizer.get_vocab()
368
- newline_1 = vocab["\n"]
369
- newline_2 = vocab["\n\n"]
370
- newline_3 = vocab["\n\n\n"]
371
- newline_4 = vocab["\n\n\n\n"]
372
-
373
- token_ids = replace_token_matches(
374
- token_ids,
375
- [newline_1, newline_2],
376
- [newline_3],
377
- )
378
- token_ids = replace_token_matches(
379
- token_ids,
380
- [newline_2, newline_1],
381
- [newline_3],
382
- )
383
- token_ids = replace_token_matches(
384
- token_ids,
385
- [newline_2, newline_2],
386
- [newline_4],
387
- )
388
-
389
- return token_ids, res
390
-
391
- def _find_mm_placeholders(
392
- self,
393
- new_token_ids: list[int],
394
- mm_prompt_updates: MultiModalPromptUpdates,
395
- ) -> Mapping[str, list[PlaceholderFeaturesInfo]]:
396
- # We need to detect "\n\n" inside "\n\n\n" and "\n\n\n\n"
397
- tokenizer = self.info.get_tokenizer()
398
- vocab = tokenizer.get_vocab()
399
- newline_1 = vocab["\n"]
400
- newline_2 = vocab["\n\n"]
401
- newline_3 = vocab["\n\n\n"]
402
- newline_4 = vocab["\n\n\n\n"]
403
-
404
- def get_repl_toks(tok: int) -> list[int]:
405
- if tok == newline_3:
406
- return [newline_1, newline_2]
407
- if tok == newline_4:
408
- return [newline_2, newline_2]
409
-
410
- return [tok]
411
-
412
- repl_token_ids = list[int]()
413
- repl_orig_idxs = list[int]()
414
- for orig_idx, orig_tok in enumerate(new_token_ids):
415
- repl_toks = get_repl_toks(orig_tok)
416
- repl_token_ids.extend(repl_toks)
417
- repl_orig_idxs.extend(orig_idx for _ in range(len(repl_toks)))
418
-
419
- repls = super()._find_mm_placeholders(repl_token_ids, mm_prompt_updates)
420
-
421
- return {
422
- modality: [
423
- PlaceholderFeaturesInfo(
424
- modality=p.modality,
425
- item_idx=p.item_idx,
426
- start_idx=repl_orig_idxs[p.start_idx],
427
- tokens=p.tokens,
428
- is_embed=p.is_embed,
429
- )
430
- for p in placeholders
431
- ]
432
- for modality, placeholders in repls.items()
433
- }
434
-
435
-
436
-class Gemma3MultiModalProjector(nn.Module):
437
- def __init__(self, config: Gemma3Config):
438
- super().__init__()
439
-
440
- self.mm_input_projection_weight = nn.Parameter(
441
- torch.zeros(
442
- config.vision_config.hidden_size, config.text_config.hidden_size
443
- )
444
- )
445
-
446
- self.mm_soft_emb_norm = GemmaRMSNorm(
447
- config.vision_config.hidden_size, eps=config.vision_config.layer_norm_eps
448
- )
449
-
450
- self.patches_per_image = int(
451
- config.vision_config.image_size // config.vision_config.patch_size
452
- )
453
- self.tokens_per_side = int(config.mm_tokens_per_image**0.5)
454
- self.kernel_size = self.patches_per_image // self.tokens_per_side
455
- self.avg_pool = nn.AvgPool2d(
456
- kernel_size=self.kernel_size, stride=self.kernel_size
457
- )
458
-
459
- def forward(self, vision_outputs: torch.Tensor):
460
- batch_size, _, seq_length = vision_outputs.shape
461
-
462
- reshaped_vision_outputs = vision_outputs.transpose(1, 2)
463
- reshaped_vision_outputs = reshaped_vision_outputs.reshape(
464
- batch_size, seq_length, self.patches_per_image, self.patches_per_image
465
- )
466
- reshaped_vision_outputs = reshaped_vision_outputs.contiguous()
467
-
468
- pooled_vision_outputs = self.avg_pool(reshaped_vision_outputs)
469
- pooled_vision_outputs = pooled_vision_outputs.flatten(2)
470
- pooled_vision_outputs = pooled_vision_outputs.transpose(1, 2)
471
-
472
- normed_vision_outputs = self.mm_soft_emb_norm(pooled_vision_outputs)
473
-
474
- projected_vision_outputs = torch.matmul(
475
- normed_vision_outputs, self.mm_input_projection_weight
476
- )
477
- return projected_vision_outputs.type_as(vision_outputs)
478
-
479
-
480
-@MULTIMODAL_REGISTRY.register_processor(
481
- Gemma3MultiModalProcessor,
482
- info=Gemma3ProcessingInfo,
483
- dummy_inputs=Gemma3DummyInputsBuilder,
484
-)
485
-class Gemma3ForConditionalGeneration(
486
- nn.Module, SupportsMultiModal, SupportsPP, SupportsLoRA
487
-):
488
- merge_by_field_config = True
489
-
490
- packed_modules_mapping = {
491
- "qkv_proj": [
492
- "q_proj",
493
- "k_proj",
494
- "v_proj",
495
- ],
496
- "gate_up_proj": [
497
- "gate_proj",
498
- "up_proj",
499
- ],
500
- }
501
-
502
- hf_to_vllm_mapper = WeightsMapper(
503
- orig_to_new_prefix={
504
- # mapping for new names in checkpoint saved after transformers v4.52
505
- "model.language_model.": "language_model.model.",
506
- "model.vision_tower.": "vision_tower.",
507
- "model.multi_modal_projector.": "multi_modal_projector.",
508
- "lm_head.": "language_model.lm_head.",
509
- }
510
- )
511
-
512
- @classmethod
513
- def get_placeholder_str(cls, modality: str, i: int) -> str | None:
514
- if modality.startswith("image"):
515
- return "<start_of_image>"
516
-
517
- raise ValueError("Only image modality is supported")
518
-
519
- def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
520
- super().__init__()
521
- config = vllm_config.model_config.hf_config
522
- quant_config = vllm_config.quant_config
523
- multimodal_config = vllm_config.model_config.multimodal_config
524
- self.config = config
525
- self.quant_config = quant_config
526
- self.multimodal_config = multimodal_config
527
-
528
- self.vision_tower = SiglipVisionModel(
529
- config.vision_config,
530
- quant_config,
531
- prefix=maybe_prefix(prefix, "vision_tower"),
532
- )
533
- self.multi_modal_projector = Gemma3MultiModalProjector(config)
534
-
535
- self.language_model = init_vllm_registered_model(
536
- vllm_config=vllm_config,
537
- hf_config=config.text_config,
538
- prefix=maybe_prefix(prefix, "language_model"),
539
- architectures=["Gemma3ForCausalLM"],
540
- )
541
- logit_scale = getattr(config, "logit_scale", 1.0)
542
-
543
- if hasattr(self.language_model, "logits_processor"):
544
- # The logits processor can be unset if we're using
545
- # automatic conversion to pooling model.
546
- self.language_model.logits_processor.scale *= logit_scale
547
-
548
- self.make_empty_intermediate_tensors = (
549
- self.language_model.make_empty_intermediate_tensors
550
- )
551
-
552
- @property
553
- def dtype(self):
554
- return next(self.parameters()).dtype
555
-
556
- def _parse_and_validate_image_input(
557
- self, **kwargs: object
558
- ) -> Gemma3ImageInputs | None:
559
- pixel_values = kwargs.pop("pixel_values", None)
560
- num_patches = kwargs.pop("num_patches", None)
561
- image_embeds = kwargs.pop("image_embeds", None)
562
- assert image_embeds is None, "Gemma3 does not support image_embeds."
563
- if pixel_values is None:
564
- return None
565
-
566
- image_size = self.config.vision_config.image_size
567
-
568
- return Gemma3ImagePixelInputs(
569
- pixel_values=pixel_values,
570
- num_patches=num_patches,
571
- resolve_bindings={"h": image_size, "w": image_size},
572
- )
573
-
574
- def _image_pixels_to_features(
575
- self,
576
- vision_tower: SiglipVisionModel,
577
- pixel_values: torch.Tensor,
578
- ) -> torch.Tensor:
579
- return vision_tower(pixel_values)
580
-
581
- def _process_image_input(
582
- self,
583
- image_input: Gemma3ImageInputs,
584
- ) -> list[torch.Tensor]:
585
- assert self.vision_tower is not None
586
-
587
- pixel_values = image_input["pixel_values"]
588
- num_patches = image_input["num_patches"]
589
-
590
- image_features = self._image_pixels_to_features(
591
- self.vision_tower,
592
- pixel_values,
593
- )
594
- image_embeds = self.multi_modal_projector(image_features)
595
-
596
- return [e.flatten(0, 1) for e in image_embeds.split(num_patches.tolist())]
597
-
598
- def get_language_model(self) -> torch.nn.Module:
599
- return self.language_model
600
-
601
- def get_multimodal_embeddings(self, **kwargs: object) -> MultiModalEmbeddings:
602
- image_input = self._parse_and_validate_image_input(**kwargs)
603
- if image_input is None:
604
- return []
605
-
606
- return self._process_image_input(image_input)
607
-
608
- def forward(
609
- self,
610
- input_ids: torch.Tensor,
611
- positions: torch.Tensor,
612
- intermediate_tensors: IntermediateTensors | None = None,
613
- inputs_embeds: torch.Tensor | None = None,
614
- **kwargs: object,
615
- ) -> IntermediateTensors:
616
- if intermediate_tensors is not None:
617
- inputs_embeds = None
618
-
619
- hidden_states = self.language_model.model(
620
- input_ids,
621
- positions,
622
- intermediate_tensors,
623
- inputs_embeds=inputs_embeds,
624
- **kwargs,
625
- )
626
-
627
- return hidden_states
628
-
629
- def prepare_attn_masks(
630
- self,
631
- input_ids: torch.Tensor,
632
- positions: torch.Tensor,
633
- mask_dtype: torch.dtype,
634
- **kwargs,
635
- ):
636
- kwargs["has_images"] = True
637
- # NOTE(woosuk): Here, we distinguish the sequences by the position id 0.
638
- # This is a HACK. Fix this.
639
- start_indices = (positions == 0).cpu().nonzero()
640
- num_seqs = len(start_indices)
641
- seq_lens = []
642
- for i in range(num_seqs):
643
- start_idx = start_indices[i].item()
644
- if i < num_seqs - 1:
645
- end_idx = start_indices[i + 1].item()
646
- else:
647
- end_idx = len(input_ids)
648
- seq_lens.append(end_idx - start_idx)
649
- kwargs["seq_lens"] = seq_lens
650
-
651
- global_attn_masks = []
652
- local_attn_masks = []
653
- start_idx = 0
654
- for seq_len in seq_lens:
655
- end_idx = start_idx + seq_len
656
- input_token_ids = input_ids[start_idx:end_idx]
657
- start_idx = end_idx
658
- # Create a global causal mask.
659
- global_attn_mask = torch.empty(
660
- 1,
661
- 1,
662
- seq_len,
663
- seq_len,
664
- dtype=mask_dtype,
665
- device=input_ids.device,
666
- )
667
- global_attn_mask.fill_(float("-inf"))
668
- # Fill the lower triangle with 0.
669
- global_attn_mask = global_attn_mask.triu(diagonal=1)
670
-
671
- # Consider the bidirectional attention between image tokens.
672
- img_mask = torch.zeros_like(global_attn_mask)
673
- img_pos = input_token_ids == self.config.image_token_index
674
- img_mask[:, :, :, img_pos] += 1
675
- img_mask[:, :, img_pos, :] += 1
676
- global_attn_mask = torch.where(img_mask == 2, 0, global_attn_mask)
677
- global_attn_masks.append(global_attn_mask)
678
-
679
- sliding_window = self.config.text_config.sliding_window
680
- if sliding_window is not None:
681
- # Create a local causal mask with sliding window (1024).
682
- local_attn_mask = torch.ones_like(global_attn_mask)
683
- local_attn_mask = torch.tril(local_attn_mask, diagonal=-sliding_window)
684
- local_attn_mask = torch.where(
685
- local_attn_mask == 0, global_attn_mask, float("-inf")
686
- )
687
- local_attn_masks.append(local_attn_mask)
688
- kwargs["global_attn_masks"] = global_attn_masks
689
- kwargs["local_attn_masks"] = local_attn_masks
690
- return kwargs
691
-
692
- def compute_logits(
693
- self,
694
- hidden_states: torch.Tensor,
695
- ) -> torch.Tensor | None:
696
- return self.language_model.compute_logits(hidden_states)
697
-
698
- def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
699
- loader = AutoWeightsLoader(self)
700
- return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
701
-
702
- def get_mm_mapping(self) -> MultiModelKeys:
703
- """
704
- Get the module prefix in multimodal models
705
- """
706
- return MultiModelKeys.from_string_field(
707
- language_model="language_model",
708
- connector="multi_modal_projector",
709
- tower_model="vision_tower",
710
- )
vllm/model_executor/models/paligemma.py
deleted
-412
@@ -1,412 +0,0 @@
1
-# SPDX-License-Identifier: Apache-2.0
2
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-from collections.abc import Iterable, Mapping, Sequence
4
-from typing import Annotated, Literal, TypeAlias
5
-
6
-import torch
7
-from torch import nn
8
-from transformers import BatchFeature, PaliGemmaConfig
9
-
10
-from vllm.config import VllmConfig
11
-from vllm.config.multimodal import BaseDummyOptions
12
-from vllm.logger import init_logger
13
-from vllm.multimodal import MULTIMODAL_REGISTRY
14
-from vllm.multimodal.inputs import (
15
- MultiModalDataDict,
16
- MultiModalFieldConfig,
17
- MultiModalInputs,
18
- MultiModalKwargsItems,
19
- MultiModalUUIDDict,
20
-)
21
-from vllm.multimodal.parse import (
22
- ImageEmbeddingItems,
23
- ImageProcessorItems,
24
- MultiModalDataItems,
25
-)
26
-from vllm.multimodal.processing import (
27
- BaseMultiModalProcessor,
28
- BaseProcessingInfo,
29
- PromptIndexTargets,
30
- PromptInsertion,
31
- PromptUpdate,
32
- PromptUpdateDetails,
33
-)
34
-from vllm.multimodal.profiling import BaseDummyInputsBuilder
35
-from vllm.sequence import IntermediateTensors
36
-from vllm.utils.tensor_schema import TensorSchema, TensorShape
37
-
38
-from .interfaces import MultiModalEmbeddings, SupportsMultiModal, SupportsPP
39
-from .siglip import SiglipVisionModel
40
-from .utils import (
41
- AutoWeightsLoader,
42
- WeightsMapper,
43
- flatten_bn,
44
- init_vllm_registered_model,
45
- maybe_prefix,
46
-)
47
-from .vision import get_vision_encoder_info
48
-
49
-logger = init_logger(__name__)
50
-
51
-
52
-class PaliGemmaImagePixelInputs(TensorSchema):
53
- """
54
- Dimensions:
55
- - bn: Batch size * number of images
56
- - c: Number of channels (3)
57
- - h: Height
58
- - w: Width
59
- """
60
-
61
- type: Literal["pixel_values"] = "pixel_values"
62
- data: Annotated[torch.Tensor, TensorShape("bn", 3, "h", "w")]
63
-
64
-
65
-class PaliGemmaImageEmbeddingInputs(TensorSchema):
66
- """
67
- Dimensions:
68
- - bn: Batch size * number of images
69
- - ifs: Image feature size
70
- - hs: Hidden size (must match language model backbone)
71
- """
72
-
73
- type: Literal["image_embeds"] = "image_embeds"
74
- data: Annotated[torch.Tensor, TensorShape("bn", "ifs", "hs")]
75
-
76
-
77
-PaliGemmaImageInputs: TypeAlias = (
78
- PaliGemmaImagePixelInputs | PaliGemmaImageEmbeddingInputs
79
-)
80
-
81
-
82
-class PaliGemmaMultiModalProjector(nn.Module):
83
- def __init__(self, vision_hidden_size: int, projection_dim: int):
84
- super().__init__()
85
-
86
- self.linear = nn.Linear(vision_hidden_size, projection_dim, bias=True)
87
-
88
- def forward(self, image_features: torch.Tensor) -> torch.Tensor:
89
- hidden_states = self.linear(image_features)
90
- return hidden_states
91
-
92
-
93
-class PaliGemmaProcessingInfo(BaseProcessingInfo):
94
- def get_hf_config(self):
95
- return self.ctx.get_hf_config(PaliGemmaConfig)
96
-
97
- def get_vision_encoder_info(self):
98
- return get_vision_encoder_info(self.get_hf_config())
99
-
100
- def get_supported_mm_limits(self) -> Mapping[str, int | None]:
101
- return {"image": 1}
102
-
103
- def get_num_image_tokens(
104
- self,
105
- *,
106
- image_width: int,
107
- image_height: int,
108
- ) -> int:
109
- vision_encoder_info = self.get_vision_encoder_info()
110
-
111
- return vision_encoder_info.get_num_image_tokens(
112
- image_width=image_width,
113
- image_height=image_height,
114
- )
115
-
116
-
117
-class PaliGemmaDummyInputsBuilder(BaseDummyInputsBuilder[PaliGemmaProcessingInfo]):
118
- def get_dummy_text(self, mm_counts: Mapping[str, int]) -> str:
119
- return ""
120
-
121
- def get_dummy_mm_data(
122
- self,
123
- seq_len: int,
124
- mm_counts: Mapping[str, int],
125
- mm_options: Mapping[str, BaseDummyOptions] | None = None,
126
- ) -> MultiModalDataDict:
127
- hf_config = self.info.get_hf_config()
128
- vision_config = hf_config.vision_config
129
- max_image_size = vision_config.image_size
130
-
131
- num_images = mm_counts.get("image", 0)
132
-
133
- image_overrides = mm_options.get("image") if mm_options else None
134
-
135
- return {
136
- "image": self._get_dummy_images(
137
- width=max_image_size,
138
- height=max_image_size,
139
- num_images=num_images,
140
- overrides=image_overrides,
141
- )
142
- }
143
-
144
-
145
-class PaliGemmaMultiModalProcessor(BaseMultiModalProcessor[PaliGemmaProcessingInfo]):
146
- def _call_hf_processor(
147
- self,
148
- prompt: str,
149
- mm_data: Mapping[str, object],
150
- mm_kwargs: Mapping[str, object],
151
- tok_kwargs: Mapping[str, object],
152
- ) -> BatchFeature:
153
- tokenizer = self.info.get_tokenizer()
154
- if not mm_data:
155
- prompt_ids = tokenizer.encode(prompt, add_special_tokens=False)
156
- return BatchFeature(dict(input_ids=[prompt_ids]), tensor_type="pt")
157
-
158
- return super()._call_hf_processor(
159
- prompt=prompt,
160
- mm_data=mm_data,
161
- mm_kwargs=mm_kwargs,
162
- tok_kwargs=tok_kwargs,
163
- )
164
-
165
- def _get_mm_fields_config(
166
- self,
167
- hf_inputs: BatchFeature,
168
- hf_processor_mm_kwargs: Mapping[str, object],
169
- ) -> Mapping[str, MultiModalFieldConfig]:
170
- return dict(pixel_values=MultiModalFieldConfig.batched("image"))
171
-
172
- def _get_prompt_updates(
173
- self,
174
- mm_items: MultiModalDataItems,
175
- hf_processor_mm_kwargs: Mapping[str, object],
176
- out_mm_kwargs: MultiModalKwargsItems,
177
- ) -> Sequence[PromptUpdate]:
178
- hf_config = self.info.get_hf_config()
179
- image_token_id = hf_config.image_token_index
180
-
181
- tokenizer = self.info.get_tokenizer()
182
-
183
- bos_token_id = tokenizer.bos_token_id
184
- assert isinstance(bos_token_id, int)
185
-
186
- def get_insertion(item_idx: int):
187
- images = mm_items.get_items(
188
- "image", (ImageEmbeddingItems, ImageProcessorItems)
189
- )
190
-
191
- if isinstance(images, ImageEmbeddingItems):
192
- num_image_tokens = images.get_feature_size(item_idx)
193
- else:
194
- image_size = images.get_image_size(item_idx)
195
- num_image_tokens = self.info.get_num_image_tokens(
196
- image_width=image_size.width,
197
- image_height=image_size.height,
198
- )
199
-
200
- image_tokens = [image_token_id] * num_image_tokens
201
-
202
- return PromptUpdateDetails.select_token_id(
203
- image_tokens + [bos_token_id],
204
- embed_token_id=image_token_id,
205
- )
206
-
207
- # Paligemma 1 and 2 have different tokenizer.add_bos_token
208
- # Insert <image>*n + <bos> after <bos> for Paligemma 1
209
- # Insert <image>*n + <bos> for Paligemma 2
210
- return [
211
- PromptInsertion(
212
- modality="image",
213
- target=PromptIndexTargets.prefix(
214
- [bos_token_id] if tokenizer.add_bos_token else []
215
- ),
216
- insertion=get_insertion,
217
- )
218
- ]
219
-
220
- def apply(
221
- self,
222
- prompt: str | list[int],
223
- mm_data: MultiModalDataDict,
224
- hf_processor_mm_kwargs: Mapping[str, object],
225
- tokenization_kwargs: Mapping[str, object] | None = None,
226
- mm_uuids: MultiModalUUIDDict | None = None,
227
- ) -> MultiModalInputs:
228
- mm_inputs = super().apply(
229
- prompt,
230
- mm_data,
231
- hf_processor_mm_kwargs,
232
- tokenization_kwargs,
233
- mm_uuids=mm_uuids,
234
- )
235
- prompt_token_ids = mm_inputs["prompt_token_ids"]
236
-
237
- tokenizer = self.info.get_tokenizer()
238
- newline_prompt = "\n"
239
- newline_token_id = tokenizer.encode(newline_prompt)[-1] # 108
240
- # Force to add newline at the end of prompt for paligemma's format
241
- # This step can NOT be replacemented by current PromptUpdate methods
242
- if len(prompt_token_ids) and prompt_token_ids[-1] != newline_token_id:
243
- prompt_token_ids.append(newline_token_id)
244
- mm_inputs["prompt_token_ids"] = prompt_token_ids
245
-
246
- return mm_inputs
247
-
248
-
249
-@MULTIMODAL_REGISTRY.register_processor(
250
- PaliGemmaMultiModalProcessor,
251
- info=PaliGemmaProcessingInfo,
252
- dummy_inputs=PaliGemmaDummyInputsBuilder,
253
-)
254
-class PaliGemmaForConditionalGeneration(nn.Module, SupportsMultiModal, SupportsPP):
255
- packed_modules_mapping = {
256
- "qkv_proj": [
257
- "q_proj",
258
- "k_proj",
259
- "v_proj",
260
- ],
261
- "gate_up_proj": [
262
- "gate_proj",
263
- "up_proj",
264
- ],
265
- }
266
-
267
- hf_to_vllm_mapper = WeightsMapper(
268
- orig_to_new_prefix={
269
- # mapping for new names in checkpoint saved after transformers v4.52
270
- "model.language_model.": "language_model.model.",
271
- "model.vision_tower.": "vision_tower.",
272
- "model.multi_modal_projector.": "multi_modal_projector.",
273
- "lm_head.": "language_model.lm_head.",
274
- }
275
- )
276
-
277
- @classmethod
278
- def get_placeholder_str(cls, modality: str, i: int) -> str | None:
279
- if modality.startswith("image"):
280
- return None
281
-
282
- raise ValueError("Only image modality is supported")
283
-
284
- def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
285
- super().__init__()
286
- config = vllm_config.model_config.hf_config
287
- quant_config = vllm_config.quant_config
288
- multimodal_config = vllm_config.model_config.multimodal_config
289
- self.config = config
290
- self.multimodal_config = multimodal_config
291
-
292
- self.vision_tower = SiglipVisionModel(
293
- config.vision_config,
294
- quant_config,
295
- prefix=maybe_prefix(prefix, "vision_tower"),
296
- )
297
- self.multi_modal_projector = PaliGemmaMultiModalProjector(
298
- vision_hidden_size=config.vision_config.hidden_size,
299
- projection_dim=config.vision_config.projection_dim,
300
- )
301
-
302
- self.quant_config = quant_config
303
-
304
- if config.text_config.model_type == "gemma":
305
- config.text_config.architectures = ["GemmaForCausalLM"]
306
- else:
307
- config.text_config.architectures = ["Gemma2ForCausalLM"]
308
- self.language_model = init_vllm_registered_model(
309
- vllm_config=vllm_config,
310
- hf_config=config.text_config,
311
- prefix=maybe_prefix(prefix, "language_model"),
312
- )
313
- logit_scale = getattr(config, "logit_scale", 1.0)
314
- self.language_model.logits_processor.scale *= logit_scale
315
-
316
- self.make_empty_intermediate_tensors = (
317
- self.language_model.make_empty_intermediate_tensors
318
- )
319
-
320
- def _parse_and_validate_image_input(
321
- self, **kwargs: object
322
- ) -> PaliGemmaImageInputs | None:
323
- pixel_values = kwargs.pop("pixel_values", None)
324
- image_embeds = kwargs.pop("image_embeds", None)
325
-
326
- if pixel_values is None and image_embeds is None:
327
- return None
328
-
329
- if pixel_values is not None:
330
- pixel_values = flatten_bn(pixel_values, concat=True)
331
-
332
- h = w = self.config.vision_config.image_size
333
- return PaliGemmaImagePixelInputs(
334
- type="pixel_values",
335
- data=pixel_values,
336
- resolve_bindings={"h": h, "w": w},
337
- )
338
-
339
- if image_embeds is not None:
340
- image_embeds = flatten_bn(image_embeds, concat=True)
341
-
342
- return PaliGemmaImageEmbeddingInputs(
343
- type="image_embeds",
344
- data=image_embeds,
345
- )
346
-
347
- raise AssertionError("This line should be unreachable.")
348
-
349
- def _image_pixels_to_features(
350
- self,
351
- vision_tower: SiglipVisionModel,
352
- pixel_values: torch.Tensor,
353
- ) -> torch.Tensor:
354
- target_dtype = vision_tower.get_input_embeddings().weight.dtype
355
- image_features = vision_tower(pixel_values.to(dtype=target_dtype))
356
-
357
- return image_features
358
-
359
- def _process_image_input(
360
- self,
361
- image_input: PaliGemmaImageInputs,
362
- ) -> torch.Tensor:
363
- if image_input["type"] == "image_embeds":
364
- return image_input["data"]
365
-
366
- assert self.vision_tower is not None
367
- pixel_values = image_input["data"]
368
- image_features = self._image_pixels_to_features(
369
- self.vision_tower,
370
- pixel_values,
371
- )
372
-
373
- return self.multi_modal_projector(image_features)
374
-
375
- def get_language_model(self) -> torch.nn.Module:
376
- return self.language_model
377
-
378
- def get_multimodal_embeddings(self, **kwargs: object) -> MultiModalEmbeddings:
379
- image_input = self._parse_and_validate_image_input(**kwargs)
380
- if image_input is None:
381
- return []
382
- vision_embeddings = self._process_image_input(image_input)
383
- # https://github.com/huggingface/transformers/blob/main/src/transformers/models/paligemma/modeling_paligemma.py#L294 # noqa
384
- vision_embeddings = vision_embeddings * (self.config.hidden_size**-0.5)
385
- return vision_embeddings
386
-
387
- def forward(
388
- self,
389
- input_ids: torch.Tensor,
390
- positions: torch.Tensor,
391
- intermediate_tensors: IntermediateTensors | None = None,
392
- inputs_embeds: torch.Tensor | None = None,
393
- **kwargs: object,
394
- ) -> IntermediateTensors:
395
- if intermediate_tensors is not None:
396
- inputs_embeds = None
397
-
398
- hidden_states = self.language_model.model(
399
- input_ids, positions, intermediate_tensors, inputs_embeds=inputs_embeds
400
- )
401
-
402
- return hidden_states
403
-
404
- def compute_logits(
405
- self,
406
- hidden_states: torch.Tensor,
407
- ) -> torch.Tensor | None:
408
- return self.language_model.compute_logits(hidden_states)
409
-
410
- def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
411
- loader = AutoWeightsLoader(self)
412
- return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
vllm/model_executor/models/registry.py
+8
-5
@@ -263,7 +263,6 @@ _MULTIMODAL_MODELS = {
263
"Ernie4_5_VLMoeForConditionalGeneration",
264
),
265
"FuyuForCausalLM": ("fuyu", "FuyuForCausalLM"),
266
- "Gemma3ForConditionalGeneration": ("gemma3_mm", "Gemma3ForConditionalGeneration"), # noqa: E501
266
"Gemma3nForConditionalGeneration": (
267
"gemma3n_mm",
268
"Gemma3nForConditionalGeneration",
@@ -329,10 +328,6 @@ _MULTIMODAL_MODELS = {
328
"NVLM_D": ("nvlm_d", "NVLM_D_Model"),
329
"Ovis": ("ovis", "Ovis"),
330
"Ovis2_5": ("ovis2_5", "Ovis2_5"),
332
- "PaliGemmaForConditionalGeneration": (
333
- "paligemma",
334
- "PaliGemmaForConditionalGeneration",
335
- ),
331
"Phi3VForCausalLM": ("phi3v", "Phi3VForCausalLM"),
332
"Phi4MMForCausalLM": ("phi4mm", "Phi4MMForCausalLM"),
333
"Phi4MultimodalForCausalLM": ("phi4_multimodal", "Phi4MultimodalForCausalLM"), # noqa: E501
@@ -405,6 +400,14 @@ _TRANSFORMERS_SUPPORTED_MODELS = {
400
"transformers",
401
"TransformersMultiModalForCausalLM",
402
),
403
+ "Gemma3ForConditionalGeneration": (
404
+ "transformers",
405
+ "TransformersMultiModalForCausalLM",
406
+ ),
407
+ "PaliGemmaForConditionalGeneration": (
408
+ "transformers",
409
+ "TransformersMultiModalForCausalLM",
410
+ ),
411
}
412
413
_TRANSFORMERS_BACKEND_MODELS = {
vllm/platforms/rocm.py
-3
@@ -59,9 +59,6 @@ _ROCM_PARTIALLY_SUPPORTED_MODELS: dict[str, str] = {
59
"Qwen2ForCausalLM": _ROCM_SWA_REASON,
60
"MistralForCausalLM": _ROCM_SWA_REASON,
61
"MixtralForCausalLM": _ROCM_SWA_REASON,
62
- "PaliGemmaForConditionalGeneration": (
63
- "ROCm flash attention does not yet fully support 32-bit precision on PaliGemma"
64
- ),
62
"Phi3VForCausalLM": (
63
"ROCm Triton flash attention may run into compilation errors due to "
64
"excessive use of shared memory. If this happens, disable Triton FA "