@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / e05a6754a8

Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 14, 05:30 UTC

[Model] Revert PR #26715: Restore custom PaliGemma and Gemma3-MM impl… (#27309)

Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com> Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com>

Luciano Martins committed Oct 22, 2025 at 14:05 UTC e05a6754a8e6adf710b3882d6fe4301fa0ad90e8

12 files changed +1219 -54

docs/models/hardware_supported_models/tpu.md

+2 -2

@@ -16,8 +16,8 @@

16 | meta-llama/Llama-4-* | Llama4ForConditionalGeneration | ❌ |

17 | microsoft/Phi-3-mini-128k-instruct | Phi3ForCausalLM | 🟨 |

18 | microsoft/phi-4 | Phi3ForCausalLM | ❌ |

19 -| google/gemma-3-27b-it | TransformersForMultimodalLM | 🟨 |

20 -| google/gemma-3-4b-it | TransformersForMultimodalLM | ❌ |

19 +| google/gemma-3-27b-it | Gemma3ForConditionalGeneration | 🟨 |

20 +| google/gemma-3-4b-it | Gemma3ForConditionalGeneration | ❌ |

21 | deepseek-ai/DeepSeek-R1 | DeepseekV3ForCausalLM | ❌ |

22 | deepseek-ai/DeepSeek-V3 | DeepseekV3ForCausalLM | ❌ |

23 | RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8 | LlamaForCausalLM | ✅ |

docs/models/supported_models.md

+20 -3

@@ -642,6 +642,7 @@ These models primarily accept the [`LLM.generate`](./generative_models.md#llmgen

642 | `DeepseekOCRForCausalLM` | DeepSeek-OCR | T + I+ | `deepseek-ai/DeepSeek-OCR`, etc. | | ✅︎ |

643 | `Ernie4_5_VLMoeForConditionalGeneration` | Ernie4.5-VL | T + I+/ V+ | `baidu/ERNIE-4.5-VL-28B-A3B-PT`, `baidu/ERNIE-4.5-VL-424B-A47B-PT` | | ✅︎ |

644 | `FuyuForCausalLM` | Fuyu | T + I | `adept/fuyu-8b`, etc. | | ✅︎ |

645 +| `Gemma3ForConditionalGeneration` | Gemma 3 | T + I+ | `google/gemma-3-4b-it`, `google/gemma-3-27b-it`, etc. | ✅︎ | ✅︎ |

647 | `GLM4VForCausalLM`^ | GLM-4V | T + I | `zai-org/glm-4v-9b`, `zai-org/cogagent-9b-20241220`, etc. | ✅︎ | ✅︎ |

648 | `Glm4vForConditionalGeneration` | GLM-4.1V-Thinking | T + IE+ + VE+ | `zai-org/GLM-4.1V-9B-Thinking`, etc. | ✅︎ | ✅︎ |

@@ -671,6 +672,7 @@ These models primarily accept the [`LLM.generate`](./generative_models.md#llmgen

672 | `NVLM_D_Model` | NVLM-D 1.0 | T + I+ | `nvidia/NVLM-D-72B`, etc. | | ✅︎ |

673 | `Ovis` | Ovis2, Ovis1.6 | T + I+ | `AIDC-AI/Ovis2-1B`, `AIDC-AI/Ovis1.6-Llama3.2-3B`, etc. | | ✅︎ |

675 +| `PaliGemmaForConditionalGeneration` | PaliGemma, PaliGemma 2 | T + IE | `google/paligemma-3b-pt-224`, `google/paligemma-3b-mix-224`, `google/paligemma2-3b-ft-docci-448`, etc. | | ✅︎ |

676 | `Phi3VForCausalLM` | Phi-3-Vision, Phi-3.5-Vision | T + IE+ | `microsoft/Phi-3-vision-128k-instruct`, `microsoft/Phi-3.5-vision-instruct`, etc. | | ✅︎ |

677 | `Phi4MMForCausalLM` | Phi-4-multimodal | T + I+ / T + A+ / I+ + A+ | `microsoft/Phi-4-multimodal-instruct`, etc. | ✅︎ | ✅︎ |

678 | `Phi4MultimodalForCausalLM` | Phi-4-multimodal (HF Transformers) | T + I+ / T + A+ / I+ + A+ | `microsoft/Phi-4-multimodal-instruct` (with revision `refs/pr/70`), etc. | ✅︎ | ✅︎ |

@@ -695,8 +697,6 @@ Some models are supported only via the [Transformers backend](#transformers). Th

698 |--------------|--------|--------|-------------------|-----------------------------|-----------------------------------------|

699 | `Emu3ForConditionalGeneration` | Emu3 | T + I | `BAAI/Emu3-Chat-hf` | ✅︎ | ✅︎ |

698 -| `Gemma3ForConditionalGeneration` | Gemma 3 | T + I+ | `google/gemma-3-4b-it`, `google/gemma-3-27b-it`, etc. | ✅︎ | ✅︎ |

699 -| `PaliGemmaForConditionalGeneration` | PaliGemma, PaliGemma 2 | T + IE | `google/paligemma-3b-pt-224`, `google/paligemma-3b-mix-224`, `google/paligemma2-3b-ft-docci-448`, etc. | ✅︎ | ✅︎ |

700

701 ^ You need to set the architecture name via `--hf-overrides` to match the one in vLLM.

702     • For example, to use DeepSeek-VL2 series models:

@@ -705,7 +705,21 @@ Some models are supported only via the [Transformers backend](#transformers). Th

705 + Multiple items can be inputted per text prompt for this modality.

706

707 !!! warning

708 - For `Gemma3ForConditionalGeneration`, `{"do_pan_and_scan": true}` is not supported in Transformers backend yet.

708 + Both V0 and V1 support `Gemma3ForConditionalGeneration` for text-only inputs.

709 + However, there are differences in how they handle text + image inputs:

710 +

711 + V0 correctly implements the model's attention pattern:

712 + - Uses bidirectional attention between the image tokens corresponding to the same image

713 + - Uses causal attention for other tokens

714 + - Implemented via (naive) PyTorch SDPA with masking tensors

715 + - Note: May use significant memory for long prompts with image

716 +

717 + V1 currently uses a simplified attention pattern:

718 + - Uses causal attention for all tokens, including image tokens

719 + - Generates reasonable outputs but does not match the original model's attention for text + image inputs, especially when `{"do_pan_and_scan": true}`

720 + - Will be updated in the future to support the correct behavior

721 +

722 + This limitation exists because the model's mixed attention pattern (bidirectional for images, causal otherwise) is not yet supported by vLLM's attention backends.

723

724 !!! note

725 `Gemma3nForConditionalGeneration` is only supported on V1 due to shared KV caching and it depends on `timm>=1.0.17` to make use of its

@@ -757,6 +771,9 @@ Some models are supported only via the [Transformers backend](#transformers). Th

771 The official `openbmb/MiniCPM-V-2` doesn't work yet, so we need to use a fork (`HwwwH/MiniCPM-V-2`) for now.

772 For more details, please see: <https://github.com/vllm-project/vllm/pull/4087#issuecomment-2250397630>

773

774 +!!! warning

775 + Our PaliGemma implementations have the same problem as Gemma 3 (see above) for both V0 and V1.

776 +

777 !!! note

778 For Qwen2.5-Omni and Qwen3-Omni, reading audio from video pre-processing (`--mm-processor-kwargs '{"use_audio_in_video": true}'`) is currently work in progress and not yet supported.

779

examples/offline_inference/vision_language.py

+1 -2

@@ -319,8 +319,7 @@ def run_gemma3(questions: list[str], modality: str) -> ModelRequestData:

319 model=model_name,

320 max_model_len=2048,

321 max_num_seqs=2,

322 - # TODO: Support this in transformers backend

323 - # mm_processor_kwargs={"do_pan_and_scan": True},

322 + mm_processor_kwargs={"do_pan_and_scan": True},

323 limit_mm_per_prompt={modality: 1},

324 )

325

tests/models/language/generation/test_gemma.py

+11 -5

@@ -3,7 +3,7 @@

3 import numpy as np

4 import pytest

6 -MODELS = ["google/gemma-2b", "google/gemma-2-2b"]

6 +MODELS = ["google/gemma-2b", "google/gemma-2-2b", "google/gemma-3-4b-it"]

9 @pytest.mark.parametrize("model", MODELS)

@@ -14,8 +14,14 @@ def test_dummy_loader(vllm_runner, monkeypatch, model: str) -> None:

14 model,

15 load_format="dummy",

16 ) as llm:

17 - normalizers = llm.apply_model(

18 - lambda model: model.model.normalizer.cpu().item()

19 - )

20 - config = llm.llm.llm_engine.model_config.hf_config

17 + if model == "google/gemma-3-4b-it":

18 + normalizers = llm.llm.collective_rpc(

19 + lambda self: self.model_runner.model.language_model.model.normalizer.cpu().item() # noqa: E501

20 + )

21 + config = llm.llm.llm_engine.model_config.hf_config.text_config

22 + else:

23 + normalizers = llm.llm.collective_rpc(

24 + lambda self: self.model_runner.model.model.normalizer.cpu().item()

25 + )

26 + config = llm.llm.llm_engine.model_config.hf_config

27 assert np.allclose(normalizers, config.hidden_size**0.5, rtol=2e-3)

tests/models/multimodal/generation/test_common.py

+40 -34

@@ -112,6 +112,25 @@ VLM_TEST_SETTINGS = {

112 vllm_runner_kwargs={"enable_mm_embeds": True},

113 marks=[pytest.mark.core_model, pytest.mark.cpu_model],

114 ),

115 + "paligemma": VLMTestInfo(

116 + models=["google/paligemma-3b-mix-224"],

117 + test_type=VLMTestType.IMAGE,

118 + prompt_formatter=identity,

119 + img_idx_to_prompt=lambda idx: "",

120 + # Paligemma uses its own sample prompts because the default one fails

121 + single_image_prompts=IMAGE_ASSETS.prompts(

122 + {

123 + "stop_sign": "caption es",

124 + "cherry_blossom": "What is in the picture?",

125 + }

126 + ),

127 + auto_cls=AutoModelForImageTextToText,

128 + vllm_output_post_proc=model_utils.paligemma_vllm_to_hf_output,

129 + dtype="bfloat16",

130 + marks=[

131 + pytest.mark.skip(reason="vLLM does not support PrefixLM attention mask")

132 + ],

133 + ),

134 "qwen2_5_vl": VLMTestInfo(

135 models=["Qwen/Qwen2.5-VL-3B-Instruct"],

136 test_type=(VLMTestType.IMAGE, VLMTestType.MULTI_IMAGE, VLMTestType.VIDEO),

@@ -176,24 +195,14 @@ VLM_TEST_SETTINGS = {

195 # Gemma3 has bidirectional mask on images

196 "gemma3-transformers": VLMTestInfo(

197 models=["google/gemma-3-4b-it"],

179 - test_type=(VLMTestType.IMAGE, VLMTestType.MULTI_IMAGE),

180 - prompt_formatter=lambda img_prompt: f"<bos><start_of_turn>user\n{img_prompt}<end_of_turn>\n<start_of_turn>model\n", # noqa: E501

181 - single_image_prompts=IMAGE_ASSETS.prompts(

182 - {

183 - "stop_sign": "<start_of_image>What's the content in the center of the image?", # noqa: E501

184 - "cherry_blossom": "<start_of_image>What is the season?",

185 - }

186 - ),

187 - multi_image_prompt="<start_of_image><start_of_image>Describe the two images in detail.", # noqa: E501

188 - max_model_len=8192,

198 + test_type=VLMTestType.IMAGE,

199 + prompt_formatter=lambda vid_prompt: f"<'<bos><start_of_turn>user\n{vid_prompt}<start_of_image><end_of_turn>\n<start_of_turn>model\n", # noqa: E501

200 + max_model_len=4096,

201 auto_cls=AutoModelForImageTextToText,

190 - # TODO: Support `do_pan_and_scan` in transformers backend

191 - # patch_hf_runner=model_utils.gemma3_patch_hf_runner,

202 vllm_output_post_proc=model_utils.gemma3_vllm_to_hf_output,

203 image_size_factors=[(0.25, 0.5, 1.0)],

204 vllm_runner_kwargs={

205 "model_impl": "transformers",

196 - # "mm_processor_kwargs": {"do_pan_and_scan": True},

206 },

207 marks=[pytest.mark.core_model],

208 ),

@@ -212,27 +221,6 @@ VLM_TEST_SETTINGS = {

221 },

222 marks=[pytest.mark.core_model],

223 ),

215 - # PaliGemma has PrefixLM attention

216 - "paligemma-transformers": VLMTestInfo(

217 - models=["google/paligemma-3b-mix-224"],

218 - test_type=VLMTestType.IMAGE,

219 - prompt_formatter=identity,

220 - img_idx_to_prompt=lambda idx: "",

221 - # PaliGemma uses its own sample prompts because the default one fails

222 - single_image_prompts=IMAGE_ASSETS.prompts(

223 - {

224 - "stop_sign": "caption es",

225 - "cherry_blossom": "What is in the picture?",

226 - }

227 - ),

228 - auto_cls=AutoModelForImageTextToText,

229 - vllm_output_post_proc=model_utils.paligemma_vllm_to_hf_output,

230 - image_size_factors=[(0.25, 0.5, 1.0)],

231 - vllm_runner_kwargs={

232 - "model_impl": "transformers",

233 - },

234 - marks=[pytest.mark.core_model],

235 - ),

224 # Pixel values from processor are not 4D or 5D arrays

225 "qwen2_5_vl-transformers": VLMTestInfo(

226 models=["Qwen/Qwen2.5-VL-3B-Instruct"],

@@ -359,6 +347,24 @@ VLM_TEST_SETTINGS = {

347 image_size_factors=[(), (0.25,), (0.25, 0.25, 0.25), (0.25, 0.2, 0.15)],

348 marks=[large_gpu_mark(min_gb=32)],

349 ),

350 + "gemma3": VLMTestInfo(

351 + models=["google/gemma-3-4b-it"],

352 + test_type=(VLMTestType.IMAGE, VLMTestType.MULTI_IMAGE),

353 + prompt_formatter=lambda img_prompt: f"<bos><start_of_turn>user\n{img_prompt}<end_of_turn>\n<start_of_turn>model\n", # noqa: E501

354 + single_image_prompts=IMAGE_ASSETS.prompts(

355 + {

356 + "stop_sign": "<start_of_image>What's the content in the center of the image?", # noqa: E501

357 + "cherry_blossom": "<start_of_image>What is the season?",

358 + }

359 + ),

360 + multi_image_prompt="<start_of_image><start_of_image>Describe the two images in detail.", # noqa: E501

361 + max_model_len=4096,

362 + max_num_seqs=2,

363 + auto_cls=AutoModelForImageTextToText,

364 + vllm_runner_kwargs={"mm_processor_kwargs": {"do_pan_and_scan": True}},

365 + patch_hf_runner=model_utils.gemma3_patch_hf_runner,

366 + num_logprobs=10,

367 + ),

368 "glm4v": VLMTestInfo(

369 models=["zai-org/glm-4v-9b"],

370 test_type=VLMTestType.IMAGE,

tests/models/multimodal/generation/vlm_utils/model_utils.py

+10

@@ -328,6 +328,16 @@ def gemma3_patch_hf_runner(hf_model: HfRunner) -> HfRunner:

328

329 hf_model.processor = processor

330

331 + orig_generate = hf_model.model.generate

332 +

333 + def _generate(self, *args, **kwargs):

334 + # FIXME: https://github.com/huggingface/transformers/issues/38333

335 + kwargs["disable_compile"] = True

336 +

337 + return orig_generate(*args, **kwargs)

338 +

339 + hf_model.model.generate = types.MethodType(_generate, hf_model.model)

340 +

341 return hf_model

342

343

tests/models/multimodal/processing/test_common.py

@@ -222,6 +222,7 @@ def _test_processing_correctness(

222 _ADD_SPECIAL_TOKENS_OVERRIDES = {

223 "ovis": False,

224 "ovis2_5": False,

225 + "paligemma": False,

226 "ultravox": False,

227 "whisper": False,

228 }

@@ -333,6 +334,7 @@ def _test_processing_correctness_one(

334 "deepseek-ai/deepseek-vl2-tiny",

335 "baidu/ERNIE-4.5-VL-28B-A3B-PT",

336 "adept/fuyu-8b",

337 + "google/gemma-3-4b-it",

338 "google/gemma-3n-E2B-it",

339 "zai-org/glm-4v-9b",

340 "zai-org/GLM-4.1V-9B-Thinking",

@@ -369,6 +371,8 @@ def _test_processing_correctness_one(

371 "AIDC-AI/Ovis1.6-Llama3.2-3B",

372 "AIDC-AI/Ovis2-1B",

373 "AIDC-AI/Ovis2.5-2B",

374 + "google/paligemma-3b-mix-224",

375 + "google/paligemma2-3b-ft-docci-448",

376 "microsoft/Phi-3.5-vision-instruct",

377 "microsoft/Phi-4-multimodal-instruct",

378 "mistralai/Pixtral-12B-2409",

tests/models/multimodal/processing/test_tensor_schema.py

@@ -48,6 +48,7 @@ ARCH_NEEDS_EXTRAS = [

48 "Idefics3ForConditionalGeneration",

49 "LlavaForConditionalGeneration",

50 "MiniCPMV",

51 + "PaliGemmaForConditionalGeneration",

52 ]

53 REPO_ID_TO_SKIP = {

54 "nm-testing/pixtral-12b-FP8-dynamic": "duplicated test",

vllm/model_executor/models/gemma3_mm.py new

+710

@@ -0,0 +1,710 @@

1 +# SPDX-License-Identifier: Apache-2.0

2 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project

3 +import math

4 +from collections.abc import Iterable, Mapping, Sequence

5 +from typing import Annotated, Any, Literal

6 +

7 +import torch

8 +from torch import nn

9 +from transformers import BatchFeature, Gemma3Config, Gemma3Processor

10 +from transformers.models.gemma3.processing_gemma3 import Gemma3ProcessorKwargs

11 +

12 +import vllm.envs as envs

13 +from vllm.config import VllmConfig

14 +from vllm.config.multimodal import BaseDummyOptions

15 +from vllm.logger import init_logger

16 +from vllm.model_executor.layers.layernorm import GemmaRMSNorm

17 +from vllm.model_executor.models.module_mapping import MultiModelKeys

18 +from vllm.multimodal import MULTIMODAL_REGISTRY

19 +from vllm.multimodal.inputs import (

20 + MultiModalDataDict,

21 + MultiModalFieldConfig,

22 + MultiModalKwargsItems,

23 +)

24 +from vllm.multimodal.parse import ImageProcessorItems, ImageSize, MultiModalDataItems

25 +from vllm.multimodal.processing import (

26 + BaseMultiModalProcessor,

27 + BaseProcessingInfo,

28 + MultiModalPromptUpdates,

29 + MultiModalPromptUpdatesApplyResult,

30 + PlaceholderFeaturesInfo,

31 + PromptReplacement,

32 + PromptUpdate,

33 + PromptUpdateDetails,

34 + replace_token_matches,

35 +)

36 +from vllm.multimodal.profiling import BaseDummyInputsBuilder

37 +from vllm.sequence import IntermediateTensors

38 +from vllm.utils.tensor_schema import TensorSchema, TensorShape

39 +

40 +from .interfaces import (

41 + MultiModalEmbeddings,

42 + SupportsLoRA,

43 + SupportsMultiModal,

44 + SupportsPP,

45 +)

46 +from .siglip import SiglipVisionModel

47 +from .utils import (

48 + AutoWeightsLoader,

49 + WeightsMapper,

50 + init_vllm_registered_model,

51 + maybe_prefix,

52 +)

53 +

54 +logger = init_logger(__name__)

55 +

56 +

57 +class Gemma3ImagePixelInputs(TensorSchema):

58 + """

59 + Dimensions:

60 + - p: Number of patches total (over each image over each prompt in the

61 + batch)

62 + - c: Number of channels (3)

63 + - h: Height of each patch

64 + - w: Width of each patch

65 + - bn: Batch size * number of images

66 + """

67 +

68 + type: Literal["pixel_values"] = "pixel_values"

69 +

70 + pixel_values: Annotated[torch.Tensor, TensorShape("p", 3, "h", "w")]

71 +

72 + num_patches: Annotated[torch.Tensor, TensorShape("bn")]

73 +

74 +

75 +Gemma3ImageInputs = Gemma3ImagePixelInputs

76 +

77 +

78 +class Gemma3ProcessingInfo(BaseProcessingInfo):

79 + def get_hf_config(self):

80 + return self.ctx.get_hf_config(Gemma3Config)

81 +

82 + def get_hf_processor(self, **kwargs: object):

83 + return self.ctx.get_hf_processor(Gemma3Processor, **kwargs)

84 +

85 + def get_supported_mm_limits(self) -> Mapping[str, int | None]:

86 + return {"image": None}

87 +

88 + def _resolve_image_kwargs(

89 + self,

90 + processor: Gemma3Processor,

91 + keys: set[str],

92 + ) -> dict[str, Any]:

93 + image_processor = processor.image_processor

94 + kwargs = processor._merge_kwargs(

95 + Gemma3ProcessorKwargs,

96 + tokenizer_init_kwargs=processor.tokenizer.init_kwargs,

97 + )

98 +

99 + images_kwargs = kwargs["images_kwargs"]

100 +

101 + def _resolve_kw(key: str):

102 + val = getattr(image_processor, key)

103 + if val is None:

104 + val = images_kwargs[key]

105 +

106 + return val

107 +

108 + return {k: _resolve_kw(k) for k in keys}

109 +

110 + def get_num_crops(

111 + self,

112 + *,

113 + image_width: int,

114 + image_height: int,

115 + processor: Gemma3Processor | None,

116 + ) -> int:

117 + if processor is None:

118 + processor = self.get_hf_processor()

119 +

120 + images_kwargs = self._resolve_image_kwargs(

121 + processor,

122 + {

123 + "do_pan_and_scan",

124 + "pan_and_scan_min_crop_size",

125 + "pan_and_scan_max_num_crops",

126 + "pan_and_scan_min_ratio_to_activate",

127 + },

128 + )

129 +

130 + do_pan_and_scan = images_kwargs["do_pan_and_scan"]

131 + pan_and_scan_min_crop_size = images_kwargs["pan_and_scan_min_crop_size"]

132 + pan_and_scan_max_num_crops = images_kwargs["pan_and_scan_max_num_crops"]

133 + pan_and_scan_min_ratio_to_activate = images_kwargs[

134 + "pan_and_scan_min_ratio_to_activate"

135 + ]

136 +

137 + if not do_pan_and_scan:

138 + return 0

139 +

140 + if envs.VLLM_USE_V1:

141 + logger.warning_once(

142 + "`do_pan_and_scan=True` has suboptimal results on V1 "

143 + "because of the simplified attention pattern being used."

144 + )

145 +

146 + # Based on Gemma3ImageProcessor.pan_and_scan

147 + if image_width >= image_height:

148 + if image_width / image_height < pan_and_scan_min_ratio_to_activate:

149 + return 0

150 +

151 + num_crops_w = min(

152 + int(math.floor(image_width / pan_and_scan_min_crop_size)),

153 + int(math.floor(image_width / image_height + 0.5)),

154 + )

155 +

156 + num_crops_w = max(2, num_crops_w)

157 + num_crops_w = min(pan_and_scan_max_num_crops, num_crops_w)

158 + num_crops_h = 1

159 + else:

160 + if image_height / image_width < pan_and_scan_min_ratio_to_activate:

161 + return 0

162 +

163 + num_crops_h = min(

164 + int(math.floor(image_height / pan_and_scan_min_crop_size)),

165 + int(math.floor(image_height / image_width + 0.5)),

166 + )

167 +

168 + num_crops_h = max(2, num_crops_h)

169 + num_crops_h = min(pan_and_scan_max_num_crops, num_crops_h)

170 + num_crops_w = 1

171 +

172 + crop_size_w = int(math.ceil(image_width / num_crops_w))

173 + crop_size_h = int(math.ceil(image_height / num_crops_h))

174 +

175 + if min(crop_size_w, crop_size_h) < pan_and_scan_min_crop_size:

176 + return 0

177 +

178 + return num_crops_w * num_crops_h

179 +

180 + def get_image_repl(

181 + self,

182 + *,

183 + image_width: int,

184 + image_height: int,

185 + processor: Gemma3Processor | None,

186 + ) -> PromptUpdateDetails[str]:

187 + if processor is None:

188 + processor = self.get_hf_processor()

189 +

190 + boi_token = processor.boi_token

191 +

192 + num_crops = self.get_num_crops(

193 + image_width=image_width,

194 + image_height=image_height,

195 + processor=processor,

196 + )

197 +

198 + if num_crops == 0:

199 + image_text = boi_token

200 + else:

201 + crops_image_tokens = " ".join(boi_token for _ in range(num_crops))

202 + image_text = (

203 + f"Here is the original image {boi_token} and here are some "

204 + f"crops to help you see better {crops_image_tokens}"

205 + )

206 +

207 + repl_full = image_text.replace(boi_token, processor.full_image_sequence)

208 +

209 + tokenizer = processor.tokenizer

210 + vocab = tokenizer.get_vocab()

211 + image_token_id = vocab[tokenizer.image_token]

212 +

213 + return PromptUpdateDetails.select_token_id(repl_full, image_token_id)

214 +

215 + def get_num_image_tokens(

216 + self,

217 + *,

218 + image_width: int,

219 + image_height: int,

220 + processor: Gemma3Processor | None,

221 + ) -> int:

222 + if processor is None:

223 + processor = self.get_hf_processor()

224 +

225 + num_crops = self.get_num_crops(

226 + image_width=image_width,

227 + image_height=image_height,

228 + processor=processor,

229 + )

230 + image_seq_len = processor.image_seq_length

231 +

232 + return (num_crops + 1) * image_seq_len

233 +

234 + def get_image_size_with_most_features(self) -> ImageSize:

235 + processor = self.get_hf_processor()

236 +

237 + images_kwargs = self._resolve_image_kwargs(

238 + processor, {"pan_and_scan_max_num_crops"}

239 + )

240 + max_num_crops = images_kwargs["pan_and_scan_max_num_crops"]

241 +

242 + # Result in the max possible feature size (h:w = max_num_crops:1)

243 + return ImageSize(height=50 * max_num_crops, width=50)

244 +

245 +

246 +class Gemma3DummyInputsBuilder(BaseDummyInputsBuilder[Gemma3ProcessingInfo]):

247 + def get_dummy_text(self, mm_counts: Mapping[str, int]) -> str:

248 + num_images = mm_counts.get("image", 0)

249 +

250 + processor = self.info.get_hf_processor()

251 + image_token = processor.boi_token

252 +

253 + return image_token * num_images

254 +

255 + def get_dummy_mm_data(

256 + self,

257 + seq_len: int,

258 + mm_counts: Mapping[str, int],

259 + mm_options: Mapping[str, BaseDummyOptions] | None = None,

260 + ) -> MultiModalDataDict:

261 + num_images = mm_counts.get("image", 0)

262 +

263 + target_width, target_height = self.info.get_image_size_with_most_features()

264 +

265 + image_overrides = mm_options.get("image") if mm_options else None

266 +

267 + return {

268 + "image": self._get_dummy_images(

269 + width=target_width,

270 + height=target_height,

271 + num_images=num_images,

272 + overrides=image_overrides,

273 + )

274 + }

275 +

276 +

277 +class Gemma3MultiModalProcessor(BaseMultiModalProcessor[Gemma3ProcessingInfo]):

278 + def _call_hf_processor(

279 + self,

280 + prompt: str,

281 + mm_data: Mapping[str, object],

282 + mm_kwargs: Mapping[str, object],

283 + tok_kwargs: Mapping[str, object],

284 + ) -> BatchFeature:

285 + processed_outputs = super()._call_hf_processor(

286 + prompt,

287 + mm_data,

288 + mm_kwargs,

289 + tok_kwargs,

290 + )

291 +

292 + # HF processor pops the `num_crops` kwarg, which is needed by vLLM

293 + if (images := mm_data.get("images")) is not None:

294 + parsed_images = (

295 + self._get_data_parser()

296 + .parse_mm_data({"image": images})

297 + .get_items("image", ImageProcessorItems)

298 + )

299 + image_sizes = [

300 + parsed_images.get_image_size(i) for i in range(len(parsed_images))

301 + ]

302 + hf_processor = self.info.get_hf_processor(**mm_kwargs)

303 +

304 + num_crops = [

305 + self.info.get_num_crops(

306 + image_width=size.width,

307 + image_height=size.height,

308 + processor=hf_processor,

309 + )

310 + for size in image_sizes

311 + ]

312 + processed_outputs["num_patches"] = torch.tensor(num_crops) + 1

313 +

314 + return processed_outputs

315 +

316 + def _get_mm_fields_config(

317 + self,

318 + hf_inputs: BatchFeature,

319 + hf_processor_mm_kwargs: Mapping[str, object],

320 + ) -> Mapping[str, MultiModalFieldConfig]:

321 + num_patches = hf_inputs.get("num_patches", torch.empty(0))

322 +

323 + return dict(

324 + pixel_values=MultiModalFieldConfig.flat_from_sizes("image", num_patches),

325 + num_patches=MultiModalFieldConfig.batched("image"),

326 + )

327 +

328 + def _get_prompt_updates(

329 + self,

330 + mm_items: MultiModalDataItems,

331 + hf_processor_mm_kwargs: Mapping[str, Any],

332 + out_mm_kwargs: MultiModalKwargsItems,

333 + ) -> Sequence[PromptUpdate]:

334 + hf_processor = self.info.get_hf_processor(**hf_processor_mm_kwargs)

335 + image_token = hf_processor.boi_token

336 +

337 + def get_replacement_gemma3(item_idx: int):

338 + images = mm_items.get_items("image", ImageProcessorItems)

339 +

340 + image_size = images.get_image_size(item_idx)

341 + return self.info.get_image_repl(

342 + image_width=image_size.width,

343 + image_height=image_size.height,

344 + processor=hf_processor,

345 + )

346 +

347 + return [

348 + PromptReplacement(

349 + modality="image",

350 + target=image_token,

351 + replacement=get_replacement_gemma3,

352 + )

353 + ]

354 +

355 + def _apply_token_matches(

356 + self,

357 + prompt: list[int],

358 + mm_prompt_updates: MultiModalPromptUpdates,

359 + ) -> tuple[list[int], MultiModalPromptUpdatesApplyResult]:

360 + token_ids, res = super()._apply_token_matches(prompt, mm_prompt_updates)

361 +

362 + # "\n\n\n" and "\n\n\n\n" are single tokens

363 + # Since our replacement can insert "\n\n" next to "\n"

364 + # tokens, we have to combine them to be consistent with

365 + # the output of the tokenizer

366 + tokenizer = self.info.get_tokenizer()

367 + vocab = tokenizer.get_vocab()

368 + newline_1 = vocab["\n"]

369 + newline_2 = vocab["\n\n"]

370 + newline_3 = vocab["\n\n\n"]

371 + newline_4 = vocab["\n\n\n\n"]

372 +

373 + token_ids = replace_token_matches(

374 + token_ids,

375 + [newline_1, newline_2],

376 + [newline_3],

377 + )

378 + token_ids = replace_token_matches(

379 + token_ids,

380 + [newline_2, newline_1],

381 + [newline_3],

382 + )

383 + token_ids = replace_token_matches(

384 + token_ids,

385 + [newline_2, newline_2],

386 + [newline_4],

387 + )

388 +

389 + return token_ids, res

390 +

391 + def _find_mm_placeholders(

392 + self,

393 + new_token_ids: list[int],

394 + mm_prompt_updates: MultiModalPromptUpdates,

395 + ) -> Mapping[str, list[PlaceholderFeaturesInfo]]:

396 + # We need to detect "\n\n" inside "\n\n\n" and "\n\n\n\n"

397 + tokenizer = self.info.get_tokenizer()

398 + vocab = tokenizer.get_vocab()

399 + newline_1 = vocab["\n"]

400 + newline_2 = vocab["\n\n"]

401 + newline_3 = vocab["\n\n\n"]

402 + newline_4 = vocab["\n\n\n\n"]

403 +

404 + def get_repl_toks(tok: int) -> list[int]:

405 + if tok == newline_3:

406 + return [newline_1, newline_2]

407 + if tok == newline_4:

408 + return [newline_2, newline_2]

409 +

410 + return [tok]

411 +

412 + repl_token_ids = list[int]()

413 + repl_orig_idxs = list[int]()

414 + for orig_idx, orig_tok in enumerate(new_token_ids):

415 + repl_toks = get_repl_toks(orig_tok)

416 + repl_token_ids.extend(repl_toks)

417 + repl_orig_idxs.extend(orig_idx for _ in range(len(repl_toks)))

418 +

419 + repls = super()._find_mm_placeholders(repl_token_ids, mm_prompt_updates)

420 +

421 + return {

422 + modality: [

423 + PlaceholderFeaturesInfo(

424 + modality=p.modality,

425 + item_idx=p.item_idx,

426 + start_idx=repl_orig_idxs[p.start_idx],

427 + tokens=p.tokens,

428 + is_embed=p.is_embed,

429 + )

430 + for p in placeholders

431 + ]

432 + for modality, placeholders in repls.items()

433 + }

434 +

435 +

436 +class Gemma3MultiModalProjector(nn.Module):

437 + def __init__(self, config: Gemma3Config):

438 + super().__init__()

439 +

440 + self.mm_input_projection_weight = nn.Parameter(

441 + torch.zeros(

442 + config.vision_config.hidden_size, config.text_config.hidden_size

443 + )

444 + )

445 +

446 + self.mm_soft_emb_norm = GemmaRMSNorm(

447 + config.vision_config.hidden_size, eps=config.vision_config.layer_norm_eps

448 + )

449 +

450 + self.patches_per_image = int(

451 + config.vision_config.image_size // config.vision_config.patch_size

452 + )

453 + self.tokens_per_side = int(config.mm_tokens_per_image**0.5)

454 + self.kernel_size = self.patches_per_image // self.tokens_per_side

455 + self.avg_pool = nn.AvgPool2d(

456 + kernel_size=self.kernel_size, stride=self.kernel_size

457 + )

458 +

459 + def forward(self, vision_outputs: torch.Tensor):

460 + batch_size, _, seq_length = vision_outputs.shape

461 +

462 + reshaped_vision_outputs = vision_outputs.transpose(1, 2)

463 + reshaped_vision_outputs = reshaped_vision_outputs.reshape(

464 + batch_size, seq_length, self.patches_per_image, self.patches_per_image

465 + )

466 + reshaped_vision_outputs = reshaped_vision_outputs.contiguous()

467 +

468 + pooled_vision_outputs = self.avg_pool(reshaped_vision_outputs)

469 + pooled_vision_outputs = pooled_vision_outputs.flatten(2)

470 + pooled_vision_outputs = pooled_vision_outputs.transpose(1, 2)

471 +

472 + normed_vision_outputs = self.mm_soft_emb_norm(pooled_vision_outputs)

473 +

474 + projected_vision_outputs = torch.matmul(

475 + normed_vision_outputs, self.mm_input_projection_weight

476 + )

477 + return projected_vision_outputs.type_as(vision_outputs)

478 +

479 +

480 +@MULTIMODAL_REGISTRY.register_processor(

481 + Gemma3MultiModalProcessor,

482 + info=Gemma3ProcessingInfo,

483 + dummy_inputs=Gemma3DummyInputsBuilder,

484 +)

485 +class Gemma3ForConditionalGeneration(

486 + nn.Module, SupportsMultiModal, SupportsPP, SupportsLoRA

487 +):

488 + merge_by_field_config = True

489 +

490 + packed_modules_mapping = {

491 + "qkv_proj": [

492 + "q_proj",

493 + "k_proj",

494 + "v_proj",

495 + ],

496 + "gate_up_proj": [

497 + "gate_proj",

498 + "up_proj",

499 + ],

500 + }

501 +

502 + hf_to_vllm_mapper = WeightsMapper(

503 + orig_to_new_prefix={

504 + # mapping for new names in checkpoint saved after transformers v4.52

505 + "model.language_model.": "language_model.model.",

506 + "model.vision_tower.": "vision_tower.",

507 + "model.multi_modal_projector.": "multi_modal_projector.",

508 + "lm_head.": "language_model.lm_head.",

509 + }

510 + )

511 +

512 + @classmethod

513 + def get_placeholder_str(cls, modality: str, i: int) -> str | None:

514 + if modality.startswith("image"):

515 + return "<start_of_image>"

516 +

517 + raise ValueError("Only image modality is supported")

518 +

519 + def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):

520 + super().__init__()

521 + config = vllm_config.model_config.hf_config

522 + quant_config = vllm_config.quant_config

523 + multimodal_config = vllm_config.model_config.multimodal_config

524 + self.config = config

525 + self.quant_config = quant_config

526 + self.multimodal_config = multimodal_config

527 +

528 + self.vision_tower = SiglipVisionModel(

529 + config.vision_config,

530 + quant_config,

531 + prefix=maybe_prefix(prefix, "vision_tower"),

532 + )

533 + self.multi_modal_projector = Gemma3MultiModalProjector(config)

534 +

535 + self.language_model = init_vllm_registered_model(

536 + vllm_config=vllm_config,

537 + hf_config=config.text_config,

538 + prefix=maybe_prefix(prefix, "language_model"),

539 + architectures=["Gemma3ForCausalLM"],

540 + )

541 + logit_scale = getattr(config, "logit_scale", 1.0)

542 +

543 + if hasattr(self.language_model, "logits_processor"):

544 + # The logits processor can be unset if we're using

545 + # automatic conversion to pooling model.

546 + self.language_model.logits_processor.scale *= logit_scale

547 +

548 + self.make_empty_intermediate_tensors = (

549 + self.language_model.make_empty_intermediate_tensors

550 + )

551 +

552 + @property

553 + def dtype(self):

554 + return next(self.parameters()).dtype

555 +

556 + def _parse_and_validate_image_input(

557 + self, **kwargs: object

558 + ) -> Gemma3ImageInputs | None:

559 + pixel_values = kwargs.pop("pixel_values", None)

560 + num_patches = kwargs.pop("num_patches", None)

561 + image_embeds = kwargs.pop("image_embeds", None)

562 + assert image_embeds is None, "Gemma3 does not support image_embeds."

563 + if pixel_values is None:

564 + return None

565 +

566 + image_size = self.config.vision_config.image_size

567 +

568 + return Gemma3ImagePixelInputs(

569 + pixel_values=pixel_values,

570 + num_patches=num_patches,

571 + resolve_bindings={"h": image_size, "w": image_size},

572 + )

573 +

574 + def _image_pixels_to_features(

575 + self,

576 + vision_tower: SiglipVisionModel,

577 + pixel_values: torch.Tensor,

578 + ) -> torch.Tensor:

579 + return vision_tower(pixel_values)

580 +

581 + def _process_image_input(

582 + self,

583 + image_input: Gemma3ImageInputs,

584 + ) -> list[torch.Tensor]:

585 + assert self.vision_tower is not None

586 +

587 + pixel_values = image_input["pixel_values"]

588 + num_patches = image_input["num_patches"]

589 +

590 + image_features = self._image_pixels_to_features(

591 + self.vision_tower,

592 + pixel_values,

593 + )

594 + image_embeds = self.multi_modal_projector(image_features)

595 +

596 + return [e.flatten(0, 1) for e in image_embeds.split(num_patches.tolist())]

597 +

598 + def get_language_model(self) -> torch.nn.Module:

599 + return self.language_model

600 +

601 + def get_multimodal_embeddings(self, **kwargs: object) -> MultiModalEmbeddings:

602 + image_input = self._parse_and_validate_image_input(**kwargs)

603 + if image_input is None:

604 + return []

605 +

606 + return self._process_image_input(image_input)

607 +

608 + def forward(

609 + self,

610 + input_ids: torch.Tensor,

611 + positions: torch.Tensor,

612 + intermediate_tensors: IntermediateTensors | None = None,

613 + inputs_embeds: torch.Tensor | None = None,

614 + **kwargs: object,

615 + ) -> IntermediateTensors:

616 + if intermediate_tensors is not None:

617 + inputs_embeds = None

618 +

619 + hidden_states = self.language_model.model(

620 + input_ids,

621 + positions,

622 + intermediate_tensors,

623 + inputs_embeds=inputs_embeds,

624 + **kwargs,

625 + )

626 +

627 + return hidden_states

628 +

629 + def prepare_attn_masks(

630 + self,

631 + input_ids: torch.Tensor,

632 + positions: torch.Tensor,

633 + mask_dtype: torch.dtype,

634 + **kwargs,

635 + ):

636 + kwargs["has_images"] = True

637 + # NOTE(woosuk): Here, we distinguish the sequences by the position id 0.

638 + # This is a HACK. Fix this.

639 + start_indices = (positions == 0).cpu().nonzero()

640 + num_seqs = len(start_indices)

641 + seq_lens = []

642 + for i in range(num_seqs):

643 + start_idx = start_indices[i].item()

644 + if i < num_seqs - 1:

645 + end_idx = start_indices[i + 1].item()

646 + else:

647 + end_idx = len(input_ids)

648 + seq_lens.append(end_idx - start_idx)

649 + kwargs["seq_lens"] = seq_lens

650 +

651 + global_attn_masks = []

652 + local_attn_masks = []

653 + start_idx = 0

654 + for seq_len in seq_lens:

655 + end_idx = start_idx + seq_len

656 + input_token_ids = input_ids[start_idx:end_idx]

657 + start_idx = end_idx

658 + # Create a global causal mask.

659 + global_attn_mask = torch.empty(

660 + 1,

661 + 1,

662 + seq_len,

663 + seq_len,

664 + dtype=mask_dtype,

665 + device=input_ids.device,

666 + )

667 + global_attn_mask.fill_(float("-inf"))

668 + # Fill the lower triangle with 0.

669 + global_attn_mask = global_attn_mask.triu(diagonal=1)

670 +

671 + # Consider the bidirectional attention between image tokens.

672 + img_mask = torch.zeros_like(global_attn_mask)

673 + img_pos = input_token_ids == self.config.image_token_index

674 + img_mask[:, :, :, img_pos] += 1

675 + img_mask[:, :, img_pos, :] += 1

676 + global_attn_mask = torch.where(img_mask == 2, 0, global_attn_mask)

677 + global_attn_masks.append(global_attn_mask)

678 +

679 + sliding_window = self.config.text_config.sliding_window

680 + if sliding_window is not None:

681 + # Create a local causal mask with sliding window (1024).

682 + local_attn_mask = torch.ones_like(global_attn_mask)

683 + local_attn_mask = torch.tril(local_attn_mask, diagonal=-sliding_window)

684 + local_attn_mask = torch.where(

685 + local_attn_mask == 0, global_attn_mask, float("-inf")

686 + )

687 + local_attn_masks.append(local_attn_mask)

688 + kwargs["global_attn_masks"] = global_attn_masks

689 + kwargs["local_attn_masks"] = local_attn_masks

690 + return kwargs

691 +

692 + def compute_logits(

693 + self,

694 + hidden_states: torch.Tensor,

695 + ) -> torch.Tensor | None:

696 + return self.language_model.compute_logits(hidden_states)

697 +

698 + def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:

699 + loader = AutoWeightsLoader(self)

700 + return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)

701 +

702 + def get_mm_mapping(self) -> MultiModelKeys:

703 + """

704 + Get the module prefix in multimodal models

705 + """

706 + return MultiModelKeys.from_string_field(

707 + language_model="language_model",

708 + connector="multi_modal_projector",

709 + tower_model="vision_tower",

710 + )

vllm/model_executor/models/paligemma.py new

+412

@@ -0,0 +1,412 @@

1 +# SPDX-License-Identifier: Apache-2.0

2 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project

3 +from collections.abc import Iterable, Mapping, Sequence

4 +from typing import Annotated, Literal, TypeAlias

5 +

6 +import torch

7 +from torch import nn

8 +from transformers import BatchFeature, PaliGemmaConfig

9 +

10 +from vllm.config import VllmConfig

11 +from vllm.config.multimodal import BaseDummyOptions

12 +from vllm.logger import init_logger

13 +from vllm.multimodal import MULTIMODAL_REGISTRY

14 +from vllm.multimodal.inputs import (

15 + MultiModalDataDict,

16 + MultiModalFieldConfig,

17 + MultiModalInputs,

18 + MultiModalKwargsItems,

19 + MultiModalUUIDDict,

20 +)

21 +from vllm.multimodal.parse import (

22 + ImageEmbeddingItems,

23 + ImageProcessorItems,

24 + MultiModalDataItems,

25 +)

26 +from vllm.multimodal.processing import (

27 + BaseMultiModalProcessor,

28 + BaseProcessingInfo,

29 + PromptIndexTargets,

30 + PromptInsertion,

31 + PromptUpdate,

32 + PromptUpdateDetails,

33 +)

34 +from vllm.multimodal.profiling import BaseDummyInputsBuilder

35 +from vllm.sequence import IntermediateTensors

36 +from vllm.utils.tensor_schema import TensorSchema, TensorShape

37 +

38 +from .interfaces import MultiModalEmbeddings, SupportsMultiModal, SupportsPP

39 +from .siglip import SiglipVisionModel

40 +from .utils import (

41 + AutoWeightsLoader,

42 + WeightsMapper,

43 + flatten_bn,

44 + init_vllm_registered_model,

45 + maybe_prefix,

46 +)

47 +from .vision import get_vision_encoder_info

48 +

49 +logger = init_logger(__name__)

50 +

51 +

52 +class PaliGemmaImagePixelInputs(TensorSchema):

53 + """

54 + Dimensions:

55 + - bn: Batch size * number of images

56 + - c: Number of channels (3)

57 + - h: Height

58 + - w: Width

59 + """

60 +

61 + type: Literal["pixel_values"] = "pixel_values"

62 + data: Annotated[torch.Tensor, TensorShape("bn", 3, "h", "w")]

63 +

64 +

65 +class PaliGemmaImageEmbeddingInputs(TensorSchema):

66 + """

67 + Dimensions:

68 + - bn: Batch size * number of images

69 + - ifs: Image feature size

70 + - hs: Hidden size (must match language model backbone)

71 + """

72 +

73 + type: Literal["image_embeds"] = "image_embeds"

74 + data: Annotated[torch.Tensor, TensorShape("bn", "ifs", "hs")]

75 +

76 +

77 +PaliGemmaImageInputs: TypeAlias = (

78 + PaliGemmaImagePixelInputs | PaliGemmaImageEmbeddingInputs

79 +)

80 +

81 +

82 +class PaliGemmaMultiModalProjector(nn.Module):

83 + def __init__(self, vision_hidden_size: int, projection_dim: int):

84 + super().__init__()

85 +

86 + self.linear = nn.Linear(vision_hidden_size, projection_dim, bias=True)

87 +

88 + def forward(self, image_features: torch.Tensor) -> torch.Tensor:

89 + hidden_states = self.linear(image_features)

90 + return hidden_states

91 +

92 +

93 +class PaliGemmaProcessingInfo(BaseProcessingInfo):

94 + def get_hf_config(self):

95 + return self.ctx.get_hf_config(PaliGemmaConfig)

96 +

97 + def get_vision_encoder_info(self):

98 + return get_vision_encoder_info(self.get_hf_config())

99 +

100 + def get_supported_mm_limits(self) -> Mapping[str, int | None]:

101 + return {"image": 1}

102 +

103 + def get_num_image_tokens(

104 + self,

105 + *,

106 + image_width: int,

107 + image_height: int,

108 + ) -> int:

109 + vision_encoder_info = self.get_vision_encoder_info()

110 +

111 + return vision_encoder_info.get_num_image_tokens(

112 + image_width=image_width,

113 + image_height=image_height,

114 + )

115 +

116 +

117 +class PaliGemmaDummyInputsBuilder(BaseDummyInputsBuilder[PaliGemmaProcessingInfo]):

118 + def get_dummy_text(self, mm_counts: Mapping[str, int]) -> str:

119 + return ""

120 +

121 + def get_dummy_mm_data(

122 + self,

123 + seq_len: int,

124 + mm_counts: Mapping[str, int],

125 + mm_options: Mapping[str, BaseDummyOptions] | None = None,

126 + ) -> MultiModalDataDict:

127 + hf_config = self.info.get_hf_config()

128 + vision_config = hf_config.vision_config

129 + max_image_size = vision_config.image_size

130 +

131 + num_images = mm_counts.get("image", 0)

132 +

133 + image_overrides = mm_options.get("image") if mm_options else None

134 +

135 + return {

136 + "image": self._get_dummy_images(

137 + width=max_image_size,

138 + height=max_image_size,

139 + num_images=num_images,

140 + overrides=image_overrides,

141 + )

142 + }

143 +

144 +

145 +class PaliGemmaMultiModalProcessor(BaseMultiModalProcessor[PaliGemmaProcessingInfo]):

146 + def _call_hf_processor(

147 + self,

148 + prompt: str,

149 + mm_data: Mapping[str, object],

150 + mm_kwargs: Mapping[str, object],

151 + tok_kwargs: Mapping[str, object],

152 + ) -> BatchFeature:

153 + tokenizer = self.info.get_tokenizer()

154 + if not mm_data:

155 + prompt_ids = tokenizer.encode(prompt, add_special_tokens=False)

156 + return BatchFeature(dict(input_ids=[prompt_ids]), tensor_type="pt")

157 +

158 + return super()._call_hf_processor(

159 + prompt=prompt,

160 + mm_data=mm_data,

161 + mm_kwargs=mm_kwargs,

162 + tok_kwargs=tok_kwargs,

163 + )

164 +

165 + def _get_mm_fields_config(

166 + self,

167 + hf_inputs: BatchFeature,

168 + hf_processor_mm_kwargs: Mapping[str, object],

169 + ) -> Mapping[str, MultiModalFieldConfig]:

170 + return dict(pixel_values=MultiModalFieldConfig.batched("image"))

171 +

172 + def _get_prompt_updates(

173 + self,

174 + mm_items: MultiModalDataItems,

175 + hf_processor_mm_kwargs: Mapping[str, object],

176 + out_mm_kwargs: MultiModalKwargsItems,

177 + ) -> Sequence[PromptUpdate]:

178 + hf_config = self.info.get_hf_config()

179 + image_token_id = hf_config.image_token_index

180 +

181 + tokenizer = self.info.get_tokenizer()

182 +

183 + bos_token_id = tokenizer.bos_token_id

184 + assert isinstance(bos_token_id, int)

185 +

186 + def get_insertion(item_idx: int):

187 + images = mm_items.get_items(

188 + "image", (ImageEmbeddingItems, ImageProcessorItems)

189 + )

190 +

191 + if isinstance(images, ImageEmbeddingItems):

192 + num_image_tokens = images.get_feature_size(item_idx)

193 + else:

194 + image_size = images.get_image_size(item_idx)

195 + num_image_tokens = self.info.get_num_image_tokens(

196 + image_width=image_size.width,

197 + image_height=image_size.height,

198 + )

199 +

200 + image_tokens = [image_token_id] * num_image_tokens

201 +

202 + return PromptUpdateDetails.select_token_id(

203 + image_tokens + [bos_token_id],

204 + embed_token_id=image_token_id,

205 + )

206 +

207 + # Paligemma 1 and 2 have different tokenizer.add_bos_token

208 + # Insert <image>*n + <bos> after <bos> for Paligemma 1

209 + # Insert <image>*n + <bos> for Paligemma 2

210 + return [

211 + PromptInsertion(

212 + modality="image",

213 + target=PromptIndexTargets.prefix(

214 + [bos_token_id] if tokenizer.add_bos_token else []

215 + ),

216 + insertion=get_insertion,

217 + )

218 + ]

219 +

220 + def apply(

221 + self,

222 + prompt: str | list[int],

223 + mm_data: MultiModalDataDict,

224 + hf_processor_mm_kwargs: Mapping[str, object],

225 + tokenization_kwargs: Mapping[str, object] | None = None,

226 + mm_uuids: MultiModalUUIDDict | None = None,

227 + ) -> MultiModalInputs:

228 + mm_inputs = super().apply(

229 + prompt,

230 + mm_data,

231 + hf_processor_mm_kwargs,

232 + tokenization_kwargs,

233 + mm_uuids=mm_uuids,

234 + )

235 + prompt_token_ids = mm_inputs["prompt_token_ids"]

236 +

237 + tokenizer = self.info.get_tokenizer()

238 + newline_prompt = "\n"

239 + newline_token_id = tokenizer.encode(newline_prompt)[-1] # 108

240 + # Force to add newline at the end of prompt for paligemma's format

241 + # This step can NOT be replacemented by current PromptUpdate methods

242 + if len(prompt_token_ids) and prompt_token_ids[-1] != newline_token_id:

243 + prompt_token_ids.append(newline_token_id)

244 + mm_inputs["prompt_token_ids"] = prompt_token_ids

245 +

246 + return mm_inputs

247 +

248 +

249 +@MULTIMODAL_REGISTRY.register_processor(

250 + PaliGemmaMultiModalProcessor,

251 + info=PaliGemmaProcessingInfo,

252 + dummy_inputs=PaliGemmaDummyInputsBuilder,

253 +)

254 +class PaliGemmaForConditionalGeneration(nn.Module, SupportsMultiModal, SupportsPP):

255 + packed_modules_mapping = {

256 + "qkv_proj": [

257 + "q_proj",

258 + "k_proj",

259 + "v_proj",

260 + ],

261 + "gate_up_proj": [

262 + "gate_proj",

263 + "up_proj",

264 + ],

265 + }

266 +

267 + hf_to_vllm_mapper = WeightsMapper(

268 + orig_to_new_prefix={

269 + # mapping for new names in checkpoint saved after transformers v4.52

270 + "model.language_model.": "language_model.model.",

271 + "model.vision_tower.": "vision_tower.",

272 + "model.multi_modal_projector.": "multi_modal_projector.",

273 + "lm_head.": "language_model.lm_head.",

274 + }

275 + )

276 +

277 + @classmethod

278 + def get_placeholder_str(cls, modality: str, i: int) -> str | None:

279 + if modality.startswith("image"):

280 + return None

281 +

282 + raise ValueError("Only image modality is supported")

283 +

284 + def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):

285 + super().__init__()

286 + config = vllm_config.model_config.hf_config

287 + quant_config = vllm_config.quant_config

288 + multimodal_config = vllm_config.model_config.multimodal_config

289 + self.config = config

290 + self.multimodal_config = multimodal_config

291 +

292 + self.vision_tower = SiglipVisionModel(

293 + config.vision_config,

294 + quant_config,

295 + prefix=maybe_prefix(prefix, "vision_tower"),

296 + )

297 + self.multi_modal_projector = PaliGemmaMultiModalProjector(

298 + vision_hidden_size=config.vision_config.hidden_size,

299 + projection_dim=config.vision_config.projection_dim,

300 + )

301 +

302 + self.quant_config = quant_config

303 +

304 + if config.text_config.model_type == "gemma":

305 + config.text_config.architectures = ["GemmaForCausalLM"]

306 + else:

307 + config.text_config.architectures = ["Gemma2ForCausalLM"]

308 + self.language_model = init_vllm_registered_model(

309 + vllm_config=vllm_config,

310 + hf_config=config.text_config,

311 + prefix=maybe_prefix(prefix, "language_model"),

312 + )

313 + logit_scale = getattr(config, "logit_scale", 1.0)

314 + self.language_model.logits_processor.scale *= logit_scale

315 +

316 + self.make_empty_intermediate_tensors = (

317 + self.language_model.make_empty_intermediate_tensors

318 + )

319 +

320 + def _parse_and_validate_image_input(

321 + self, **kwargs: object

322 + ) -> PaliGemmaImageInputs | None:

323 + pixel_values = kwargs.pop("pixel_values", None)

324 + image_embeds = kwargs.pop("image_embeds", None)

325 +

326 + if pixel_values is None and image_embeds is None:

327 + return None

328 +

329 + if pixel_values is not None:

330 + pixel_values = flatten_bn(pixel_values, concat=True)

331 +

332 + h = w = self.config.vision_config.image_size

333 + return PaliGemmaImagePixelInputs(

334 + type="pixel_values",

335 + data=pixel_values,

336 + resolve_bindings={"h": h, "w": w},

337 + )

338 +

339 + if image_embeds is not None:

340 + image_embeds = flatten_bn(image_embeds, concat=True)

341 +

342 + return PaliGemmaImageEmbeddingInputs(

343 + type="image_embeds",

344 + data=image_embeds,

345 + )

346 +

347 + raise AssertionError("This line should be unreachable.")

348 +

349 + def _image_pixels_to_features(

350 + self,

351 + vision_tower: SiglipVisionModel,

352 + pixel_values: torch.Tensor,

353 + ) -> torch.Tensor:

354 + target_dtype = vision_tower.get_input_embeddings().weight.dtype

355 + image_features = vision_tower(pixel_values.to(dtype=target_dtype))

356 +

357 + return image_features

358 +

359 + def _process_image_input(

360 + self,

361 + image_input: PaliGemmaImageInputs,

362 + ) -> torch.Tensor:

363 + if image_input["type"] == "image_embeds":

364 + return image_input["data"]

365 +

366 + assert self.vision_tower is not None

367 + pixel_values = image_input["data"]

368 + image_features = self._image_pixels_to_features(

369 + self.vision_tower,

370 + pixel_values,

371 + )

372 +

373 + return self.multi_modal_projector(image_features)

374 +

375 + def get_language_model(self) -> torch.nn.Module:

376 + return self.language_model

377 +

378 + def get_multimodal_embeddings(self, **kwargs: object) -> MultiModalEmbeddings:

379 + image_input = self._parse_and_validate_image_input(**kwargs)

380 + if image_input is None:

381 + return []

382 + vision_embeddings = self._process_image_input(image_input)

383 + # https://github.com/huggingface/transformers/blob/main/src/transformers/models/paligemma/modeling_paligemma.py#L294 # noqa

384 + vision_embeddings = vision_embeddings * (self.config.hidden_size**-0.5)

385 + return vision_embeddings

386 +

387 + def forward(

388 + self,

389 + input_ids: torch.Tensor,

390 + positions: torch.Tensor,

391 + intermediate_tensors: IntermediateTensors | None = None,

392 + inputs_embeds: torch.Tensor | None = None,

393 + **kwargs: object,

394 + ) -> IntermediateTensors:

395 + if intermediate_tensors is not None:

396 + inputs_embeds = None

397 +

398 + hidden_states = self.language_model.model(

399 + input_ids, positions, intermediate_tensors, inputs_embeds=inputs_embeds

400 + )

401 +

402 + return hidden_states

403 +

404 + def compute_logits(

405 + self,

406 + hidden_states: torch.Tensor,

407 + ) -> torch.Tensor | None:

408 + return self.language_model.compute_logits(hidden_states)

409 +

410 + def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:

411 + loader = AutoWeightsLoader(self)

412 + return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)

vllm/model_executor/models/registry.py

+5 -8

@@ -265,6 +265,7 @@ _MULTIMODAL_MODELS = {

265 "Ernie4_5_VLMoeForConditionalGeneration",

266 ),

267 "FuyuForCausalLM": ("fuyu", "FuyuForCausalLM"),

268 + "Gemma3ForConditionalGeneration": ("gemma3_mm", "Gemma3ForConditionalGeneration"), # noqa: E501

269 "Gemma3nForConditionalGeneration": (

270 "gemma3n_mm",

271 "Gemma3nForConditionalGeneration",

@@ -334,6 +335,10 @@ _MULTIMODAL_MODELS = {

335 "NVLM_D": ("nvlm_d", "NVLM_D_Model"),

336 "Ovis": ("ovis", "Ovis"),

337 "Ovis2_5": ("ovis2_5", "Ovis2_5"),

338 + "PaliGemmaForConditionalGeneration": (

339 + "paligemma",

340 + "PaliGemmaForConditionalGeneration",

341 + ),

342 "Phi3VForCausalLM": ("phi3v", "Phi3VForCausalLM"),

343 "Phi4MMForCausalLM": ("phi4mm", "Phi4MMForCausalLM"),

344 "Phi4MultimodalForCausalLM": ("phi4_multimodal", "Phi4MultimodalForCausalLM"), # noqa: E501

@@ -406,14 +411,6 @@ _TRANSFORMERS_SUPPORTED_MODELS = {

411 "transformers",

412 "TransformersMultiModalForCausalLM",

413 ),

409 - "Gemma3ForConditionalGeneration": (

410 - "transformers",

411 - "TransformersMultiModalForCausalLM",

412 - ),

413 - "PaliGemmaForConditionalGeneration": (

414 - "transformers",

415 - "TransformersMultiModalForCausalLM",

416 - ),

414 }

415

416 _TRANSFORMERS_BACKEND_MODELS = {

vllm/platforms/rocm.py

@@ -59,6 +59,9 @@ _ROCM_PARTIALLY_SUPPORTED_MODELS: dict[str, str] = {

59 "Qwen2ForCausalLM": _ROCM_SWA_REASON,

60 "MistralForCausalLM": _ROCM_SWA_REASON,

61 "MixtralForCausalLM": _ROCM_SWA_REASON,

62 + "PaliGemmaForConditionalGeneration": (

63 + "ROCm flash attention does not yet fully support 32-bit precision on PaliGemma"

64 + ),

65 "Phi3VForCausalLM": (

66 "ROCm Triton flash attention may run into compilation errors due to "

67 "excessive use of shared memory. If this happens, disable Triton FA "