@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 9499e26e2a

Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 15, 01:00 UTC

[Model] Support VLMs with transformers backend (#20543)

Signed-off-by: raushan <raushan@huggingface.co> Signed-off-by: Isotr0py <2037008807@qq.com> Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: Isotr0py <2037008807@qq.com> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

Raushan Turganbay committed Jul 20, 2025 at 15:25 UTC 9499e26e2ae18826bcda99ae7e0883268cde03db

7 files changed +625 -87

docs/models/supported_models.md

+6 -3

@@ -18,7 +18,7 @@ These models are what we list in [supported-text-models][supported-text-models]

19 ### Transformers

21 -vLLM also supports model implementations that are available in Transformers. This does not currently work for all models, but most decoder language models are supported, and vision language model support is planned!

21 +vLLM also supports model implementations that are available in Transformers. This does not currently work for all models, but most decoder language models and common vision language models are supported! Vision-language models currently accept only image inputs, and require setting `--disable_mm_preprocessor_cache` when running. Support for video inputs and caching of multi-modal preprocessors will be added in future releases.

23 To check if the modeling backend is Transformers, you can simply do this:

@@ -28,7 +28,7 @@ llm = LLM(model=..., task="generate") # Name or path of your model

28 llm.apply_model(lambda model: print(type(model)))

29 ```

31 -If it is `TransformersForCausalLM` then it means it's based on Transformers!

31 +If it is `TransformersForCausalLM` or `TransformersForMultimodalLM` then it means it's based on Transformers!

33 !!! tip

34 You can force the use of `TransformersForCausalLM` by setting `model_impl="transformers"` for [offline-inference](../serving/offline_inference.md) or `--model-impl transformers` for the [openai-compatible-server](../serving/openai_compatible_server.md).

@@ -36,6 +36,9 @@ If it is `TransformersForCausalLM` then it means it's based on Transformers!

36 !!! note

37 vLLM may not fully optimise the Transformers implementation so you may see degraded performance if comparing a native model to a Transformers model in vLLM.

39 +!!! note

40 + In case of vision language models if you are loading with `dtype="auto"`, vLLM loads the whole model with config's `dtype` if it exists. In contrast the native Transformers will respect the `dtype` attribute of each backbone in the model. That might cause a slight difference in performance.

41 +

42 #### Custom models

44 If a model is neither supported natively by vLLM or Transformers, it can still be used in vLLM!

@@ -99,7 +102,7 @@ Here is what happens in the background when this model is loaded:

102

103 1. The config is loaded.

104 2. `MyModel` Python class is loaded from the `auto_map` in config, and we check that the model `is_backend_compatible()`.

102 -3. `MyModel` is loaded into `TransformersForCausalLM` (see <gh-file:vllm/model_executor/models/transformers.py>) which sets `self.config._attn_implementation = "vllm"` so that vLLM's attention layer is used.

105 +3. `MyModel` is loaded into `TransformersForCausalLM` or `TransformersForMultimodalLM` (see <gh-file:vllm/model_executor/models/transformers.py>) which sets `self.config._attn_implementation = "vllm"` so that vLLM's attention layer is used.

106

107 That's it!

108

tests/models/multimodal/generation/test_common.py

+75

@@ -35,6 +35,8 @@ if current_platform.is_rocm():

35 REQUIRES_V0_MODELS = [

36 # V1 Test: not enough KV cache space in C1.

37 "fuyu",

38 + # V1 Test: Deadlock issue when processing mm_inputs

39 + "llava-onevision-transformers",

40 ]

42 # yapf: disable

@@ -170,6 +172,79 @@ VLM_TEST_SETTINGS = {

172 hf_output_post_proc=model_utils.ultravox_trunc_hf_output,

173 marks=[pytest.mark.core_model, pytest.mark.cpu_model],

174 ),

175 + #### Transformers fallback to test

176 + ## To reduce test burden, we only test batching arbitrary image size

177 + # Dynamic image length and number of patches

178 + "llava-onevision-transformers": VLMTestInfo(

179 + models=["llava-hf/llava-onevision-qwen2-0.5b-ov-hf"],

180 + test_type=VLMTestType.IMAGE,

181 + prompt_formatter=lambda vid_prompt: f"<|im_start|>user\n{vid_prompt}<|im_end|>\n<|im_start|>assistant\n", # noqa: E501

182 + max_model_len=16384,

183 + hf_model_kwargs=model_utils.llava_onevision_hf_model_kwargs("llava-hf/llava-onevision-qwen2-0.5b-ov-hf"), # noqa: E501

184 + auto_cls=AutoModelForImageTextToText,

185 + vllm_output_post_proc=model_utils.llava_onevision_vllm_to_hf_output,

186 + image_size_factors=[(0.25, 0.5, 1.0)],

187 + vllm_runner_kwargs={

188 + "model_impl": "transformers",

189 + "disable_mm_preprocessor_cache": True,

190 + "enable_prefix_caching": False,

191 + },

192 + marks=[pytest.mark.core_model],

193 + ),

194 + # FIXME(Isotr0py): Enable this test after

195 + # https://github.com/huggingface/transformers/pull/39470 released

196 + # "idefics3-transformers": VLMTestInfo(

197 + # models=["HuggingFaceTB/SmolVLM-256M-Instruct"],

198 + # test_type=(VLMTestType.IMAGE, VLMTestType.MULTI_IMAGE),

199 + # prompt_formatter=lambda img_prompt:f"<|begin_of_text|>User:{img_prompt}<end_of_utterance>\nAssistant:", # noqa: E501

200 + # img_idx_to_prompt=lambda idx: "<image>",

201 + # max_model_len=8192,

202 + # max_num_seqs=2,

203 + # auto_cls=AutoModelForImageTextToText,

204 + # hf_output_post_proc=model_utils.idefics3_trunc_hf_output,

205 + # image_size_factors=[(0.25, 0.5, 1.0)],

206 + # vllm_runner_kwargs={

207 + # "model_impl": "transformers",

208 + # "disable_mm_preprocessor_cache": True,

209 + # "enable_prefix_caching": False,

210 + # },

211 + # marks=[pytest.mark.core_model],

212 + # ),

213 + # Pixel values from processor are not 4D or 5D arrays

214 + "qwen2_5_vl-transformers": VLMTestInfo(

215 + models=["Qwen/Qwen2.5-VL-3B-Instruct"],

216 + test_type=VLMTestType.IMAGE,

217 + prompt_formatter=lambda img_prompt: f"<|im_start|>User\n{img_prompt}<|im_end|>\n<|im_start|>assistant\n", # noqa: E501

219 + max_model_len=4096,

220 + max_num_seqs=2,

221 + auto_cls=AutoModelForImageTextToText,

222 + vllm_output_post_proc=model_utils.qwen2_vllm_to_hf_output,

223 + image_size_factors=[(0.25, 0.2, 0.15)],

224 + vllm_runner_kwargs={

225 + "model_impl": "transformers",

226 + "disable_mm_preprocessor_cache": True,

227 + "enable_prefix_caching": False,

228 + },

229 + marks=[large_gpu_mark(min_gb=32)],

230 + ),

231 + # Check "auto" with fallback to transformers

232 + "internvl-transformers": VLMTestInfo(

233 + models=["OpenGVLab/InternVL3-1B-hf"],

234 + test_type=(VLMTestType.IMAGE, VLMTestType.MULTI_IMAGE),

235 + prompt_formatter=lambda img_prompt: f"<|im_start|>User\n{img_prompt}<|im_end|>\n<|im_start|>Assistant\n", # noqa: E501

236 + img_idx_to_prompt=lambda idx: "<IMG_CONTEXT>",

237 + max_model_len=4096,

238 + use_tokenizer_eos=True,

239 + image_size_factors=[(0.25, 0.5, 1.0)],

240 + vllm_runner_kwargs={

241 + "model_impl": "auto",

242 + "disable_mm_preprocessor_cache": True,

243 + "enable_prefix_caching": False,

244 + },

245 + auto_cls=AutoModelForImageTextToText,

246 + marks=[pytest.mark.core_model],

247 + ),

248 #### Extended model tests

249 "aria": VLMTestInfo(

250 models=["rhymes-ai/Aria"],

tests/models/registry.py

@@ -499,6 +499,7 @@ _SPECULATIVE_DECODING_EXAMPLE_MODELS = {

499

500 _TRANSFORMERS_MODELS = {

501 "TransformersForCausalLM": _HfExamplesInfo("ArthurZ/Ilama-3.2-1B", trust_remote_code=True), # noqa: E501

502 + "TransformersForMultimodalLM": _HfExamplesInfo("OpenGVLab/InternVL3-1B-hf"),

503 }

504

505 _EXAMPLE_MODELS = {

vllm/config.py

+30 -9

@@ -562,6 +562,10 @@ class ModelConfig:

562

563 self.task = "embed"

564

565 + model_info, arch = self.registry.inspect_model_cls(self.architectures)

566 + self._model_info = model_info

567 + self._architecture = arch

568 +

569 all_supported_tasks = self._get_supported_tasks(self.task)

570 logger.debug("Tasks supported by runner type: %s", all_supported_tasks)

571 supported_runner_types = self._get_supported_runner_types(

@@ -587,10 +591,6 @@ class ModelConfig:

591 else:

592 self.truncation_side = "right"

593

590 - model_info, arch = self.registry.inspect_model_cls(self.architectures)

591 - self._model_info = model_info

592 - self._architecture = arch

593 -

594 self.pooler_config = self._init_pooler_config()

595

596 self.dtype = _get_and_verify_dtype(

@@ -674,6 +674,16 @@ class ModelConfig:

674 "max_model_len must be an integer after __post_init__.")

675 return self

676

677 + def _get_transformers_backend_cls(self) -> str:

678 + """Determine which Transformers backend class will be used if

679 + `model_impl` is set to `transformers` or `auto`."""

680 + if self.hf_config != self.hf_text_config:

681 + # If 'hf_text_config' is the same as 'hf_config'. If not, it is

682 + # probably a composite config, i.e. multimodal

683 + return "TransformersForMultimodalLM"

684 + else:

685 + return "TransformersForCausalLM"

686 +

687 @property

688 def registry(self):

689 return me_models.ModelRegistry

@@ -681,7 +691,19 @@ class ModelConfig:

691 @property

692 def architectures(self) -> list[str]:

693 # architectures in the model config.

684 - return getattr(self.hf_config, "architectures", [])

694 + architectures = getattr(self.hf_config, "architectures", [])

695 + # The registry assumes that it can always inspect the vLLM model class

696 + # for a given architecture. This assumption breaks down for the

697 + # Transformers backend, which may use a different class depending on

698 + # the model type. To work around this, we add the correct Transformers

699 + # backend class to the architectures list. We must do this here because

700 + # we need access to the `hf_config` to determine the backend class.

701 + transformers_backend_cls = self._get_transformers_backend_cls()

702 + if (self.model_impl != ModelImpl.VLLM.value

703 + and all(arch != transformers_backend_cls

704 + for arch in architectures)):

705 + architectures.append(transformers_backend_cls)

706 + return architectures

707

708 @property

709 def architecture(self) -> str:

@@ -827,10 +849,9 @@ class ModelConfig:

849 ("EmbeddingModel", "embed"),

850 ("RewardModel", "reward"),

851 ]

830 - _, arch = self.registry.inspect_model_cls(architectures)

852

853 for suffix, pref_task in suffix_to_preferred_task:

833 - if arch.endswith(suffix):

854 + if self.architecture.endswith(suffix):

855 return pref_task

856

857 return "embed"

@@ -944,10 +965,10 @@ class ModelConfig:

965 ("EmbeddingModel", "pooling"),

966 ("RewardModel", "pooling"),

967 ]

947 - _, arch = self.registry.inspect_model_cls(self.architectures)

968

969 for suffix, pref_runner in suffix_to_preferred_runner:

950 - if arch.endswith(suffix) and pref_runner in supported_runner_types:

970 + if self.architecture.endswith(

971 + suffix) and pref_runner in supported_runner_types:

972 return pref_runner

973

974 if "generate" in supported_runner_types:

vllm/model_executor/model_loader/utils.py

+26 -23

@@ -25,6 +25,7 @@ from vllm.model_executor.models.adapters import (as_embedding_model,

25 as_reward_model,

26 as_seq_cls_model)

27 from vllm.model_executor.models.interfaces import SupportsQuant

28 +from vllm.model_executor.models.registry import _TRANSFORMERS_MODELS

29 from vllm.utils import is_pin_memory_available

31 logger = init_logger(__name__)

@@ -169,9 +170,22 @@ def device_loading_context(module: torch.nn.Module,

170

171 def resolve_transformers_arch(model_config: ModelConfig,

172 architectures: list[str]):

173 + if model_config.model_impl == ModelImpl.VLLM:

174 + raise ValueError(

175 + "Attempting to resolve architecture from the Transformers library "

176 + "but the model implementation is set to vLLM. This should never "

177 + "happen.")

178 +

179 for i, arch in enumerate(architectures):

173 - if arch == "TransformersForCausalLM":

180 + if arch in _TRANSFORMERS_MODELS:

181 continue

182 +

183 + if model_config.model_impl == ModelImpl.AUTO:

184 + logger.warning(

185 + "%s has no vLLM implementation, falling back to Transformers "

186 + "implementation. Some features may not be supported and "

187 + "performance may not be optimal.", arch)

188 +

189 auto_map: dict[str, str] = getattr(model_config.hf_config, "auto_map",

190 None) or dict()

191 # Make sure that config class is always initialized before model class,

@@ -199,25 +213,13 @@ def resolve_transformers_arch(model_config: ModelConfig,

213 "not present in the model config's 'auto_map' (relevant "

214 "if the model is custom).")

215 model_module = auto_modules["AutoModel"]

202 - # TODO(Isotr0py): Further clean up these raises.

203 - # perhaps handled them in _ModelRegistry._raise_for_unsupported?

204 - if model_config.model_impl == ModelImpl.TRANSFORMERS:

205 - if not model_module.is_backend_compatible():

206 - raise ValueError(

207 - f"The Transformers implementation of {arch} is not "

208 - "compatible with vLLM.")

209 - architectures[i] = "TransformersForCausalLM"

210 - if model_config.model_impl == ModelImpl.AUTO:

211 - if not model_module.is_backend_compatible():

212 - raise ValueError(

213 - f"{arch} has no vLLM implementation and the Transformers "

214 - "implementation is not compatible with vLLM. Try setting "

215 - "VLLM_USE_V1=0.")

216 - logger.warning(

217 - "%s has no vLLM implementation, falling back to Transformers "

218 - "implementation. Some features may not be supported and "

219 - "performance may not be optimal.", arch)

220 - architectures[i] = "TransformersForCausalLM"

216 +

217 + if not model_module.is_backend_compatible():

218 + raise ValueError(

219 + f"The Transformers implementation of '{arch}' is not "

220 + "compatible with vLLM.")

221 +

222 + architectures[i] = model_config._get_transformers_backend_cls()

223 return architectures

224

225

@@ -237,8 +239,9 @@ def get_model_architecture(

239 ]

240

241 vllm_supported_archs = ModelRegistry.get_supported_archs()

240 - vllm_not_supported = not any(arch in vllm_supported_archs

241 - for arch in architectures)

242 + is_supported = lambda arch: (arch in vllm_supported_archs and arch not in

243 + _TRANSFORMERS_MODELS)

244 + vllm_not_supported = not any(is_supported(arch) for arch in architectures)

245

246 if vllm_not_supported:

247 # try automatic conversion in adapters.py

@@ -259,7 +262,7 @@ def get_model_architecture(

262 break

263

264 if (model_config.model_impl == ModelImpl.TRANSFORMERS or

262 - model_config.model_impl != ModelImpl.VLLM and vllm_not_supported):

265 + model_config.model_impl == ModelImpl.AUTO and vllm_not_supported):

266 architectures = resolve_transformers_arch(model_config, architectures)

267 logger.debug_once("Resolve transformers arch %s", str(architectures))

268 elif (model_config.quantization is not None

vllm/model_executor/models/registry.py

+9 -3

@@ -253,6 +253,7 @@ _SPECULATIVE_DECODING_MODELS = {

253 }

254

255 _TRANSFORMERS_MODELS = {

256 + "TransformersForMultimodalLM": ("transformers", "TransformersForMultimodalLM"), # noqa: E501

257 "TransformersForCausalLM": ("transformers", "TransformersForCausalLM"),

258 }

259 # yapf: enable

@@ -504,9 +505,14 @@ class _ModelRegistry:

505 if causal_lm_arch in self.models:

506 normalized_arch.append(arch)

507

507 - # make sure Transformers backend is put at the last as a fallback

508 - if len(normalized_arch) != len(architectures):

509 - normalized_arch.append("TransformersForCausalLM")

508 + # NOTE(Isotr0py): Be careful of architectures' order!

509 + # Make sure Transformers backend architecture is at the end of the

510 + # list, otherwise pooling models automatic conversion will fail!

511 + for arch in normalized_arch:

512 + if arch.startswith("TransformersFor"):

513 + normalized_arch.remove(arch)

514 + normalized_arch.append(arch)

515 +

516 return normalized_arch

517

518 def inspect_model_cls(

vllm/model_executor/models/transformers.py

+478 -49

@@ -15,8 +15,8 @@

15 # See the License for the specific language governing permissions and

16 # limitations under the License.

17 """Wrapper around `transformers` models"""

18 -from collections.abc import Iterable

19 -from contextlib import nullcontext

18 +from collections.abc import Iterable, Mapping

19 +from contextlib import contextmanager, nullcontext

20 from typing import Literal, Optional, Union

22 import regex as re

@@ -41,11 +41,21 @@ from vllm.model_executor.layers.vocab_parallel_embedding import (

41 ParallelLMHead, VocabParallelEmbedding)

42 from vllm.model_executor.model_loader.weight_utils import default_weight_loader

43 from vllm.model_executor.sampling_metadata import SamplingMetadata

44 +from vllm.multimodal import MULTIMODAL_REGISTRY, MultiModalKwargs

45 +from vllm.multimodal.inputs import (MultiModalDataDict, MultiModalFieldConfig,

46 + MultiModalInputs, PlaceholderRange)

47 +from vllm.multimodal.parse import ImageProcessorItems, MultiModalDataItems

48 +from vllm.multimodal.processing import (BaseMultiModalProcessor,

49 + BaseProcessingInfo)

50 +from vllm.multimodal.profiling import BaseDummyInputsBuilder

51 from vllm.sequence import IntermediateTensors

52 +from vllm.transformers_utils.processor import cached_get_processor

53 +from vllm.utils import is_list_of

46 -from .interfaces import SupportsLoRA, SupportsPP, SupportsQuant

55 +from .interfaces import (SupportsLoRA, SupportsMultiModal, SupportsPP,

56 + SupportsQuant)

57 from .utils import (AutoWeightsLoader, PPMissingLayer, WeightsMapper,

48 - is_pp_missing_parameter,

58 + flatten_bn, is_pp_missing_parameter,

59 make_empty_intermediate_tensors_factory, maybe_prefix)

61 logger = init_logger(__name__)

@@ -112,6 +122,269 @@ def replace_linear_class(

122 )

123

124

125 +# Copied from `accelerate`

126 +@contextmanager

127 +def init_on_device_without_buffers(device: torch.device):

128 + """

129 + A context manager under which models are initialized with all

130 + parameters on the specified device. However buffers are not

131 + initialized on specified device.

132 +

133 + Args:

134 + device (`torch.device`):

135 + Device to initialize all parameters on.

136 + """

137 +

138 + old_register_parameter = nn.Module.register_parameter

139 +

140 + def register_empty_parameter(module, name, param):

141 + old_register_parameter(module, name, param)

142 + if param is not None:

143 + param_cls = type(module._parameters[name])

144 + kwargs = module._parameters[name].__dict__

145 + kwargs["requires_grad"] = param.requires_grad

146 + module._parameters[name] = param_cls(

147 + module._parameters[name].to(device), **kwargs)

148 +

149 + tensor_constructors_to_patch = {}

150 +

151 + def patch_tensor_constructor(fn):

152 +

153 + def wrapper(*args, **kwargs):

154 + kwargs["device"] = device

155 + return fn(*args, **kwargs)

156 +

157 + return wrapper

158 +

159 + try:

160 + nn.Module.register_parameter = register_empty_parameter

161 + for torch_function_name in tensor_constructors_to_patch:

162 + setattr(

163 + torch, torch_function_name,

164 + patch_tensor_constructor(getattr(torch, torch_function_name)))

165 + yield

166 + finally:

167 + nn.Module.register_parameter = old_register_parameter

168 + for torch_function_name, old_torch_function in (

169 + tensor_constructors_to_patch.items()):

170 + setattr(torch, torch_function_name, old_torch_function)

171 +

172 +

173 +class MultiModalProcessingInfo(BaseProcessingInfo):

174 +

175 + def get_hf_config(self):

176 + return self.ctx.model_config.hf_config

177 +

178 + def get_supported_mm_limits(self):

179 + return {"image": None}

180 +

181 + def get_mm_max_tokens_per_item(self, seq_len, mm_counts):

182 + return {"image": self.get_max_image_tokens()}

183 +

184 + def get_max_image_tokens(self) -> int:

185 + width, height = self.get_max_image_size()

186 + processor = self.get_hf_processor()

187 + mm_processor_kwargs = self.ctx.model_config.mm_processor_kwargs or {}

188 + mm_tokens = processor._get_num_multimodal_tokens(

189 + image_sizes=([height, width], ), **mm_processor_kwargs)

190 + image_tokens = mm_tokens["num_image_tokens"][0]

191 + return image_tokens

192 +

193 + def get_hf_processor(self):

194 + processor = cached_get_processor(self.ctx.model_config.model)

195 + return processor

196 +

197 + def get_max_image_size(self):

198 + return 10_000, 10_000 # hardcode for arbitrary very large size

199 +

200 +

201 +class MultiModalDummyInputsBuilder(

202 + BaseDummyInputsBuilder[MultiModalProcessingInfo]):

203 +

204 + def get_dummy_text(self, mm_counts: Mapping[str, int]) -> str:

205 + num_images = mm_counts.get("image", 0)

206 +

207 + processor = self.info.get_hf_processor()

208 + if "gemma3" in processor.__class__.__name__.lower():

209 + image_token = processor.boi_token

210 + else:

211 + image_token = getattr(processor, "image_token", "")

212 + return image_token * num_images

213 +

214 + def get_dummy_mm_data(

215 + self,

216 + seq_len: int,

217 + mm_counts: Mapping[str, int],

218 + ) -> MultiModalDataDict:

219 + num_images = mm_counts.get("image", 0)

220 +

221 + target_width, target_height = self.info.get_max_image_size()

222 +

223 + return {

224 + "image":

225 + self._get_dummy_images(width=target_width,

226 + height=target_height,

227 + num_images=num_images),

228 + }

229 +

230 +

231 +class MultiModalProcessor(BaseMultiModalProcessor[MultiModalProcessingInfo]):

232 +

233 + def _get_prompt_updates(

234 + self,

235 + mm_items: MultiModalDataItems,

236 + hf_processor_mm_kwargs: Mapping[str, object],

237 + out_mm_kwargs: MultiModalKwargs,

238 + ):

239 + """

240 + Given the original multi-modal items for this modality

241 + and HF-processed data, output the updates to perform.

242 +

243 + The information returned by this method is used to update token inputs

244 + which bypass the HF processor. It is also used to update the output of

245 + HF processor if the HF process does not apply prompt updates to text

246 + inputs.

247 +

248 + Moreover, this information is critical to determine the token positions

249 + in order to construct :class:`~vllm-multimodal.input.PlaceholderRange`

250 + for each multi-modal item.

251 + """

252 + return None

253 +

254 + def _get_mm_fields_config(

255 + self,

256 + hf_inputs,

257 + hf_processor_mm_kwargs,

258 + num_image_patches: torch.Tensor = None,

259 + ):

260 + # HF Processors always return a mask but vLLM doesn't need it

261 + hf_inputs.pop("attention_mask", None)

262 + mm_fields = {

263 + key: MultiModalFieldConfig.flat_from_sizes("image",

264 + num_image_patches)

265 + for key in hf_inputs

266 + }

267 + mm_fields["image_embeds"] = MultiModalFieldConfig.flat_from_sizes(

268 + "image", num_image_patches)

269 + mm_fields["num_image_patches"] = MultiModalFieldConfig.batched("image")

270 + return mm_fields

271 +

272 + def _apply_hf_processor_text_mm(

273 + self,

274 + prompt_text: str,

275 + mm_items: MultiModalDataItems,

276 + hf_processor_mm_kwargs: Mapping[str, object],

277 + tokenization_kwargs: Mapping[str, object],

278 + ):

279 + """

280 + Apply the HF processor on the prompt text and multi-modal data

281 + together.

282 +

283 + In addition, return whether prompt replacements have been applied.

284 + """

285 + processor_data, passthrough_data = self._get_hf_mm_data(mm_items)

286 + processor_data["return_mm_token_type_ids"] = True

287 +

288 + processed_data = self._call_hf_processor(

289 + prompt=prompt_text,

290 + mm_data=processor_data,

291 + mm_kwargs=hf_processor_mm_kwargs,

292 + tok_kwargs=tokenization_kwargs,

293 + )

294 + processed_data.update(passthrough_data)

295 +

296 + prompt_ids, = processed_data.pop("input_ids").tolist()

297 + mm_token_type_ids = processed_data.pop(

298 + "mm_token_type_ids"

299 + ) if "mm_token_type_ids" in processed_data else processed_data.pop(

300 + "token_type_ids") # for gemma3 only

301 +

302 + return prompt_ids, processed_data, mm_token_type_ids

303 +

304 + def apply(

305 + self,

306 + prompt: Union[str, list[int]],

307 + mm_data: MultiModalDataDict,

308 + hf_processor_mm_kwargs: Mapping[str, object],

309 + tokenization_kwargs: Optional[Mapping[str, object]] = None,

310 + return_mm_hashes: bool = False,

311 + ) -> MultiModalInputs:

312 + """

313 + Process multi-modal inputs to be used in vLLM.

314 +

315 + Apply HF Processor on prompt text and multi-modal data together,

316 + outputting token IDs and processed tensors.

317 + """

318 + if return_mm_hashes:

319 + raise ValueError(

320 + "TransformersForMultimodalLM doesn't support mm hashing yet! "

321 + "Probably you didn't set `disable_mm_preprocessor_cache=True`")

322 +

323 + if tokenization_kwargs is None:

324 + tokenization_kwargs = {}

325 +

326 + mm_items = self._to_mm_items(mm_data)

327 + hf_processor = self.info.get_hf_processor(**hf_processor_mm_kwargs)

328 +

329 + (prompt_ids, processed_data,

330 + mm_token_type_ids) = self._apply_hf_processor_text_mm(

331 + prompt_text=prompt,

332 + mm_items=mm_items,

333 + hf_processor_mm_kwargs=hf_processor_mm_kwargs,

334 + tokenization_kwargs=tokenization_kwargs,

335 + )

336 +

337 + # HF processor will return `mm_token_type_ids` from which

338 + # we can infer mm_placeholders. Until then hardcode to make code run

339 + # Below tested on Llava. Prompts and `mm_token_type_ids` are always bs=1

340 + mm_positions = torch.where(mm_token_type_ids == 1)[1]

341 + images = mm_items.get_items("image", ImageProcessorItems)

342 + mm_processor_kwargs = (self.info.ctx.model_config.mm_processor_kwargs

343 + or {})

344 + image_sizes = []

345 + for item_idx in range(len(images)):

346 + image_size = images.get_image_size(item_idx)

347 + image_sizes.append((image_size.height, image_size.width))

348 +

349 + mm_tokens_per_modality = hf_processor._get_num_multimodal_tokens(

350 + image_sizes=image_sizes, **mm_processor_kwargs)

351 +

352 + mm_placeholders = {}

353 + split_sizes = mm_tokens_per_modality["num_image_tokens"]

354 + if split_sizes:

355 + chunked_mm_positions = torch.split(mm_positions, split_sizes)

356 + mm_tokens = torch.tensor(prompt_ids)[mm_token_type_ids[0].bool()]

357 + chunked_mm_tokens = torch.split(mm_tokens, split_sizes)

358 + ranges = [

359 + PlaceholderRange(

360 + offset=positions[0].item(),

361 + length=positions.shape[0],

362 + is_embed=(mm_tokens == hf_processor.image_token_id).bool())

363 + for positions, mm_tokens in zip(chunked_mm_positions,

364 + chunked_mm_tokens)

365 + ]

366 + mm_placeholders = {"image": ranges}

367 +

368 + num_image_patches = torch.tensor(

369 + mm_tokens_per_modality["num_image_patches"]

370 + ) if "num_image_patches" in mm_tokens_per_modality else None

371 + processed_data['num_image_patches'] = num_image_patches

372 + mm_kwargs = MultiModalKwargs.from_hf_inputs(

373 + processed_data,

374 + self._get_mm_fields_config(processed_data, hf_processor_mm_kwargs,

375 + num_image_patches),

376 + )

377 +

378 + return MultiModalInputs(

379 + type="multimodal",

380 + prompt=prompt,

381 + prompt_token_ids=prompt_ids,

382 + mm_kwargs=mm_kwargs,

383 + mm_hashes=None,

384 + mm_placeholders=mm_placeholders,

385 + )

386 +

387 +

388 class ConfigOverride:

389 """Context manager to temporarily override config attributes."""

390

@@ -153,6 +426,7 @@ class TransformersModel(nn.Module):

426 quant_config: QuantizationConfig = vllm_config.quant_config

427

428 self.config = config

429 + self.text_config = config.get_text_config()

430 self.cache_config = cache_config

431 self.device_config = device_config

432 self.model_config = model_config

@@ -173,14 +447,16 @@ class TransformersModel(nn.Module):

447 config_override = ConfigOverride(

448 config, sliding_window=config.interleaved_sliding_window)

449

176 - # Use meta device to delay allocating GPU tensors

177 - with torch.device("meta"), config_override:

450 + # Set correct attn and init on "meta" to delay allocating GPU tensors

451 + # TODO: @raushan, use the public `model.set_attn_implementation()`

452 + # method after v4.54.0 is released

453 + self.text_config._attn_implementation = "vllm"

454 + with init_on_device_without_buffers("meta"), config_override:

455 # FIXME(Isotr0py): We need to refactor this part in the future to

456 # avoid registering an extra model layer, otherwise we will need a

457 # weights mapper to rename weights.

458 self.model: PreTrainedModel = AutoModel.from_config(

459 config,

183 - attn_implementation="vllm",

460 torch_dtype=model_config.dtype,

461 trust_remote_code=model_config.trust_remote_code,

462 )

@@ -189,27 +465,25 @@ class TransformersModel(nn.Module):

465 self.tensor_parallel()

466

467 # Input embeddings

468 + text_config = config.get_text_config()

469 if not isinstance(self.model.get_input_embeddings(), PPMissingLayer):

470 self.model.set_input_embeddings(

471 VocabParallelEmbedding(

195 - config.vocab_size,

196 - config.hidden_size,

197 - org_num_embeddings=config.vocab_size,

472 + text_config.vocab_size,

473 + text_config.hidden_size,

474 + org_num_embeddings=text_config.vocab_size,

475 quant_config=quant_config,

476 ))

477

478 # Attention layers

479 self.attention_instances = self.create_attention_instances()

480

204 - # Initialize buffers (e.g. rotary embedding inverse frequency)

205 - self.init_buffers(self.model)

206 -

481 # Initialize any parameters that have not had their modules replaced

482 self.init_parameters(self.model)

483

484 self.make_empty_intermediate_tensors = (

485 make_empty_intermediate_tensors_factory(["hidden_states"],

212 - config.hidden_size))

486 + text_config.hidden_size))

487

488 def pipeline_parallel(self):

489 """

@@ -240,14 +514,15 @@ class TransformersModel(nn.Module):

514

515 # Layers before module list

516 for name in pp_plan[:module_list_idx]:

243 - if self.pp_group.is_first_rank or (self.config.tie_word_embeddings

244 - and self.pp_group.is_last_rank):

517 + if self.pp_group.is_first_rank or (

518 + self.text_config.tie_word_embeddings

519 + and self.pp_group.is_last_rank):

520 continue

521 setattr(self.model, name, PPMissingLayer())

522

523 # Module list

249 - start_layer, end_layer = get_pp_indices(self.config.num_hidden_layers,

250 - self.pp_rank, self.pp_size)

524 + start_layer, end_layer = get_pp_indices(

525 + self.text_config.num_hidden_layers, self.pp_rank, self.pp_size)

526 layers_name = pp_plan[module_list_idx]

527 layers = getattr(self.model, layers_name)

528 for i in range(len(layers)):

@@ -298,7 +573,7 @@ class TransformersModel(nn.Module):

573 self.parallel_config)

574 head_size = self.model_config.get_head_size()

575 num_kv_heads = self.model_config.get_num_kv_heads(self.parallel_config)

301 - start, end = get_pp_indices(self.config.num_hidden_layers,

576 + start, end = get_pp_indices(self.text_config.num_hidden_layers,

577 self.pp_rank, self.pp_size)

578

579 attention_instances = {}

@@ -323,35 +598,6 @@ class TransformersModel(nn.Module):

598 prefix=f"{i}.attn")

599 return attention_instances

600

326 - def init_buffers(self, module: nn.Module):

327 - """

328 - If a `buffer` is on the `meta` device, then its parent

329 - `module` is the original module created by:

330 -

331 - ```python

332 - with torch.device("meta"):

333 - self.model: PreTrainedModel = AutoModel.from_config(...)

334 - ```

335 -

336 - This means that:

337 - - `type(module)` is a class from `transformers`

338 - - This class is constructed using a `PretrainedConfig`

339 - """

340 - for name, buffer in module.named_buffers(recurse=False):

341 - if buffer.device == torch.device("meta"):

342 - if module == self.model:

343 - logger.warning(

344 - "To initialize buffers correctly, we instantiate the "

345 - "parent module and and extract the value of the "

346 - "buffer from it. In this case, the parent module is "

347 - "the base model. Instantiating the entire model here "

348 - "risks GPU OOM. Could this buffer be moved to a child "

349 - "module?")

350 - new_buffer = getattr(type(module)(self.config), name)

351 - setattr(module, name, new_buffer)

352 - for child in module.children():

353 - self.init_buffers(child)

354 -

601 def init_parameters(self, module: nn.Module):

602 """

603 If a `parameter` is on the `meta` device, then its parent

@@ -366,6 +612,7 @@ class TransformersModel(nn.Module):

612 if param.device == torch.device("meta"):

613 new_param = nn.Parameter(

614 torch.empty_like(param.data,

615 + dtype=self.model_config.dtype,

616 device=self.device_config.device))

617 setattr(module, name, new_param)

618 for child in module.children():

@@ -391,11 +638,16 @@ class TransformersModel(nn.Module):

638 if inputs_embeds is not None:

639 inputs_embeds = inputs_embeds[None, ...]

640

641 + if self.model_config.uses_mrope:

642 + position_ids = positions[:, None]

643 + else:

644 + position_ids = positions[None, ...]

645 +

646 hidden_states = self.model(

647 input_ids=input_ids,

648 inputs_embeds=inputs_embeds,

649 use_cache=False,

398 - position_ids=positions[None, ...],

650 + position_ids=position_ids,

651 attention_instances=self.attention_instances,

652 return_dict=False)[0][0, ...] # we remove batch dimension for now

653

@@ -507,3 +759,180 @@ class TransformersForCausalLM(nn.Module, SupportsQuant, SupportsLoRA,

759 if self.config.tie_word_embeddings else None),

760 )

761 return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)

762 +

763 +

764 +@MULTIMODAL_REGISTRY.register_processor(

765 + MultiModalProcessor,

766 + info=MultiModalProcessingInfo,

767 + dummy_inputs=MultiModalDummyInputsBuilder)

768 +class TransformersForMultimodalLM(nn.Module, SupportsQuant, SupportsLoRA,

769 + SupportsPP, SupportsMultiModal):

770 + embedding_padding_modules = ["lm_head"]

771 + embedding_modules = ["embed_tokens"]

772 +

773 + def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):

774 + super().__init__()

775 + config: PretrainedConfig = vllm_config.model_config.hf_config

776 + quant_config: QuantizationConfig = vllm_config.quant_config

777 +

778 + self.config = config

779 + self.dtype = vllm_config.model_config.dtype

780 +

781 + self.model = TransformersModel(vllm_config=vllm_config, prefix=prefix)

782 + text_config = config.get_text_config()

783 +

784 + if get_pp_group().is_last_rank:

785 + self.unpadded_vocab_size = text_config.vocab_size

786 + self.lm_head = ParallelLMHead(

787 + text_config.vocab_size,

788 + text_config.hidden_size,

789 + quant_config=quant_config,

790 + prefix=maybe_prefix(prefix, "lm_head"),

791 + )

792 + if text_config.tie_word_embeddings:

793 + self.lm_head = self.lm_head.tie_weights(

794 + self.model.get_input_embeddings())

795 +

796 + logit_scale = getattr(config, "logit_scale", 1.0)

797 + self.logits_processor = LogitsProcessor(self.unpadded_vocab_size,

798 + text_config.vocab_size,

799 + logit_scale)

800 + else:

801 + self.lm_head = PPMissingLayer()

802 +

803 + self.make_empty_intermediate_tensors = (

804 + self.model.make_empty_intermediate_tensors)

805 +

806 + @property

807 + def hf_to_vllm_mapper(self):

808 + # Backwards compatibility for prev released models

809 + # State dicts back then had different formats

810 + # and cannot be loaded with `AutoModel` mapping

811 + # as is

812 + prefix_mapper = {

813 + "language_model.model": "model.language_model",

814 + "text_model.model": "model.text_model",

815 + "vision_tower": "model.vision_tower",

816 + "vqmodel": "model.vqmodel",

817 + "vision_model": "model.vision_model",

818 + "vision_embed_tokens": "model.vision_embed_tokens",

819 + "image_newline": "model.image_newline",

820 + "multi_modal_projector": "model.multi_modal_projector",

821 + "text_model.lm_head": "lm_head",

822 + "language_model.lm_head": "lm_head",

823 + }

824 + # Don't change the order for QwenVL

825 + if 'Qwen2' in self.config.__class__.__name__:

826 + prefix_mapper["model"] = "model.language_model"

827 + prefix_mapper["visual"] = "model.visual"

828 +

829 + return WeightsMapper(orig_to_new_prefix=prefix_mapper, )

830 +

831 + def forward(

832 + self,

833 + input_ids: Optional[torch.Tensor],

834 + positions: torch.Tensor,

835 + intermediate_tensors: Optional[IntermediateTensors] = None,

836 + inputs_embeds: Optional[torch.Tensor] = None,

837 + **kwargs: object,

838 + ) -> Union[torch.Tensor, IntermediateTensors]:

839 + # NOTE: In v1, inputs_embeds is always generated at model runner from

840 + # `get_multimodal_embeddings` and `get_input_embeddings`, this

841 + # condition is only for v0 compatibility.

842 + if inputs_embeds is None:

843 + multimodal_embeds = self.get_multimodal_embeddings(**kwargs)

844 + if multimodal_embeds is not None:

845 + inputs_embeds = self.get_input_embeddings(

846 + input_ids, multimodal_embeds)

847 + input_ids = None

848 +

849 + model_output = self.model(input_ids, positions, intermediate_tensors,

850 + inputs_embeds)

851 + return model_output

852 +

853 + def compute_logits(

854 + self,

855 + hidden_states: torch.Tensor,

856 + sampling_metadata: SamplingMetadata,

857 + ) -> Optional[torch.Tensor]:

858 + logits = self.logits_processor(self.lm_head, hidden_states,

859 + sampling_metadata)

860 + return logits

861 +

862 + def load_weights(self, weights: Iterable[tuple[str,

863 + torch.Tensor]]) -> set[str]:

864 + loader = AutoWeightsLoader(

865 + self,

866 + skip_prefixes=([

867 + "lm_head."

868 + ] if self.config.get_text_config().tie_word_embeddings else None),

869 + )

870 + return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)

871 +

872 + def get_multimodal_embeddings(self, **kwargs):

873 + pixel_values = kwargs.pop("pixel_values", None)

874 + pixel_values = pixel_values if pixel_values is not None else kwargs.pop(

875 + "image_patches", None)

876 + image_embeds = kwargs.pop("image_embeds", None)

877 +

878 + if image_embeds is not None:

879 + return image_embeds

880 +

881 + if pixel_values is None and image_embeds is None:

882 + return None

883 +

884 + num_image_patches = kwargs.pop("num_image_patches")

885 + if pixel_values is not None:

886 + if isinstance(pixel_values, torch.Tensor):

887 + pixel_values = flatten_bn(pixel_values).to(self.dtype)

888 + elif is_list_of(pixel_values, torch.Tensor):

889 + pixel_values = flatten_bn(flatten_bn(pixel_values),

890 + concat=True).to(self.dtype)

891 + else:

892 + raise ValueError(

893 + f"Unsupported pixel_values type {type(pixel_values)}. "

894 + "Expected `torch.Tensor` or list of `torch.Tensor`.")

895 +

896 + if isinstance(num_image_patches, list):

897 + num_image_patches = torch.cat(num_image_patches)

898 +

899 + vision_embeddings = self.model.model.get_image_features(

900 + pixel_values,

901 + **{

902 + k: v.flatten(0, 1)

903 + for k, v in kwargs.items()

904 + },

905 + )

906 +

907 + if isinstance(vision_embeddings, torch.Tensor):

908 + if vision_embeddings.ndim == 2:

909 + vision_embeddings = vision_embeddings.unsqueeze(0)

910 +

911 + # Embeddings have to be 2D tensors of length `num_images`

912 + # but transformers returns concat tensors if each patch

913 + # is of different size. We split it back to make vLLM happy

914 + vision_embeddings = torch.split(

915 + vision_embeddings,

916 + num_image_patches.flatten().tolist())

917 + vision_embeddings = [

918 + embed.flatten(start_dim=0, end_dim=-2)

919 + for embed in vision_embeddings

920 + ]

921 +

922 + return vision_embeddings

923 +

924 + def get_input_embeddings(

925 + self,

926 + input_ids: torch.Tensor,

927 + multimodal_embeddings=None,

928 + ) -> torch.Tensor:

929 + inputs_embeds = self.model.model.get_input_embeddings()(input_ids)

930 + if (multimodal_embeddings is not None

931 + and len(multimodal_embeddings) != 0):

932 + mask = (input_ids == self.config.image_token_id)

933 + mask = mask.unsqueeze(-1).expand_as(inputs_embeds)

934 + multimodal_embeddings = torch.cat(multimodal_embeddings)

935 +

936 + inputs_embeds = inputs_embeds.masked_scatter(

937 + mask, multimodal_embeddings)

938 + return inputs_embeds