@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 9831aec49f
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 22:00 UTC

[Core] Dynamic image size support for VLMs (#5276)

Signed-off-by: Xiaowei Jiang <xwjiang2010@gmail.com> Co-authored-by: Xiaowei Jiang <xwjiang2010@gmail.com> Co-authored-by: ywang96 <ywang@roblox.com> Co-authored-by: xwjiang2010 <87673679+xwjiang2010@users.noreply.github.com> Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>

Cyrus Leung committed Jul 3, 2024 at 11:34 UTC 9831aec49fd541a8fe6cb65f0c3a65b03eccffe0
38 files changed +1455 -666
docs/source/dev/input_processing/model_inputs_index.rst
+1 -1
@@ -8,7 +8,7 @@ Input Processing
8 vLLM provides a mechanism for defining input processors for each model so that the inputs are processed
9 in :class:`~vllm.LLMEngine` before they are passed to model executors.
10
11 -Currently, this mechanism is only utilized in **multi-modal models** for preprocessing multi-modal input
11 +Currently, this mechanism is only utilized in :ref:`multi-modal models <multi_modality>` for preprocessing multi-modal input
12 data in addition to input prompt, but it can be extended to text-only language models when needed.
13
14 Guides
docs/source/dev/multimodal/adding_multimodal_model.rst new
+124
@@ -0,0 +1,124 @@
1 +.. _adding_a_new_multimodal_model:
2 +
3 +Adding a New Multimodal Model
4 +=============================
5 +
6 +This document provides a high-level guide on integrating a :ref:`multi-modal model <multi_modality>` into vLLM.
7 +
8 +.. note::
9 + The complexity of adding a new model depends heavily on the model's architecture.
10 + The process is considerably straightforward if the model shares a similar architecture with an existing model in vLLM.
11 + However, for models that include new operators (e.g., a new attention mechanism), the process can be a bit more complex.
12 +
13 +.. tip::
14 + If you are encountering issues while integrating your model into vLLM, feel free to open an issue on our `GitHub <https://github.com/vllm-project/vllm/issues>`_ repository.
15 + We will be happy to help you out!
16 +
17 +
18 +1. Set up the base vLLM model
19 +-----------------------------
20 +
21 +As usual, follow :ref:`these steps <adding_a_new_model>` to implement the model in vLLM, but note the following:
22 +
23 +- You should additionally implement the :class:`~vllm.model_executor.models.interfaces.SupportsVision` interface.
24 +
25 + .. code-block:: diff
26 +
27 + + from vllm.model_executor.models.interfaces import SupportsVision
28 +
29 + - class YourModelForImage2Seq(nn.Module):
30 + + class YourModelForImage2Seq(nn.Module, SupportsVision):
31 +
32 + .. note::
33 + The model class does not have to be named :code:`*ForCausalLM`.
34 + Check out `the HuggingFace Transformers documentation <https://huggingface.co/docs/transformers/model_doc/auto#multimodal>`__ for some examples.
35 +
36 +- While implementing the :meth:`~torch.nn.Module.forward` method, reserve a keyword parameter
37 + for each input tensor that corresponds to a multi-modal input, as shown in the following example:
38 +
39 + .. code-block:: diff
40 +
41 + def forward(
42 + self,
43 + input_ids: torch.Tensor,
44 + positions: torch.Tensor,
45 + kv_caches: List[torch.Tensor],
46 + attn_metadata: AttentionMetadata,
47 + + pixel_values: torch.Tensor,
48 + ) -> SamplerOutput:
49 +
50 +
51 +2. Register input mappers
52 +-------------------------
53 +
54 +For each modality type to support, decorate the model class with :meth:`MULTIMODAL_REGISTRY.register_input_mapper <vllm.multimodal.MultiModalRegistry.register_input_mapper>`.
55 +This decorator accepts a function that maps multi-modal inputs to the keyword arguments you have previously defined in :meth:`~torch.nn.Module.forward`.
56 +
57 +.. code-block:: diff
58 +
59 + from vllm.model_executor.models.interfaces import SupportsVision
60 + + from vllm.multimodal import MULTIMODAL_REGISTRY
61 +
62 + + @MULTIMODAL_REGISTRY.register_image_feature_input_mapper()
63 + + @MULTIMODAL_REGISTRY.register_image_pixel_input_mapper()
64 + class YourModelForImage2Seq(nn.Module, SupportsVision):
65 +
66 +A default mapper is available for each modality in the core vLLM library. This input mapper will be used if you do not provide your own function.
67 +
68 +.. seealso::
69 + :ref:`input_processing_pipeline`
70 +
71 +
72 +3. (Optional) Register dummy data
73 +---------------------------------
74 +
75 +During startup, dummy data is passed to the vLLM model to allocate memory. This only consists of text input by default, which may not be applicable to multi-modal models.
76 +In such cases, you can define your own dummy data by registering a factory method via :meth:`INPUT_REGISTRY.register_dummy_data <vllm.inputs.registry.InputRegistry.register_dummy_data>`.
77 +
78 +.. code-block:: diff
79 +
80 + from vllm.inputs import INPUT_REGISTRY
81 + from vllm.model_executor.models.interfaces import SupportsVision
82 + from vllm.multimodal import MULTIMODAL_REGISTRY
83 +
84 + @MULTIMODAL_REGISTRY.register_image_feature_input_mapper()
85 + @MULTIMODAL_REGISTRY.register_image_pixel_input_mapper()
86 + + @INPUT_REGISTRY.register_dummy_data(<your_dummy_data_factory>)
87 + class YourModelForImage2Seq(nn.Module, SupportsVision):
88 +
89 +Here are some examples:
90 +
91 +- Image inputs (static feature size): `LLaVA-1.5 Model <https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/llava.py>`__
92 +- Image inputs (dynamic feature size): `LLaVA-NeXT Model <https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/llava_next.py>`__
93 +
94 +.. seealso::
95 + :ref:`input_processing_pipeline`
96 +
97 +
98 +4. (Optional) Register input processor
99 +--------------------------------------
100 +
101 +Sometimes, there is a need to process inputs at the :class:`~vllm.LLMEngine` level before they are passed to the model executor.
102 +This is often due to the fact that unlike implementations in HuggingFace Transformers, the reshaping and/or expansion of multi-modal embeddings needs to take place outside model's :meth:`~torch.nn.Module.forward` call.
103 +You can register input processors via :meth:`INPUT_REGISTRY.register_input_processor <vllm.inputs.registry.InputRegistry.register_input_processor>`.
104 +
105 +.. code-block:: diff
106 +
107 + from vllm.inputs import INPUT_REGISTRY
108 + from vllm.model_executor.models.interfaces import SupportsVision
109 + from vllm.multimodal import MULTIMODAL_REGISTRY
110 +
111 + @MULTIMODAL_REGISTRY.register_image_feature_input_mapper()
112 + @MULTIMODAL_REGISTRY.register_image_pixel_input_mapper()
113 + @INPUT_REGISTRY.register_dummy_data(<your_dummy_data_factory>)
114 + + @INPUT_REGISTRY.register_input_processor(<your_input_processor>)
115 + class YourModelForImage2Seq(nn.Module, SupportsVision):
116 +
117 +A common use case of input processors is inserting placeholder tokens to leverage the vLLM framework for attention mask generation.
118 +Here are some examples:
119 +
120 +- Insert static number of image tokens: `LLaVA-1.5 Model <https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/llava.py>`__
121 +- Insert dynamic number of image tokens: `LLaVA-NeXT Model <https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/llava_next.py>`__
122 +
123 +.. seealso::
124 + :ref:`input_processing_pipeline`
docs/source/dev/multimodal/multimodal_index.rst
+15 -3
@@ -1,3 +1,5 @@
1 +.. _multi_modality:
2 +
3 Multi-Modality
4 ==============
5
@@ -8,12 +10,18 @@ vLLM provides experimental support for multi-modal models through the :mod:`vllm
10 :class:`vllm.inputs.PromptStrictInputs` accepts an additional attribute ``multi_modal_data``
11 which allows you to pass in multi-modal input alongside text and token prompts.
12
11 -By default, vLLM models do not support multi-modal inputs. To enable multi-modal support for a model,
12 -you must decorate the model class with :meth:`InputRegistry.register_dummy_data <vllm.inputs.registry.InputRegistry.register_dummy_data>`,
13 -as well as :meth:`MULTIMODAL_REGISTRY.register_input_mapper <MultiModalRegistry.register_input_mapper>` for each modality type to support.
13 +By default, vLLM models do not support multi-modal inputs. To enable multi-modal support for a model, please follow :ref:`the guide for adding a new multimodal model. <adding_a_new_multimodal_model>`.
14
15 # TODO: Add more instructions on how to do that once embeddings is in.
16
17 +Guides
18 +++++++
19 +
20 +.. toctree::
21 + :maxdepth: 1
22 +
23 + adding_multimodal_model
24 +
25 Module Contents
26 +++++++++++++++
27
@@ -35,6 +43,10 @@ Base Classes
43 :members:
44 :show-inheritance:
45
46 +.. autoclass:: vllm.multimodal.MultiModalInputs
47 + :members:
48 + :show-inheritance:
49 +
50 .. autoclass:: vllm.multimodal.MultiModalPlugin
51 :members:
52 :show-inheritance:
docs/source/models/vlm.rst
+16 -8
@@ -23,7 +23,6 @@ The following :ref:`engine arguments <engine_args>` are specific to VLMs:
23 Currently, the support for vision language models on vLLM has the following limitations:
24
25 * Only single image input is supported per text prompt.
26 - * Dynamic ``image_input_shape`` is not supported: the input image will be resized to the static ``image_input_shape``. This means our LLaVA-NeXT output may not exactly match the huggingface implementation.
26
27 We are continuously improving user & developer experience for VLMs. Please `open an issue on GitHub <https://github.com/vllm-project/vllm/issues/new/choose>`_ if you have any feedback or feature requests.
28
@@ -42,12 +41,17 @@ To initialize a VLM, the aforementioned arguments must be passed to the ``LLM``
41 )
42
43 .. important::
44 + Currently, you have to specify ``image_feature_size`` to support memory profiling.
45 + To avoid OOM during runtime, you should set this to the maximum value supported by the model.
46 + The calculation of feature size is specific to the model. For more details, please refer to
47 + the function :code:`get_<model_name>_image_feature_size` inside the corresponding model file.
48 +
49 We will remove most of the vision-specific arguments in a future release as they can be inferred from the HuggingFace configuration.
50
51
52 To pass an image to the model, note the following in :class:`vllm.inputs.PromptStrictInputs`:
53
50 -* ``prompt``: The prompt should have a number of ``<image>`` tokens equal to ``image_feature_size``.
54 +* ``prompt``: The prompt should follow the format that is documented on HuggingFace.
55 * ``multi_modal_data``: This is a dictionary that follows the schema defined in :class:`vllm.multimodal.MultiModalDataDict`.
56
57 .. note::
@@ -57,8 +61,8 @@ To pass an image to the model, note the following in :class:`vllm.inputs.PromptS
61
62 .. code-block:: python
63
60 - prompt = "<image>" * 576 + (
61 - "\nUSER: What is the content of this image?\nASSISTANT:")
64 + # Refer to the HuggingFace repo for the correct format to use
65 + prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:"
66
67 # Load the image using PIL.Image
68 image = ...
@@ -74,8 +78,6 @@ To pass an image to the model, note the following in :class:`vllm.inputs.PromptS
78
79 A code example can be found in `examples/llava_example.py <https://github.com/vllm-project/vllm/blob/main/examples/llava_example.py>`_.
80
77 -.. important::
78 - We will remove the need to format image tokens in a future release. Afterwards, the input text will follow the same format as that for the original HuggingFace model.
81
82 Online OpenAI Vision API Compatible Inference
83 ----------------------------------------------
@@ -103,6 +105,11 @@ Below is an example on how to launch the same ``llava-hf/llava-1.5-7b-hf`` with
105 --chat-template template_llava.jinja
106
107 .. important::
108 + Currently, you have to specify ``image_feature_size`` to support memory profiling.
109 + To avoid OOM during runtime, you should set this to the maximum value supported by the model.
110 + The calculation of feature size is specific to the model. For more details, please refer to
111 + the function :code:`get_<model_name>_image_feature_size` inside the corresponding model file.
112 +
113 We will remove most of the vision-specific arguments in a future release as they can be inferred from the HuggingFace configuration.
114
115 To consume the server, you can use the OpenAI client like in the example below:
@@ -121,6 +128,8 @@ To consume the server, you can use the OpenAI client like in the example below:
128 messages=[{
129 "role": "user",
130 "content": [
131 + # NOTE: The prompt formatting with the image token `<image>` is not needed
132 + # since the prompt will be processed automatically by the API server.
133 {"type": "text", "text": "What's in this image?"},
134 {
135 "type": "image_url",
@@ -144,5 +153,4 @@ A full code example can be found in `examples/openai_vision_api_client.py <https
153 export VLLM_IMAGE_FETCH_TIMEOUT=<timeout>
154
155 .. note::
147 - The prompt formatting with the image token ``<image>`` is not needed when serving VLMs with the API server since the prompt will be
148 - processed automatically by the server.
156 + There is no need to format the prompt in the API request since it will be handled by the server.
examples/llava_example.py
+1 -2
@@ -17,8 +17,7 @@ def run_llava():
17 image_feature_size=576,
18 )
19
20 - prompt = "<image>" * 576 + (
21 - "\nUSER: What is the content of this image?\nASSISTANT:")
20 + prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:"
21
22 image = Image.open("images/stop_sign.jpg")
23
examples/llava_next_example.py
+3 -8
@@ -5,22 +5,17 @@ from PIL import Image
5
6 from vllm import LLM, SamplingParams
7
8 -# Dynamic image input is currently not supported and therefore
9 -# a fixed image input shape and its corresponding feature size is required.
10 -# See https://github.com/vllm-project/vllm/pull/4199 for the complete
11 -# configuration matrix.
12 -
8
9 def run_llava_next():
10 llm = LLM(
11 model="llava-hf/llava-v1.6-mistral-7b-hf",
12 image_token_id=32000,
13 image_input_shape="1,3,336,336",
19 - image_feature_size=1176,
14 + # Use the maximum possible value for memory profiling
15 + image_feature_size=2928,
16 )
17
22 - prompt = "[INST] " + "<image>" * 1176 + (
23 - "\nWhat is shown in this image? [/INST]")
18 + prompt = "[INST] <image>\nWhat is shown in this image? [/INST]"
19 url = "https://h2o-release.s3.amazonaws.com/h2ogpt/bigben.jpg"
20 image = Image.open(BytesIO(requests.get(url).content))
21 sampling_params = SamplingParams(temperature=0.8,
examples/phi3v_example.py
+5 -3
@@ -5,6 +5,9 @@ from PIL import Image
5
6 from vllm import LLM, SamplingParams
7
8 +# The assets are located at `s3://air-example-data-2/vllm_opensource_llava/`.
9 +# You can use `.buildkite/download-images.sh` to download them
10 +
11
12 def run_phi3v():
13 model_path = "microsoft/Phi-3-vision-128k-instruct"
@@ -18,7 +21,8 @@ def run_phi3v():
21 trust_remote_code=True,
22 image_token_id=32044,
23 image_input_shape="1,3,1008,1344",
21 - image_feature_size=1921,
24 + # Use the maximum possible value for memory profiling
25 + image_feature_size=2653,
26 max_num_seqs=5,
27 )
28
@@ -26,8 +30,6 @@ def run_phi3v():
30
31 # single-image prompt
32 prompt = "<|user|>\n<|image_1|>\nWhat is the season?<|end|>\n<|assistant|>\n" # noqa: E501
29 - prompt = prompt.replace("<|image_1|>", "<|image|>" * 1921 + "<s>")
30 -
33 sampling_params = SamplingParams(temperature=0, max_tokens=64)
34
35 outputs = llm.generate(
tests/conftest.py
+81 -33
@@ -1,12 +1,13 @@
1 import contextlib
2 import gc
3 import os
4 +import sys
5 from collections import UserList
6 from dataclasses import dataclass
7 from functools import cached_property
8 from pathlib import Path
8 -from typing import (TYPE_CHECKING, Any, Dict, List, Literal, Optional, Tuple,
9 - TypedDict, TypeVar)
9 +from typing import (Any, Dict, List, Literal, Optional, Tuple, TypedDict,
10 + TypeVar)
11
12 import pytest
13 import torch
@@ -22,13 +23,10 @@ from vllm.distributed import (destroy_distributed_environment,
23 destroy_model_parallel)
24 from vllm.inputs import TextPrompt
25 from vllm.logger import init_logger
26 +from vllm.multimodal.utils import fetch_image
27 from vllm.sequence import SampleLogprobs
28 from vllm.utils import cuda_device_count_stateless, is_cpu
29
28 -if TYPE_CHECKING:
29 - # it will call torch.cuda.device_count()
30 - from vllm.multimodal import MultiModalDataDict
31 -
30 logger = init_logger(__name__)
31
32 _TEST_DIR = os.path.dirname(__file__)
@@ -47,30 +45,42 @@ def _read_prompts(filename: str) -> List[str]:
45
46 @dataclass(frozen=True)
47 class ImageAsset:
50 - name: Literal["stop_sign", "cherry_blossom"]
48 + name: Literal["stop_sign", "cherry_blossom", "boardwalk"]
49
50 @cached_property
51 def pil_image(self) -> Image.Image:
54 - return Image.open(_IMAGE_DIR / f"{self.name}.jpg")
55 -
56 - def for_hf(self) -> Image.Image:
57 - return self.pil_image
52 + if self.name == "boardwalk":
53 + return fetch_image(
54 + "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
55 + )
56
59 - def for_vllm(self) -> Dict[str, Any]:
60 - return {"image": self.pil_image}
57 + return Image.open(_IMAGE_DIR / f"{self.name}.jpg")
58
59
60 class _ImageAssetPrompts(TypedDict):
61 stop_sign: str
62 cherry_blossom: str
63 + boardwalk: str
64 +
65 +
66 +if sys.version_info < (3, 9):
67 + # UserList cannot be subscripted
68 + class _ImageAssetsBase(UserList):
69 + pass
70 +else:
71
72 + class _ImageAssetsBase(UserList[ImageAsset]):
73 + pass
74
68 -class _ImageAssets(UserList):
75 +
76 +class _ImageAssets(_ImageAssetsBase):
77
78 def __init__(self) -> None:
71 - super().__init__(
72 - [ImageAsset("stop_sign"),
73 - ImageAsset("cherry_blossom")])
79 + super().__init__([
80 + ImageAsset("stop_sign"),
81 + ImageAsset("cherry_blossom"),
82 + ImageAsset("boardwalk")
83 + ])
84
85 def prompts(self, prompts: _ImageAssetPrompts) -> List[str]:
86 """
@@ -79,7 +89,10 @@ class _ImageAssets(UserList):
89 The order of the returned prompts matches the order of the
90 assets when iterating through this object.
91 """
82 - return [prompts["stop_sign"], prompts["cherry_blossom"]]
92 + return [
93 + prompts["stop_sign"], prompts["cherry_blossom"],
94 + prompts["boardwalk"]
95 + ]
96
97
98 IMAGE_ASSETS = _ImageAssets()
@@ -220,7 +233,7 @@ class HfRunner:
233 self,
234 prompts: List[str],
235 images: Optional[List[Image.Image]] = None,
223 - **kwargs,
236 + **kwargs: Any,
237 ) -> List[Tuple[List[List[int]], List[str]]]:
238 if images:
239 assert len(prompts) == len(images)
@@ -255,7 +268,7 @@ class HfRunner:
268 prompts: List[str],
269 max_tokens: int,
270 images: Optional[List[Image.Image]] = None,
258 - **kwargs,
271 + **kwargs: Any,
272 ) -> List[Tuple[List[int], str]]:
273 outputs = self.generate(prompts,
274 do_sample=False,
@@ -291,19 +304,30 @@ class HfRunner:
304 self,
305 prompts: List[str],
306 max_tokens: int,
307 + images: Optional[List[Image.Image]] = None,
308 + **kwargs: Any,
309 ) -> List[List[torch.Tensor]]:
295 - all_logprobs = []
296 - for prompt in prompts:
297 - input_ids = self.tokenizer(prompt, return_tensors="pt").input_ids
310 + all_logprobs: List[List[torch.Tensor]] = []
311 + for i, prompt in enumerate(prompts):
312 + processor_kwargs: Dict[str, Any] = {
313 + "text": prompt,
314 + "return_tensors": "pt",
315 + }
316 + if images is not None and images[i] is not None:
317 + processor_kwargs["images"] = images[i]
318 +
319 + inputs = self.processor(**processor_kwargs)
320 +
321 output = self.model.generate(
299 - self.wrap_device(input_ids),
322 + **self.wrap_device(inputs),
323 use_cache=True,
324 do_sample=False,
325 max_new_tokens=max_tokens,
326 output_hidden_states=True,
327 return_dict_in_generate=True,
328 + **kwargs,
329 )
306 - seq_logprobs = []
330 + seq_logprobs: List[torch.Tensor] = []
331 for hidden_states in output.hidden_states:
332 last_hidden_states = hidden_states[-1][0]
333 logits = torch.matmul(
@@ -323,20 +347,32 @@ class HfRunner:
347 prompts: List[str],
348 max_tokens: int,
349 num_logprobs: int,
350 + images: Optional[List[Image.Image]] = None,
351 + **kwargs: Any,
352 ) -> List[Tuple[List[int], str, List[Dict[int, float]]]]:
353 all_logprobs: List[List[Dict[int, float]]] = []
354 all_output_ids: List[List[int]] = []
355 all_output_strs: List[str] = []
356
331 - for prompt in prompts:
332 - input_ids = self.tokenizer(prompt, return_tensors="pt").input_ids
357 + for i, prompt in enumerate(prompts):
358 + processor_kwargs: Dict[str, Any] = {
359 + "text": prompt,
360 + "return_tensors": "pt",
361 + }
362 + if images is not None and images[i] is not None:
363 + processor_kwargs["images"] = images[i]
364 +
365 + inputs = self.processor(**processor_kwargs)
366 + input_ids = inputs.input_ids
367 +
368 output = self.model.generate(
334 - self.wrap_device(input_ids),
369 + **self.wrap_device(inputs),
370 use_cache=True,
371 do_sample=False,
372 max_new_tokens=max_tokens,
373 output_hidden_states=True,
374 return_dict_in_generate=True,
375 + **kwargs,
376 )
377
378 seq_logprobs: List[torch.Tensor] = []
@@ -431,7 +467,7 @@ class VllmRunner:
467 self,
468 prompts: List[str],
469 sampling_params: SamplingParams,
434 - images: Optional[List["MultiModalDataDict"]] = None,
470 + images: Optional[List[Image.Image]] = None,
471 ) -> List[Tuple[List[List[int]], List[str]]]:
472 if images is not None:
473 assert len(prompts) == len(images)
@@ -439,7 +475,7 @@ class VllmRunner:
475 inputs = [TextPrompt(prompt=prompt) for prompt in prompts]
476 if images is not None:
477 for i, image in enumerate(images):
442 - inputs[i]["multi_modal_data"] = image
478 + inputs[i]["multi_modal_data"] = {"image": image}
479
480 req_outputs = self.model.generate(inputs,
481 sampling_params=sampling_params)
@@ -462,10 +498,19 @@ class VllmRunner:
498 self,
499 prompts: List[str],
500 sampling_params: SamplingParams,
501 + images: Optional[List[Image.Image]] = None,
502 ) -> List[Tuple[List[int], str, Optional[SampleLogprobs]]]:
503 assert sampling_params.logprobs is not None
504
468 - req_outputs = self.model.generate(prompts,
505 + if images is not None:
506 + assert len(prompts) == len(images)
507 +
508 + inputs = [TextPrompt(prompt=prompt) for prompt in prompts]
509 + if images is not None:
510 + for i, image in enumerate(images):
511 + inputs[i]["multi_modal_data"] = {"image": image}
512 +
513 + req_outputs = self.model.generate(inputs,
514 sampling_params=sampling_params)
515 outputs: List[Tuple[List[int], str, Optional[SampleLogprobs]]] = []
516 for req_output in req_outputs:
@@ -480,7 +525,7 @@ class VllmRunner:
525 self,
526 prompts: List[str],
527 max_tokens: int,
483 - images: Optional[List["MultiModalDataDict"]] = None,
528 + images: Optional[List[Image.Image]] = None,
529 ) -> List[Tuple[List[int], str]]:
530 greedy_params = SamplingParams(temperature=0.0, max_tokens=max_tokens)
531 outputs = self.generate(prompts, greedy_params, images=images)
@@ -492,11 +537,14 @@ class VllmRunner:
537 prompts: List[str],
538 max_tokens: int,
539 num_logprobs: int,
540 + images: Optional[List[Image.Image]] = None,
541 ) -> List[Tuple[List[int], str, Optional[SampleLogprobs]]]:
542 greedy_logprobs_params = SamplingParams(temperature=0.0,
543 max_tokens=max_tokens,
544 logprobs=num_logprobs)
499 - outputs = self.generate_w_logprobs(prompts, greedy_logprobs_params)
545 + outputs = self.generate_w_logprobs(prompts,
546 + greedy_logprobs_params,
547 + images=images)
548
549 return [(output_ids, output_str, output_logprobs)
550 for output_ids, output_str, output_logprobs in outputs]
tests/distributed/test_multimodal_broadcast.py
+5 -2
@@ -30,9 +30,10 @@ else:
30 @pytest.mark.parametrize("tensor_parallel_size", [2])
31 @pytest.mark.parametrize("dtype", ["half"])
32 @pytest.mark.parametrize("max_tokens", [128])
33 +@pytest.mark.parametrize("num_logprobs", [5])
34 def test_models(hf_runner, vllm_runner, image_assets,
34 - tensor_parallel_size: int, dtype: str,
35 - max_tokens: int) -> None:
35 + tensor_parallel_size: int, dtype: str, max_tokens: int,
36 + num_logprobs: int) -> None:
37 if cuda_device_count_stateless() < tensor_parallel_size:
38 pytest.skip(
39 f"Need at least {tensor_parallel_size} GPUs to run the test.")
@@ -44,8 +45,10 @@ def test_models(hf_runner, vllm_runner, image_assets,
45 vllm_runner,
46 image_assets,
47 model_and_config=model_and_vl_config[0],
48 + size_factors=[1.0],
49 dtype=dtype,
50 max_tokens=max_tokens,
51 + num_logprobs=num_logprobs,
52 tensor_parallel_size=tensor_parallel_size,
53 distributed_executor_backend=distributed_executor_backend,
54 )
tests/models/test_llava.py
+71 -35
@@ -4,18 +4,21 @@ import pytest
4 from transformers import AutoTokenizer
5
6 from vllm.config import VisionLanguageConfig
7 +from vllm.multimodal.utils import rescale_image_size
8 +from vllm.sequence import SampleLogprobs
9
10 from ..conftest import IMAGE_ASSETS, HfRunner, VllmRunner, _ImageAssets
9 -from .utils import check_outputs_equal
11 +from .utils import check_logprobs_close
12
13 pytestmark = pytest.mark.vlm
14
13 -# The image token is placed before "user" on purpose so that the test can pass
15 HF_IMAGE_PROMPTS = IMAGE_ASSETS.prompts({
16 "stop_sign":
16 - "<image>\nUSER: What's the content of the image?\nASSISTANT:",
17 + "USER: <image>\nWhat's the content of the image?\nASSISTANT:",
18 "cherry_blossom":
18 - "<image>\nUSER: What is the season?\nASSISTANT:",
19 + "USER: <image>\nWhat is the season?\nASSISTANT:",
20 + "boardwalk":
21 + "USER: <image>\nWhat's in this image?\nASSISTANT:",
22 })
23
24
@@ -37,27 +40,34 @@ model_and_vl_config = [
40 ]
41
42
40 -def vllm_to_hf_output(vllm_output: Tuple[List[int], str],
43 +def vllm_to_hf_output(vllm_output: Tuple[List[int], str,
44 + Optional[SampleLogprobs]],
45 vlm_config: VisionLanguageConfig, model_id: str):
46 """Sanitize vllm output to be comparable with hf output.
47 The function reduces `input_ids` from 1, 32000, 32000, ..., 32000,
48 x1, x2, x3 ... to 1, 32000, x1, x2, x3 ...
49 It also reduces `output_str` from "<image><image>bla" to "bla".
50 """
47 - output_ids, output_str = vllm_output
51 + output_ids, output_str, out_logprobs = vllm_output
52 image_token_id = vlm_config.image_token_id
53
54 tokenizer = AutoTokenizer.from_pretrained(model_id)
55 image_token_str = tokenizer.decode(image_token_id)
56 + eos_token_id = tokenizer.eos_token_id
57
58 hf_output_ids = [
59 token_id for idx, token_id in enumerate(output_ids)
60 if token_id != image_token_id or output_ids[idx - 1] != image_token_id
61 ]
62 +
63 hf_output_str = output_str \
64 .replace(image_token_str * vlm_config.image_feature_size, "")
65 + assert hf_output_str[0] == " "
66 + hf_output_str = hf_output_str[1:]
67 + if hf_output_ids[-1] == eos_token_id:
68 + hf_output_str = hf_output_str + tokenizer.decode(eos_token_id)
69
60 - return hf_output_ids, hf_output_str
70 + return hf_output_ids, hf_output_str, out_logprobs
71
72
73 def run_test(
@@ -66,8 +76,10 @@ def run_test(
76 image_assets: _ImageAssets,
77 model_and_config: Tuple[str, VisionLanguageConfig],
78 *,
79 + size_factors: List[float],
80 dtype: str,
81 max_tokens: int,
82 + num_logprobs: int,
83 tensor_parallel_size: int,
84 distributed_executor_backend: Optional[str] = None,
85 ):
@@ -81,61 +93,85 @@ def run_test(
93 The text output is sanitized to be able to compare with hf.
94 """
95 model_id, vlm_config = model_and_config
84 - hf_images = [asset.for_hf() for asset in image_assets]
96 + images = [asset.pil_image for asset in image_assets]
97 +
98 + inputs_per_image = [(
99 + [prompt for _ in size_factors],
100 + [rescale_image_size(image, factor) for factor in size_factors],
101 + ) for image, prompt in zip(images, HF_IMAGE_PROMPTS)]
102
103 # NOTE: take care of the order. run vLLM first, and then run HF.
104 # vLLM needs a fresh new process without cuda initialization.
105 # if we run HF first, the cuda initialization will be done and it
106 # will hurt multiprocessing backend with fork method (the default method).
107
108 + # max_model_len should be greater than image_feature_size
109 with vllm_runner(model_id,
110 dtype=dtype,
111 tensor_parallel_size=tensor_parallel_size,
112 distributed_executor_backend=distributed_executor_backend,
113 enforce_eager=True,
114 **vlm_config.as_cli_args_dict()) as vllm_model:
97 -
98 - # NOTE: `asset.for_vllm` will call `torch.cuda.device_count()`
99 - # we must put it inside the vllm_runner context manager
100 - # i.e. after creating vLLM instance.
101 - vllm_images = [asset.for_vllm() for asset in image_assets]
102 -
103 - vllm_image_prompts = [
104 - p.replace("<image>", "<image>" * vlm_config.image_feature_size)
105 - for p in HF_IMAGE_PROMPTS
115 + vllm_outputs_per_image = [
116 + vllm_model.generate_greedy_logprobs(prompts,
117 + max_tokens,
118 + num_logprobs=num_logprobs,
119 + images=images)
120 + for prompts, images in inputs_per_image
121 ]
122
108 - vllm_outputs = vllm_model.generate_greedy(vllm_image_prompts,
109 - max_tokens,
110 - images=vllm_images)
111 -
123 with hf_runner(model_id, dtype=dtype, is_vision_model=True) as hf_model:
113 - hf_outputs = hf_model.generate_greedy(HF_IMAGE_PROMPTS,
114 - max_tokens,
115 - images=hf_images)
116 -
117 - check_outputs_equal(
118 - hf_outputs,
119 - [
120 - vllm_to_hf_output(vllm_output, vlm_config, model_id)
121 - for vllm_output in vllm_outputs
122 - ],
123 - name_0="hf",
124 - name_1="vllm",
125 - )
124 + hf_outputs_per_image = [
125 + hf_model.generate_greedy_logprobs_limit(prompts,
126 + max_tokens,
127 + num_logprobs=num_logprobs,
128 + images=images)
129 + for prompts, images in inputs_per_image
130 + ]
131 +
132 + for hf_outputs, vllm_outputs in zip(hf_outputs_per_image,
133 + vllm_outputs_per_image):
134 + # TODO: Check whether using original CLIPVisionModel can improve
135 + # consistency against HF
136 + check_logprobs_close(
137 + outputs_0_lst=hf_outputs,
138 + outputs_1_lst=[
139 + vllm_to_hf_output(vllm_output, vlm_config, model_id)
140 + for vllm_output in vllm_outputs
141 + ],
142 + name_0="hf",
143 + name_1="vllm",
144 + )
145
146
147 @pytest.mark.parametrize("model_and_config", model_and_vl_config)
148 +@pytest.mark.parametrize(
149 + "size_factors",
150 + [
151 + # No image
152 + [],
153 + # Single-scale
154 + [1.0],
155 + # Single-scale, batched
156 + [1.0, 1.0, 1.0],
157 + # Multi-scale
158 + [0.25, 0.5, 1.0],
159 + ],
160 +)
161 @pytest.mark.parametrize("dtype", ["half"])
162 @pytest.mark.parametrize("max_tokens", [128])
163 +@pytest.mark.parametrize("num_logprobs", [5])
164 def test_models(hf_runner, vllm_runner, image_assets, model_and_config,
132 - dtype: str, max_tokens: int) -> None:
165 + size_factors, dtype: str, max_tokens: int,
166 + num_logprobs: int) -> None:
167 run_test(
168 hf_runner,
169 vllm_runner,
170 image_assets,
171 model_and_config,
172 + size_factors=size_factors,
173 dtype=dtype,
174 max_tokens=max_tokens,
175 + num_logprobs=num_logprobs,
176 tensor_parallel_size=1,
177 )
tests/models/test_llava_next.py
+79 -50
@@ -1,12 +1,15 @@
1 -from typing import List, Tuple
1 +import re
2 +from typing import List, Optional, Tuple
3
4 import pytest
5 from transformers import AutoTokenizer
6
7 from vllm.config import VisionLanguageConfig
8 +from vllm.multimodal.utils import rescale_image_size
9 +from vllm.sequence import SampleLogprobs
10
11 from ..conftest import IMAGE_ASSETS
9 -from .utils import check_outputs_equal
12 +from .utils import check_logprobs_close
13
14 pytestmark = pytest.mark.vlm
15
@@ -15,21 +18,20 @@ _PREFACE = (
18 "The assistant gives helpful, detailed, and polite answers to the human's "
19 "questions.")
20
18 -# The image token is placed before "user" on purpose so that the test can pass
21 HF_IMAGE_PROMPTS = IMAGE_ASSETS.prompts({
22 "stop_sign":
21 - f"{_PREFACE} <image>\nUSER: What's the content of the image?\nASSISTANT:",
23 + f"{_PREFACE} USER: <image>\nWhat's the content of the image? ASSISTANT:",
24 "cherry_blossom":
23 - f"{_PREFACE} <image>\nUSER: What is the season?\nASSISTANT:",
25 + f"{_PREFACE} USER: <image>\nWhat is the season? ASSISTANT:",
26 + "boardwalk":
27 + f"{_PREFACE} USER: <image>\nWhat's in this image? ASSISTANT:",
28 })
29
30
31 def iter_llava_next_configs(model_name: str):
32 + # Need to use the max possible feature size for profile_run
33 image_hw_to_feature_size = {
29 - (336, 336): 1176,
30 - (672, 672): 2928,
31 - (1344, 336): 1944,
32 - (336, 1344): 1890,
34 + (336, 336): 2928,
35 }
36
37 for (h, w), f in image_hw_to_feature_size.items():
@@ -47,37 +49,55 @@ model_and_vl_config = [
49 ]
50
51
50 -def vllm_to_hf_output(vllm_output: Tuple[List[int], str],
52 +def vllm_to_hf_output(vllm_output: Tuple[List[int], str,
53 + Optional[SampleLogprobs]],
54 vlm_config: VisionLanguageConfig, model_id: str):
55 """Sanitize vllm output to be comparable with hf output.
56 The function reduces `input_ids` from 1, 32000, 32000, ..., 32000,
57 x1, x2, x3 ... to 1, 32000, x1, x2, x3 ...
58 It also reduces `output_str` from "<image><image>bla" to "bla".
59 """
57 - output_ids, output_str = vllm_output
60 + output_ids, output_str, out_logprobs = vllm_output
61 image_token_id = vlm_config.image_token_id
62
63 tokenizer = AutoTokenizer.from_pretrained(model_id)
64 image_token_str = tokenizer.decode(image_token_id)
65 + eos_token_id = tokenizer.eos_token_id
66
67 hf_output_ids = [
68 token_id for idx, token_id in enumerate(output_ids)
69 if token_id != image_token_id or output_ids[idx - 1] != image_token_id
70 ]
67 - hf_output_str = output_str \
68 - .replace(image_token_str * vlm_config.image_feature_size, " ")
71
70 - return hf_output_ids, hf_output_str
72 + hf_output_str = re.sub(fr"({image_token_str})+", "", output_str)
73 + assert hf_output_str[0] == " "
74 + hf_output_str = hf_output_str[1:]
75 + if hf_output_ids[-1] == eos_token_id:
76 + hf_output_str = hf_output_str + tokenizer.decode(eos_token_id)
77 +
78 + return hf_output_ids, hf_output_str, out_logprobs
79
80
73 -@pytest.mark.xfail(
74 - reason="Inconsistent image processor being used due to lack "
75 - "of support for dynamic image token replacement")
81 @pytest.mark.parametrize("model_and_config", model_and_vl_config)
82 +@pytest.mark.parametrize(
83 + "size_factors",
84 + [
85 + # No image
86 + [],
87 + # Single-scale
88 + [1.0],
89 + # Single-scale, batched
90 + [1.0, 1.0, 1.0],
91 + # Multi-scale
92 + [0.25, 0.5, 1.0],
93 + ],
94 +)
95 @pytest.mark.parametrize("dtype", ["half"])
96 @pytest.mark.parametrize("max_tokens", [128])
97 +@pytest.mark.parametrize("num_logprobs", [5])
98 def test_models(hf_runner, vllm_runner, image_assets, model_and_config,
80 - dtype: str, max_tokens: int) -> None:
99 + size_factors, dtype: str, max_tokens: int,
100 + num_logprobs: int) -> None:
101 """Inference result should be the same between hf and vllm.
102
103 All the image fixtures for the test is under tests/images.
@@ -88,37 +108,46 @@ def test_models(hf_runner, vllm_runner, image_assets, model_and_config,
108 The text output is sanitized to be able to compare with hf.
109 """
110 model_id, vlm_config = model_and_config
91 - hf_images = [asset.for_hf() for asset in image_assets]
92 - vllm_images = [asset.for_vllm() for asset in image_assets]
111 + images = [asset.pil_image for asset in image_assets]
112 +
113 + inputs_per_image = [(
114 + [prompt for _ in size_factors],
115 + [rescale_image_size(image, factor) for factor in size_factors],
116 + ) for image, prompt in zip(images, HF_IMAGE_PROMPTS)]
117 +
118 + # max_model_len should be greater than image_feature_size
119 + with vllm_runner(model_id,
120 + dtype=dtype,
121 + max_model_len=4096,
122 + enforce_eager=True,
123 + **vlm_config.as_cli_args_dict()) as vllm_model:
124 + vllm_outputs_per_image = [
125 + vllm_model.generate_greedy_logprobs(prompts,
126 + max_tokens,
127 + num_logprobs=num_logprobs,
128 + images=images)
129 + for prompts, images in inputs_per_image
130 + ]
131
132 with hf_runner(model_id, dtype=dtype, is_vision_model=True) as hf_model:
95 - hf_outputs = hf_model.generate_greedy(HF_IMAGE_PROMPTS,
96 - max_tokens,
97 - images=hf_images)
98 -
99 - vllm_image_prompts = [
100 - p.replace("<image>", "<image>" * vlm_config.image_feature_size)
101 - for p in HF_IMAGE_PROMPTS
102 - ]
103 -
104 - with vllm_runner(
105 - model_id,
106 - dtype=dtype,
107 - # should be greater than image_feature_size
108 - max_model_len=4096,
109 - enforce_eager=True,
110 - **vlm_config.as_cli_args_dict(),
111 - ) as vllm_model:
112 - vllm_outputs = vllm_model.generate_greedy(vllm_image_prompts,
113 - max_tokens,
114 - images=vllm_images)
115 -
116 - check_outputs_equal(
117 - hf_outputs,
118 - [
119 - vllm_to_hf_output(vllm_output, vlm_config, model_id)
120 - for vllm_output in vllm_outputs
121 - ],
122 - name_0="hf",
123 - name_1="vllm",
124 - )
133 + hf_outputs_per_image = [
134 + hf_model.generate_greedy_logprobs_limit(prompts,
135 + max_tokens,
136 + num_logprobs=num_logprobs,
137 + images=images)
138 + for prompts, images in inputs_per_image
139 + ]
140 +
141 + for hf_outputs, vllm_outputs in zip(hf_outputs_per_image,
142 + vllm_outputs_per_image):
143 + # TODO: Check whether using original CLIPVisionModel can improve
144 + # consistency against HF
145 + check_logprobs_close(
146 + outputs_0_lst=hf_outputs,
147 + outputs_1_lst=[
148 + vllm_to_hf_output(vllm_output, vlm_config, model_id)
149 + for vllm_output in vllm_outputs
150 + ],
151 + name_0="hf",
152 + name_1="vllm",
153 + )
tests/models/test_phi3v.py
+82 -58
@@ -1,29 +1,33 @@
1 +import re
2 from typing import List, Optional, Tuple, Type
3
4 import pytest
5 from transformers import AutoTokenizer
6
7 from vllm.config import VisionLanguageConfig
8 +from vllm.multimodal.utils import rescale_image_size
9 +from vllm.sequence import SampleLogprobs
10 from vllm.utils import is_cpu
11
12 from ..conftest import IMAGE_ASSETS, HfRunner, VllmRunner, _ImageAssets
10 -from .utils import check_outputs_equal
13 +from .utils import check_logprobs_close
14
15 pytestmark = pytest.mark.vlm
16
14 -# The image token is placed before "user" on purpose so that the test can pass
17 HF_IMAGE_PROMPTS = IMAGE_ASSETS.prompts({
18 "stop_sign":
19 "<|user|>\n<|image_1|>\nWhat's the content of the image?<|end|>\n<|assistant|>\n", # noqa: E501
20 "cherry_blossom":
19 - "<|user|>\n<|image_1|>\nWhat is the season?<|end|>\n<|assistant|>\n", # noqa: E501
21 + "<|user|>\n<|image_1|>\nWhat is the season?<|end|>\n<|assistant|>\n",
22 + "boardwalk":
23 + "<|user|>\n<|image_1|>\nWhat's in this image?<|end|>\n<|assistant|>\n",
24 })
25
26
27 def iter_phi3v_configs(model_name: str):
28 + # Need to use the max possible feature size for profile_run
29 image_hw_to_feature_size = {
25 - (1008, 1344): 1921,
26 - (2016, 2688): 1933,
30 + (1008, 1344): 2653,
31 }
32
33 for (h, w), f in image_hw_to_feature_size.items():
@@ -39,29 +43,29 @@ model_and_vl_config = [
43 ]
44
45
42 -def vllm_to_hf_output(vllm_output: Tuple[List[int], str],
46 +def vllm_to_hf_output(vllm_output: Tuple[List[int], str,
47 + Optional[SampleLogprobs]],
48 vlm_config: VisionLanguageConfig, model_id: str):
49 """Sanitize vllm output to be comparable with hf output.
50 The function reduces `input_ids` from 1, 32000, 32000, ..., 32000,
51 x1, x2, x3 ... to 1, 32000, x1, x2, x3 ...
52 It also reduces `output_str` from "<image><image>bla" to "bla".
53 """
49 - output_ids, output_str = vllm_output
50 - image_token_id = vlm_config.image_token_id
54 + output_ids, output_str, out_logprobs = vllm_output
55
52 - tokenizer = AutoTokenizer.from_pretrained(model_id)
53 - image_token_str = tokenizer.decode(image_token_id)
54 -
55 - hf_output_ids = [
56 - token_id if token_id != image_token_id else 0
57 - for idx, token_id in enumerate(output_ids)
58 - ]
59 - hf_output_str = output_str \
60 - .replace(image_token_str * vlm_config.image_feature_size, "") \
61 - .replace("<s>", " ").replace("<|user|>", "") \
56 + output_str_without_image = re.sub(r"(<\|image_\d+\|>)+", "", output_str)
57 + assert output_str_without_image[0] == " "
58 + output_str_without_image = output_str_without_image[1:]
59 +
60 + hf_output_str = output_str_without_image.replace("<|user|>", "") \
61 .replace("<|end|>\n<|assistant|>", " ")
62
64 - return hf_output_ids, hf_output_str
63 + tokenizer = AutoTokenizer.from_pretrained(model_id)
64 + hf_output_ids = tokenizer.encode(output_str_without_image)
65 + assert hf_output_ids[0] == 1
66 + hf_output_ids = hf_output_ids[1:]
67 +
68 + return hf_output_ids, hf_output_str, out_logprobs
69
70
71 target_dtype = "half"
@@ -75,8 +79,10 @@ def run_test(
79 image_assets: _ImageAssets,
80 model_and_config: Tuple[str, VisionLanguageConfig],
81 *,
82 + size_factors: List[float],
83 dtype: str,
84 max_tokens: int,
85 + num_logprobs: int,
86 tensor_parallel_size: int,
87 distributed_executor_backend: Optional[str] = None,
88 ):
@@ -90,73 +96,91 @@ def run_test(
96 The text output is sanitized to be able to compare with hf.
97 """
98 model_id, vlm_config = model_and_config
93 - hf_images = [asset.for_hf() for asset in image_assets]
99 + images = [asset.pil_image for asset in image_assets]
100 +
101 + inputs_per_image = [(
102 + [prompt for _ in size_factors],
103 + [rescale_image_size(image, factor) for factor in size_factors],
104 + ) for image, prompt in zip(images, HF_IMAGE_PROMPTS)]
105
106 # NOTE: take care of the order. run vLLM first, and then run HF.
107 # vLLM needs a fresh new process without cuda initialization.
108 # if we run HF first, the cuda initialization will be done and it
109 # will hurt multiprocessing backend with fork method (the default method).
110
111 + # max_model_len should be greater than image_feature_size
112 with vllm_runner(model_id,
101 - max_model_len=2048,
113 + max_model_len=4096,
114 dtype=dtype,
115 tensor_parallel_size=tensor_parallel_size,
104 - enforce_eager=True,
116 distributed_executor_backend=distributed_executor_backend,
117 + enforce_eager=True,
118 **vlm_config.as_cli_args_dict()) as vllm_model:
107 - # NOTE: `asset.for_vllm` will call `torch.cuda.device_count()`
108 - # we must put it inside the vllm_runner context manager
109 - # i.e. after creating vLLM instance.
110 -
111 - vllm_images = [asset.for_vllm() for asset in image_assets]
112 -
113 - vllm_image_prompts = [
114 - p.replace("<|image_1|>",
115 - "<|image|>" * vlm_config.image_feature_size + "<s>")
116 - for p in HF_IMAGE_PROMPTS
119 + vllm_outputs_per_image = [
120 + vllm_model.generate_greedy_logprobs(prompts,
121 + max_tokens,
122 + num_logprobs=num_logprobs,
123 + images=vllm_images)
124 + for prompts, vllm_images in inputs_per_image
125 ]
126
119 - vllm_outputs = vllm_model.generate_greedy(vllm_image_prompts,
120 - max_tokens,
121 - images=vllm_images)
122 -
127 # use eager mode for hf runner, since phi3_v didn't work with flash_attn
128 hf_model_kwargs = {"_attn_implementation": "eager"}
129 with hf_runner(model_id, dtype=dtype,
130 model_kwargs=hf_model_kwargs) as hf_model:
127 - hf_outputs = hf_model.generate_greedy(
128 - HF_IMAGE_PROMPTS,
129 - max_tokens,
130 - images=hf_images,
131 - eos_token_id=hf_model.processor.tokenizer.eos_token_id)
132 -
133 - check_outputs_equal(
134 - hf_outputs,
135 - [
136 - vllm_to_hf_output(vllm_output, vlm_config, model_id)
137 - for vllm_output in vllm_outputs
138 - ],
139 - name_0="hf",
140 - name_1="vllm",
141 - )
142 -
131 + eos_token_id = hf_model.processor.tokenizer.eos_token_id
132 + hf_outputs_per_image = [
133 + hf_model.generate_greedy_logprobs_limit(prompts,
134 + max_tokens,
135 + num_logprobs=num_logprobs,
136 + images=hf_images,
137 + eos_token_id=eos_token_id)
138 + for prompts, hf_images in inputs_per_image
139 + ]
140
144 -# Since we use _attn_implementation="eager" for hf_runner, here is
145 -# numeric difference for longer context and test can't pass
146 -@pytest.mark.xfail(
147 - reason="Inconsistent image processor being used due to lack "
148 - "of support for dynamic image token replacement")
141 + for hf_outputs, vllm_outputs in zip(hf_outputs_per_image,
142 + vllm_outputs_per_image):
143 + check_logprobs_close(
144 + outputs_0_lst=hf_outputs,
145 + outputs_1_lst=[
146 + vllm_to_hf_output(vllm_output, vlm_config, model_id)
147 + for vllm_output in vllm_outputs
148 + ],
149 + name_0="hf",
150 + name_1="vllm",
151 + )
152 +
153 +
154 +# Since we use _attn_implementation="eager" for hf_runner, there is more
155 +# significant numerical difference. The basic `logprobs=5` fails to pass.
156 @pytest.mark.parametrize("model_and_config", model_and_vl_config)
157 +@pytest.mark.parametrize(
158 + "size_factors",
159 + [
160 + # No image
161 + [],
162 + # Single-scale
163 + [1.0],
164 + # Single-scale, batched
165 + [1.0, 1.0, 1.0],
166 + # Multi-scale
167 + [0.25, 0.5, 1.0],
168 + ],
169 +)
170 @pytest.mark.parametrize("dtype", [target_dtype])
171 @pytest.mark.parametrize("max_tokens", [128])
172 +@pytest.mark.parametrize("num_logprobs", [10])
173 def test_models(hf_runner, vllm_runner, image_assets, model_and_config,
153 - dtype: str, max_tokens: int) -> None:
174 + size_factors, dtype: str, max_tokens: int,
175 + num_logprobs: int) -> None:
176 run_test(
177 hf_runner,
178 vllm_runner,
179 image_assets,
180 model_and_config,
181 + size_factors=size_factors,
182 dtype=dtype,
183 max_tokens=max_tokens,
184 + num_logprobs=num_logprobs,
185 tensor_parallel_size=1,
186 )
tests/models/utils.py
+67 -21
@@ -1,11 +1,18 @@
1 -from typing import Dict, List, Tuple
1 +import warnings
2 +from typing import Dict, List, Optional, Sequence, Tuple, Union
3 +
4 +from vllm.sequence import SampleLogprobs
5
6 TokensText = Tuple[List[int], str]
7
8
6 -def check_outputs_equal(outputs_0_lst: List[TokensText],
7 - outputs_1_lst: List[TokensText], name_0: str,
8 - name_1: str):
9 +def check_outputs_equal(
10 + *,
11 + outputs_0_lst: Sequence[TokensText],
12 + outputs_1_lst: Sequence[TokensText],
13 + name_0: str,
14 + name_1: str,
15 +):
16 """
17 Compare the two sequences generated by different models,
18 which should be equal.
@@ -18,20 +25,28 @@ def check_outputs_equal(outputs_0_lst: List[TokensText],
25 output_ids_0, output_str_0 = outputs_0
26 output_ids_1, output_str_1 = outputs_1
27
21 - assert output_str_0 == output_str_1, (f"Test{prompt_idx}:"
22 - f"\n{name_0}:\t{output_str_0!r}"
23 - f"\n{name_1}:\t{output_str_1!r}")
24 - assert output_ids_0 == output_ids_1, (f"Test{prompt_idx}:"
25 - f"\n{name_0}:\t{output_str_0!r}"
26 - f"\n{name_1}:\t{output_str_1!r}")
28 + # The text and token outputs should exactly match
29 + fail_msg = (f"Test{prompt_idx}:"
30 + f"\n{name_0}:\t{output_str_0!r}"
31 + f"\n{name_1}:\t{output_str_1!r}")
32 +
33 + assert output_str_0 == output_str_1, fail_msg
34 + assert output_ids_0 == output_ids_1, fail_msg
35
36
29 -TokensTextLogprobs = Tuple[List[int], str, List[Dict[int, float]]]
37 +TokensTextLogprobs = Tuple[List[int], str, Optional[Union[List[Dict[int,
38 + float]],
39 + SampleLogprobs]]]
40
41
32 -def check_logprobs_close(outputs_0_lst: List[TokensTextLogprobs],
33 - outputs_1_lst: List[TokensTextLogprobs], name_0: str,
34 - name_1: str):
42 +def check_logprobs_close(
43 + *,
44 + outputs_0_lst: Sequence[TokensTextLogprobs],
45 + outputs_1_lst: Sequence[TokensTextLogprobs],
46 + name_0: str,
47 + name_1: str,
48 + warn_on_mismatch: bool = True,
49 +):
50 """
51 Compare the logprobs of two sequences generated by different models,
52 which should be similar but not necessarily equal.
@@ -45,21 +60,52 @@ def check_logprobs_close(outputs_0_lst: List[TokensTextLogprobs],
60 output_ids_0, output_str_0, logprobs_0 = outputs_0
61 output_ids_1, output_str_1, logprobs_1 = outputs_1
62
63 + if logprobs_0 is None:
64 + logprobs_0 = [None] * len(output_ids_0)
65 + if logprobs_1 is None:
66 + logprobs_1 = [None] * len(output_ids_1)
67 +
68 # Loop through generated tokens.
69 for idx, (output_id_0,
70 output_id_1) in enumerate(zip(output_ids_0, output_ids_1)):
71
72 # If generated tokens don't match, then
73 if output_id_0 != output_id_1:
74 + logprobs_elem_0 = logprobs_0[idx]
75 + logprobs_elem_1 = logprobs_1[idx]
76 +
77 # Each predicted token must be in top N logprobs of the other
55 - assert output_id_0 in logprobs_1[idx], (
56 - f"Test{prompt_idx}:"
57 - f"\n{name_0}:\t{output_str_0!r}"
58 - f"\n{name_1}:\t{output_str_1!r}")
59 - assert output_id_1 in logprobs_0[idx], (
78 + fail_msg = (
79 f"Test{prompt_idx}:"
61 - f"\n{name_0}:\t{output_str_0!r}"
62 - f"\n{name_1}:\t{output_str_1!r}")
80 + f"\n{name_0}:\t{output_str_0!r}\t{logprobs_elem_0}"
81 + f"\n{name_1}:\t{output_str_1!r}\t{logprobs_elem_1}")
82 +
83 + assert logprobs_elem_0 is not None, fail_msg
84 + assert logprobs_elem_1 is not None, fail_msg
85 + assert output_id_0 in logprobs_elem_1, fail_msg
86 + assert output_id_1 in logprobs_elem_0, fail_msg
87 +
88 + if warn_on_mismatch:
89 + with warnings.catch_warnings():
90 + # This ensures that repeated warnings are shown
91 + # in the output, not just the first occurrence
92 + warnings.simplefilter("always")
93 +
94 + warnings.warn(fail_msg, stacklevel=2)
95
96 # Break out since sequences will now diverge.
97 break
98 + else:
99 + if output_str_0 != output_str_1 and warn_on_mismatch:
100 + # The token outputs exactly match,
101 + # so the text outputs should exactly match as well
102 + fail_msg = (f"Test{prompt_idx}:"
103 + f"\n{name_0}:\t{output_str_0!r}"
104 + f"\n{name_1}:\t{output_str_1!r}")
105 +
106 + with warnings.catch_warnings():
107 + # This ensures that repeated warnings are shown
108 + # in the output, not just the first occurrence
109 + warnings.simplefilter("always")
110 +
111 + warnings.warn(fail_msg, stacklevel=2)
tests/multimodal/test_mapper.py
+16 -47
@@ -4,12 +4,12 @@ from transformers import CLIPImageProcessor, LlavaNextImageProcessor
4
5 from vllm.config import ModelConfig
6 from vllm.multimodal import MULTIMODAL_REGISTRY
7 -
8 -from ..conftest import _STR_DTYPE_TO_TORCH_DTYPE
7 +from vllm.multimodal.utils import rescale_image_size
8
9
10 @pytest.mark.parametrize("dtype", ["half", "float"])
12 -def test_clip_image_processor(image_assets, dtype):
11 +@pytest.mark.parametrize("size_factor", [0.25, 0.5, 1.0])
12 +def test_clip_image_processor(image_assets, dtype, size_factor):
13 MODEL_NAME = "llava-hf/llava-1.5-7b-hf"
14
15 hf_processor = CLIPImageProcessor.from_pretrained(MODEL_NAME)
@@ -26,13 +26,15 @@ def test_clip_image_processor(image_assets, dtype):
26 )
27
28 for asset in image_assets:
29 + image = rescale_image_size(asset.pil_image, size_factor)
30 +
31 hf_result = hf_processor.preprocess(
30 - asset.pil_image,
32 + image,
33 return_tensors="pt",
32 - ).to(dtype=_STR_DTYPE_TO_TORCH_DTYPE[dtype])
34 + )
35 vllm_result = MULTIMODAL_REGISTRY.map_input(
36 model_config,
35 - {"image": asset.pil_image},
37 + {"image": image},
38 )
39
40 assert hf_result.keys() == vllm_result.keys()
@@ -44,12 +46,10 @@ def test_clip_image_processor(image_assets, dtype):
46 assert np.allclose(hf_arr, vllm_arr), f"Failed for key={key}"
47
48
47 -@pytest.mark.xfail(
48 - reason="Inconsistent image processor being used due to lack "
49 - "of support for dynamic image token replacement")
49 @pytest.mark.parametrize("dtype", ["half", "float"])
51 -def test_llava_next_image_processor(image_assets, dtype):
52 - MODEL_NAME = "llava-hf/llava-v1.6-34b-hf"
50 +@pytest.mark.parametrize("size_factor", [0.25, 0.5, 1.0])
51 +def test_llava_next_image_processor(image_assets, dtype, size_factor):
52 + MODEL_NAME = "llava-hf/llava-v1.6-vicuna-7b-hf"
53
54 hf_processor = LlavaNextImageProcessor.from_pretrained(MODEL_NAME)
55 assert isinstance(hf_processor, LlavaNextImageProcessor)
@@ -65,13 +65,15 @@ def test_llava_next_image_processor(image_assets, dtype):
65 )
66
67 for asset in image_assets:
68 + image = rescale_image_size(asset.pil_image, size_factor)
69 +
70 hf_result = hf_processor.preprocess(
69 - asset.pil_image,
71 + image,
72 return_tensors="pt",
71 - ).to(dtype=_STR_DTYPE_TO_TORCH_DTYPE[dtype])
73 + )
74 vllm_result = MULTIMODAL_REGISTRY.map_input(
75 model_config,
74 - {"image": asset.pil_image},
76 + {"image": image},
77 )
78
79 assert hf_result.keys() == vllm_result.keys()
@@ -81,36 +83,3 @@ def test_llava_next_image_processor(image_assets, dtype):
83
84 assert hf_arr.shape == vllm_arr.shape, f"Failed for key={key}"
85 assert np.allclose(hf_arr, vllm_arr), f"Failed for key={key}"
84 -
85 -
86 -@pytest.mark.xfail(
87 - reason="Example image pixels were not processed using HuggingFace")
88 -@pytest.mark.parametrize("dtype", ["float"])
89 -def test_image_pixel_types(image_assets, dtype):
90 - MODEL_NAME = "llava-hf/llava-1.5-7b-hf"
91 -
92 - model_config = ModelConfig(
93 - model=MODEL_NAME,
94 - tokenizer=MODEL_NAME,
95 - tokenizer_mode="auto",
96 - trust_remote_code=False,
97 - seed=0,
98 - dtype=dtype,
99 - revision=None,
100 - )
101 - for asset in image_assets:
102 - image_result = MULTIMODAL_REGISTRY.map_input(
103 - model_config,
104 - {"image": asset.pil_image},
105 - )
106 - tensor_result = MULTIMODAL_REGISTRY.map_input(
107 - model_config,
108 - {"image": asset.pil_image},
109 - )
110 -
111 - assert image_result.keys() == tensor_result.keys()
112 - for key, image_arr in image_result.items():
113 - tensor_arr: np.ndarray = tensor_result[key].numpy()
114 -
115 - assert image_arr.shape == tensor_arr.shape, f"Failed for key={key}"
116 - assert np.allclose(image_arr, tensor_arr), f"Failed for key={key}"
tests/multimodal/test_utils.py
+18 -11
@@ -5,10 +5,9 @@ from typing import Dict, Tuple
5
6 import numpy as np
7 import pytest
8 -import pytest_asyncio
8 from PIL import Image
9
11 -from vllm.multimodal.utils import ImageFetchAiohttp
10 +from vllm.multimodal.utils import ImageFetchAiohttp, fetch_image
11
12 # Test different image extensions (JPG/PNG) and formats (gray/RGB/RGBA)
13 TEST_IMAGE_URLS = [
@@ -19,12 +18,9 @@ TEST_IMAGE_URLS = [
18 ]
19
20
22 -@pytest_asyncio.fixture(scope="session")
23 -async def url_images() -> Dict[str, Image.Image]:
24 - return {
25 - image_url: await ImageFetchAiohttp.fetch_image(image_url)
26 - for image_url in TEST_IMAGE_URLS
27 - }
21 +@pytest.fixture(scope="module")
22 +def url_images() -> Dict[str, Image.Image]:
23 + return {image_url: fetch_image(image_url) for image_url in TEST_IMAGE_URLS}
24
25
26 def get_supported_suffixes() -> Tuple[str, ...]:
@@ -41,7 +37,15 @@ def _image_equals(a: Image.Image, b: Image.Image) -> bool:
37 return (np.asarray(a) == np.asarray(b.convert(a.mode))).all()
38
39
44 -@pytest.mark.asyncio
40 +@pytest.mark.asyncio(scope="module")
41 +@pytest.mark.parametrize("image_url", TEST_IMAGE_URLS)
42 +async def test_fetch_image_http(image_url: str):
43 + image_sync = fetch_image(image_url)
44 + image_async = await ImageFetchAiohttp.fetch_image(image_url)
45 + assert _image_equals(image_sync, image_async)
46 +
47 +
48 +@pytest.mark.asyncio(scope="module")
49 @pytest.mark.parametrize("image_url", TEST_IMAGE_URLS)
50 @pytest.mark.parametrize("suffix", get_supported_suffixes())
51 async def test_fetch_image_base64(url_images: Dict[str, Image.Image],
@@ -68,8 +72,11 @@ async def test_fetch_image_base64(url_images: Dict[str, Image.Image],
72 base64_image = base64.b64encode(f.read()).decode("utf-8")
73 data_url = f"data:{mime_type};base64,{base64_image}"
74
71 - data_image = await ImageFetchAiohttp.fetch_image(data_url)
75 + data_image_sync = fetch_image(data_url)
76 if _image_equals(url_image, Image.open(f)):
73 - assert _image_equals(url_image, data_image)
77 + assert _image_equals(url_image, data_image_sync)
78 else:
79 pass # Lossy format; only check that image can be opened
80 +
81 + data_image_async = await ImageFetchAiohttp.fetch_image(data_url)
82 + assert _image_equals(data_image_sync, data_image_async)
vllm/config.py
+1 -11
@@ -5,7 +5,7 @@ from typing import (TYPE_CHECKING, Any, ClassVar, Dict, List, Optional, Tuple,
5 Union)
6
7 import torch
8 -from transformers import PretrainedConfig, PreTrainedTokenizerBase
8 +from transformers import PretrainedConfig
9
10 import vllm.envs as envs
11 from vllm.logger import init_logger
@@ -1303,16 +1303,6 @@ class VisionLanguageConfig:
1303 image_input_shape: tuple
1304 image_feature_size: int
1305
1306 - #TODO(ywang96): make this a cached property once we refactor the
1307 - # VisionLanguageConfig class.
1308 - def get_image_token_text(
1309 - self, tokenizer: PreTrainedTokenizerBase) -> Tuple[str, str]:
1310 - """Get the image token placeholder text to be inserted into the
1311 - text prompt and the string representation of the image token id.
1312 - """
1313 - image_token_str = tokenizer.decode(self.image_token_id)
1314 - return image_token_str * self.image_feature_size, image_token_str
1315 -
1306 def as_cli_args_dict(self) -> Dict[str, Any]:
1307 """Flatten vision language config to pure args.
1308
vllm/entrypoints/openai/serving_chat.py
+51 -46
@@ -1,6 +1,7 @@
1 import codecs
2 import time
3 from dataclasses import dataclass, field
4 +from functools import cached_property
5 from typing import (AsyncGenerator, AsyncIterator, Awaitable, Dict, Iterable,
6 List, Optional)
7 from typing import Sequence as GenericSequence
@@ -10,7 +11,7 @@ from fastapi import Request
11 from openai.types.chat import (ChatCompletionContentPartImageParam,
12 ChatCompletionContentPartTextParam)
13
13 -from vllm.config import ModelConfig, VisionLanguageConfig
14 +from vllm.config import ModelConfig
15 from vllm.engine.async_llm_engine import AsyncLLMEngine
16 from vllm.entrypoints.openai.protocol import (
17 ChatCompletionContentPartParam, ChatCompletionLogProb,
@@ -27,8 +28,7 @@ from vllm.logger import init_logger
28 from vllm.model_executor.guided_decoding import (
29 get_guided_decoding_logits_processor)
30 from vllm.multimodal import MultiModalDataDict
30 -from vllm.multimodal.utils import (async_get_and_parse_image,
31 - get_full_image_text_prompt)
31 +from vllm.multimodal.utils import async_get_and_parse_image
32 from vllm.outputs import RequestOutput
33 from vllm.sequence import Logprob
34 from vllm.tracing import (contains_trace_headers, extract_trace_headers,
@@ -97,6 +97,36 @@ class OpenAIServingChat(OpenAIServing):
97 logger.warning(
98 "No chat template provided. Chat API will not work.")
99
100 + @cached_property
101 + def image_token_str(self) -> Optional[str]:
102 + # TODO: Let user specify how to insert image tokens into prompt
103 + # (similar to chat template)
104 + model_type = self.model_config.hf_config.model_type
105 + if model_type == "phi3_v":
106 + # Workaround since this token is not defined in the tokenizer
107 + return "<|image_1|>"
108 + if model_type in ("blip-2", "chatglm", "fuyu", "minicpmv",
109 + "paligemma"):
110 + # These models do not use image tokens in the prompt
111 + return None
112 +
113 + # The default behaviour assumes that the image token is
114 + # available to the tokenizer.
115 + # (Suitable for LLaVA, Idefics2, DeepSeek-VL)
116 + vlm_config = self.model_config.multimodal_config
117 + if vlm_config is None:
118 + raise ValueError(
119 + "'image_url' input is not supported as the loaded "
120 + "model is not multimodal.")
121 +
122 + image_token_id = vlm_config.image_token_id
123 + if vlm_config.image_token_id is None:
124 + raise ValueError(
125 + "'image_url' input is not supported as the loaded "
126 + "model does not specify an image token.")
127 +
128 + return self.tokenizer.decode(image_token_id)
129 +
130 def _parse_chat_message_content_parts(
131 self,
132 role: str,
@@ -105,21 +135,26 @@ class OpenAIServingChat(OpenAIServing):
135 texts: List[str] = []
136 mm_futures: List[Awaitable[MultiModalDataDict]] = []
137
108 - vlm_config: Optional[VisionLanguageConfig] = getattr(
109 - self.engine.engine, "vision_language_config", None)
110 - model_config = getattr(self.engine.engine, "model_config", None)
111 -
138 for part in parts:
139 part_type = part["type"]
140 if part_type == "text":
141 text = cast(ChatCompletionContentPartTextParam, part)["text"]
142 texts.append(text)
143 elif part_type == "image_url":
118 - if vlm_config is None:
119 - raise ValueError(
120 - "'image_url' input is not supported as the loaded "
121 - "model is not multimodal.")
122 - assert self.tokenizer is not None
144 + if len(mm_futures) > 0:
145 + raise NotImplementedError(
146 + "Multiple 'image_url' input is currently not supported."
147 + )
148 +
149 + image_token_str = self.image_token_str
150 + if image_token_str is not None:
151 + if any(image_token_str in text for text in texts):
152 + logger.warning(
153 + "Detected image token string in the text prompt. "
154 + "Skipping prompt formatting.")
155 + else:
156 + texts.append(image_token_str)
157 +
158 image_url = cast(ChatCompletionContentPartImageParam,
159 part)["image_url"]
160
@@ -128,43 +163,13 @@ class OpenAIServingChat(OpenAIServing):
163 "'image_url.detail' is currently not supported and "
164 "will be ignored.")
165
131 - mm_future = async_get_and_parse_image(image_url["url"])
132 - mm_futures.append(mm_future)
133 -
166 + image_future = async_get_and_parse_image(image_url["url"])
167 + mm_futures.append(image_future)
168 else:
169 raise NotImplementedError(f"Unknown part type: {part_type}")
170
171 text_prompt = "\n".join(texts)
138 -
139 - if vlm_config is not None and len(mm_futures):
140 -
141 - assert len(
142 - mm_futures
143 - ) == 1, "Multiple 'image_url' input is currently not supported."
144 - (image_token_prompt,
145 - image_token_str) = vlm_config.get_image_token_text(self.tokenizer)
146 -
147 - # NOTE: If image token string (e.g, <image>) is already present
148 - # in the text prompt, we assume it follows the same format required
149 - # by the engine.
150 - if image_token_str in text_prompt:
151 - logger.warning(
152 - "Detected image token string in the text prompt. "
153 - "Skipping prompt formatting.")
154 - messages = [
155 - ConversationMessage(role=role, content=text_prompt)
156 - ]
157 -
158 - else:
159 - full_prompt = get_full_image_text_prompt(
160 - image_prompt=image_token_prompt,
161 - text_prompt=text_prompt,
162 - config=model_config)
163 - messages = [
164 - ConversationMessage(role=role, content=full_prompt)
165 - ]
166 - else:
167 - messages = [ConversationMessage(role=role, content=text_prompt)]
172 + messages = [ConversationMessage(role=role, content=text_prompt)]
173
174 return ChatMessageParseResult(messages=messages, mm_futures=mm_futures)
175
@@ -267,7 +272,7 @@ class OpenAIServingChat(OpenAIServing):
272 "prompt": prompt_text,
273 "prompt_token_ids": prompt_ids,
274 }
270 - if mm_data is not None:
275 + if mm_data:
276 inputs["multi_modal_data"] = mm_data
277
278 is_tracing_enabled = await self.engine.is_tracing_enabled()
vllm/entrypoints/openai/serving_engine.py
+1
@@ -36,6 +36,7 @@ class OpenAIServing:
36 super().__init__()
37
38 self.engine = engine
39 + self.model_config = model_config
40 self.max_model_len = model_config.max_model_len
41
42 # A separate tokenizer to map token IDs to strings.
vllm/inputs/registry.py
+2 -1
@@ -140,7 +140,8 @@ class InputRegistry:
140
141 The model is identified by ``model_config``.
142
143 - TODO: Add guide [ref: PR #5276]
143 + See also:
144 + :ref:`adding_a_new_multimodal_model`
145 """
146 # Avoid circular import
147 from vllm.model_executor.model_loader import get_model_architecture
vllm/model_executor/models/clip.py
+37
@@ -8,10 +8,14 @@ from PIL import Image
8 from transformers import CLIPVisionConfig
9 from transformers.models.clip.modeling_clip import CLIPAttention
10
11 +from vllm.config import ModelConfig
12 +from vllm.inputs import LLMInputs
13 from vllm.model_executor.layers.activation import get_act_fn
14 from vllm.model_executor.layers.linear import (ColumnParallelLinear,
15 RowParallelLinear)
16 from vllm.model_executor.layers.quantization import QuantizationConfig
17 +from vllm.multimodal.image import (cached_get_tokenizer,
18 + repeat_and_pad_image_tokens)
19 from vllm.sequence import SequenceData
20
21
@@ -64,6 +68,39 @@ def dummy_image_for_clip(
68 return {"image": image}
69
70
71 +def input_processor_for_clip(
72 + model_config: ModelConfig,
73 + hf_config: CLIPVisionConfig,
74 + llm_inputs: LLMInputs,
75 + *,
76 + image_token_id: int,
77 + image_feature_size_override: Optional[int] = None,
78 +):
79 + multi_modal_data = llm_inputs.get("multi_modal_data")
80 + if multi_modal_data is None or "image" not in multi_modal_data:
81 + return llm_inputs
82 +
83 + tokenizer = cached_get_tokenizer(model_config.tokenizer)
84 +
85 + if image_feature_size_override is None:
86 + image_feature_size = get_clip_image_feature_size(hf_config)
87 + else:
88 + image_feature_size = image_feature_size_override
89 +
90 + new_prompt, new_token_ids = repeat_and_pad_image_tokens(
91 + tokenizer,
92 + llm_inputs.get("prompt"),
93 + llm_inputs["prompt_token_ids"],
94 + image_token_id=image_token_id,
95 + repeat_count=image_feature_size,
96 + )
97 +
98 + # NOTE: Create a defensive copy of the original inputs
99 + return LLMInputs(prompt_token_ids=new_token_ids,
100 + prompt=new_prompt,
101 + multi_modal_data=multi_modal_data)
102 +
103 +
104 # Adapted from https://github.com/huggingface/transformers/blob/v4.39.0/src/transformers/models/clip/modeling_clip.py#L164 # noqa
105 class CLIPVisionEmbeddings(nn.Module):
106
vllm/model_executor/models/llava.py
+27 -22
@@ -6,7 +6,7 @@ from transformers import CLIPVisionConfig, LlavaConfig
6
7 from vllm.attention import AttentionMetadata
8 from vllm.config import CacheConfig, VisionLanguageConfig
9 -from vllm.inputs import INPUT_REGISTRY, InputContext
9 +from vllm.inputs import INPUT_REGISTRY, InputContext, LLMInputs
10 from vllm.model_executor.layers.activation import get_act_fn
11 from vllm.model_executor.layers.logits_processor import LogitsProcessor
12 from vllm.model_executor.layers.quantization.base_config import (
@@ -20,8 +20,10 @@ from vllm.model_executor.sampling_metadata import SamplingMetadata
20 from vllm.multimodal import MULTIMODAL_REGISTRY
21 from vllm.sequence import IntermediateTensors, SamplerOutput
22
23 -from .clip import dummy_image_for_clip, dummy_seq_data_for_clip
23 +from .clip import (dummy_image_for_clip, dummy_seq_data_for_clip,
24 + input_processor_for_clip)
25 from .interfaces import SupportsVision
26 +from .utils import merge_vision_embeddings
27
28 _KEYS_TO_MODIFY_MAPPING = {
29 "language_model.lm_head": "lm_head",
@@ -51,28 +53,10 @@ class LlavaMultiModalProjector(nn.Module):
53 return hidden_states
54
55
54 -def merge_vision_embeddings(input_ids: torch.Tensor,
55 - inputs_embeds: torch.Tensor,
56 - vision_embeddings: torch.Tensor,
57 - image_token_id: int) -> torch.Tensor:
58 - """In place merges in vision_embeddings with inputs_embeds."""
59 - mask = (input_ids == image_token_id)
60 -
61 - image_feature_size = vision_embeddings.shape[0] * vision_embeddings.shape[1]
62 - if mask.sum() != image_feature_size:
63 - raise ValueError(f"image_feature_size should be {image_feature_size}, "
64 - f"but found: {mask.sum()}")
65 -
66 - inputs_embeds[mask] = vision_embeddings.view(image_feature_size,
67 - vision_embeddings.shape[-1])
68 -
69 - return inputs_embeds
70 -
71 -
56 class LlavaImagePixelInputs(TypedDict):
57 type: Literal["pixel_values"]
58 data: torch.Tensor
75 - """Shape: (batch_size, num_channels, height, width)"""
59 + """Shape: `(batch_size, num_channels, height, width)`"""
60
61
62 LlavaImageInputs = LlavaImagePixelInputs
@@ -96,8 +80,30 @@ def dummy_data_for_llava(ctx: InputContext, seq_len: int):
80 raise NotImplementedError(msg)
81
82
83 +def input_processor_for_llava(ctx: InputContext, llm_inputs: LLMInputs):
84 + multi_modal_data = llm_inputs.get("multi_modal_data")
85 + if multi_modal_data is None or "image" not in multi_modal_data:
86 + return llm_inputs
87 +
88 + model_config = ctx.model_config
89 + hf_config = ctx.get_hf_config(LlavaConfig)
90 + vision_config = hf_config.vision_config
91 +
92 + if isinstance(vision_config, CLIPVisionConfig):
93 + return input_processor_for_clip(
94 + model_config,
95 + vision_config,
96 + llm_inputs,
97 + image_token_id=hf_config.image_token_index,
98 + )
99 +
100 + msg = f"Unsupported vision config: {type(vision_config)}"
101 + raise NotImplementedError(msg)
102 +
103 +
104 @MULTIMODAL_REGISTRY.register_image_input_mapper()
105 @INPUT_REGISTRY.register_dummy_data(dummy_data_for_llava)
106 +@INPUT_REGISTRY.register_input_processor(input_processor_for_llava)
107 class LlavaForConditionalGeneration(nn.Module, SupportsVision):
108
109 def __init__(self,
@@ -112,7 +118,6 @@ class LlavaForConditionalGeneration(nn.Module, SupportsVision):
118
119 # TODO: Optionally initializes this for supporting embeddings.
120 self.vision_tower = CLIPVisionModel(config.vision_config)
115 -
121 self.multi_modal_projector = LlavaMultiModalProjector(
122 vision_hidden_size=config.vision_config.hidden_size,
123 text_hidden_size=config.text_config.hidden_size,
vllm/model_executor/models/llava_next.py
+108 -76
@@ -1,4 +1,4 @@
1 -from typing import Dict, Iterable, List, Literal, Optional, Tuple, TypedDict
1 +from typing import Iterable, List, Literal, Optional, Tuple, TypedDict
2
3 import torch
4 import torch.nn as nn
@@ -10,7 +10,7 @@ from typing_extensions import NotRequired
10
11 from vllm.attention import AttentionMetadata
12 from vllm.config import CacheConfig, VisionLanguageConfig
13 -from vllm.inputs import INPUT_REGISTRY, InputContext
13 +from vllm.inputs import INPUT_REGISTRY, InputContext, LLMInputs
14 from vllm.logger import init_logger
15 from vllm.model_executor.layers.logits_processor import LogitsProcessor
16 from vllm.model_executor.layers.quantization.base_config import (
@@ -21,13 +21,14 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
21 from vllm.model_executor.models.clip import CLIPVisionModel
22 from vllm.model_executor.models.llama import LlamaModel
23 from vllm.model_executor.sampling_metadata import SamplingMetadata
24 -from vllm.multimodal import MULTIMODAL_REGISTRY
24 +from vllm.multimodal import MULTIMODAL_REGISTRY, BatchedTensors
25 from vllm.sequence import IntermediateTensors, SamplerOutput
26
27 from .clip import (dummy_image_for_clip, dummy_seq_data_for_clip,
28 - get_clip_patch_grid_length)
28 + get_clip_patch_grid_length, input_processor_for_clip)
29 from .interfaces import SupportsVision
30 -from .llava import LlavaMultiModalProjector, merge_vision_embeddings
30 +from .llava import LlavaMultiModalProjector
31 +from .utils import merge_vision_embeddings
32
33 logger = init_logger(__name__)
34
@@ -39,16 +40,27 @@ _KEYS_TO_MODIFY_MAPPING = {
40
41 class LlavaNextImagePixelInputs(TypedDict):
42 type: Literal["pixel_values"]
42 - data: torch.Tensor
43 - """Shape: (batch_size, 1 + num_patches, num_channels, height, width)"""
43 + data: BatchedTensors
44 + """
45 + Shape: `(batch_size, 1 + num_patches, num_channels, height, width)`
46 +
47 + Note that `num_patches` may be different for each batch.
48 + """
49
50 image_sizes: NotRequired[torch.Tensor]
46 - """Shape: (batch_size, 2)"""
51 + """
52 + Shape: `(batch_size, 2)`
53 +
54 + This should be in `(height, width)` format.
55 + """
56
57
58 LlavaNextImageInputs = LlavaNextImagePixelInputs
59
60
61 +# Taken from: https://github.com/huggingface/text-generation-inference/blob/v2.0.4/server/text_generation_server/models/vlm_causal_lm.py#L91
62 +# NOTE: new_height and new_width are further incremented to properly invert the
63 +# floordiv operation: https://github.com/huggingface/transformers/blob/v4.42.2/src/transformers/models/llava_next/modeling_llava_next.py#L133
64 def _get_llava_next_num_unpadded_features(
65 height: int,
66 width: int,
@@ -56,7 +68,6 @@ def _get_llava_next_num_unpadded_features(
68 num_patch_height: int,
69 num_patch_width: int,
70 ) -> Tuple[int, int]:
59 - # Taken from: https://github.com/huggingface/text-generation-inference/blob/799a193b109662743bed1b18a09af1fdcd508c8b/server/text_generation_server/models/vlm_causal_lm.py#L111
71 current_height = npatches * num_patch_height
72 current_width = npatches * num_patch_width
73
@@ -64,9 +75,13 @@ def _get_llava_next_num_unpadded_features(
75 current_aspect_ratio: float = current_width / current_height
76 if aspect_ratio > current_aspect_ratio:
77 new_height = (height * current_width) // width
78 + if new_height % 2 == 1:
79 + new_height += 1
80 current_height = new_height
81 else:
82 new_width = (width * current_height) // height
83 + if new_width % 2 == 1:
84 + new_width += 1
85 current_width = new_width
86
87 unpadded_features = current_height * current_width
@@ -74,7 +89,8 @@ def _get_llava_next_num_unpadded_features(
89 return (unpadded_features, newline_features)
90
91
77 -def _get_llava_next_image_feature_size(
92 +# Based on: https://github.com/huggingface/text-generation-inference/blob/v2.0.4/server/text_generation_server/models/vlm_causal_lm.py#L111
93 +def get_llava_next_image_feature_size(
94 hf_config: LlavaNextConfig,
95 *,
96 input_height: int,
@@ -89,7 +105,9 @@ def _get_llava_next_image_feature_size(
105 )
106 base_feature_size = num_patches * num_patches
107
92 - num_patch_height, num_patch_width = get_anyres_image_grid_shape(
108 + # Note: We follow the "wrong" width/height order
109 + # [ref: PR huggingface/transformers#31588]
110 + num_patch_width, num_patch_height = get_anyres_image_grid_shape(
111 image_size=(input_height, input_width),
112 grid_pinpoints=hf_config.image_grid_pinpoints,
113 patch_size=vision_config.image_size,
@@ -110,14 +128,16 @@ def _get_llava_next_image_feature_size(
128
129
130 def dummy_data_for_llava_next(ctx: InputContext, seq_len: int):
113 - multimodal_config = ctx.get_multimodal_config()
131 hf_config = ctx.get_hf_config(LlavaNextConfig)
132 vision_config = hf_config.vision_config
133
117 - #TODO: change the logic for dummy data to support dynamic shape
118 - _, _, dummy_height, dummy_width = multimodal_config.image_input_shape
119 - image_feature_size = _get_llava_next_image_feature_size(
120 - hf_config, input_height=dummy_height, input_width=dummy_width)
134 + # Result in the max possible feature size (2x2 grid of 336x336px tiles)
135 + dummy_height = dummy_width = 448
136 + image_feature_size = get_llava_next_image_feature_size(
137 + hf_config,
138 + input_height=dummy_height,
139 + input_width=dummy_width,
140 + )
141
142 if isinstance(vision_config, CLIPVisionConfig):
143 seq_data = dummy_seq_data_for_clip(
@@ -139,27 +159,47 @@ def dummy_data_for_llava_next(ctx: InputContext, seq_len: int):
159 raise NotImplementedError(msg)
160
161
142 -def _pixel_mapper(ctx: InputContext, image: object) -> Dict[str, torch.Tensor]:
162 +def input_processor_for_llava_next(ctx: InputContext, llm_inputs: LLMInputs):
163 + multi_modal_data = llm_inputs.get("multi_modal_data")
164 + if multi_modal_data is None or "image" not in multi_modal_data:
165 + return llm_inputs
166
144 - if isinstance(image, Image.Image):
167 + model_config = ctx.model_config
168 + hf_config = ctx.get_hf_config(LlavaNextConfig)
169 + vision_config = hf_config.vision_config
170
146 - # Temporary patch before dynamic number of image tokens is supported
147 - _, _, h, w = ctx.get_multimodal_config().image_input_shape
148 - if (w, h) != (image.width, image.height):
149 - logger.warning(
150 - "Dynamic image shape is currently not supported. "
151 - "Resizing input image to (%d, %d).", w, h)
171 + image_data = multi_modal_data["image"]
172 + if isinstance(image_data, Image.Image):
173 + width, height = image_data.size
174 +
175 + image_feature_size = get_llava_next_image_feature_size(
176 + hf_config,
177 + input_height=height,
178 + input_width=width,
179 + )
180 + elif isinstance(image_data, torch.Tensor):
181 + raise NotImplementedError("Embeddings input is not supported yet")
182 + else:
183 + raise TypeError(f"Invalid image type: {type(image_data)}")
184
153 - image = image.resize((w, h))
185 + vision_config = hf_config.vision_config
186
155 - return MULTIMODAL_REGISTRY._get_plugin("image") \
156 - ._default_input_mapper(ctx, image)
187 + if isinstance(vision_config, CLIPVisionConfig):
188 + return input_processor_for_clip(
189 + model_config,
190 + vision_config,
191 + llm_inputs,
192 + image_token_id=hf_config.image_token_index,
193 + image_feature_size_override=image_feature_size,
194 + )
195
158 - raise TypeError(f"Invalid type for 'image': {type(image)}")
196 + msg = f"Unsupported vision config: {type(vision_config)}"
197 + raise NotImplementedError(msg)
198
199
161 -@MULTIMODAL_REGISTRY.register_image_input_mapper(_pixel_mapper)
200 +@MULTIMODAL_REGISTRY.register_image_input_mapper()
201 @INPUT_REGISTRY.register_dummy_data(dummy_data_for_llava_next)
202 +@INPUT_REGISTRY.register_input_processor(input_processor_for_llava_next)
203 class LlavaNextForConditionalGeneration(nn.Module, SupportsVision):
204
205 def __init__(self,
@@ -172,8 +212,8 @@ class LlavaNextForConditionalGeneration(nn.Module, SupportsVision):
212 self.config = config
213 self.vlm_config = vlm_config
214
215 + # TODO: Optionally initializes this for supporting embeddings.
216 self.vision_tower = CLIPVisionModel(config=config.vision_config)
176 -
217 self.multi_modal_projector = LlavaMultiModalProjector(
218 vision_hidden_size=config.vision_config.hidden_size,
219 text_hidden_size=config.text_config.hidden_size,
@@ -196,24 +236,6 @@ class LlavaNextForConditionalGeneration(nn.Module, SupportsVision):
236 self.image_newline = nn.Parameter(
237 torch.empty(config.text_config.hidden_size))
238
199 - def _validate_image_pixels(self, data: torch.Tensor) -> torch.Tensor:
200 - _, num_channels, _, _ = self.vlm_config.image_input_shape
201 -
202 - # Note that this is different from that of vLLM vision_language_config
203 - # since the image is resized by the HuggingFace preprocessor
204 - height = width = self.config.vision_config.image_size
205 -
206 - if list(data.shape[2:]) != [num_channels, height, width]:
207 - raise ValueError(
208 - f"The expected image tensor shape is batch dimension plus "
209 - f"num_patches plus {[num_channels, height, width]}. "
210 - f"You supplied {data.shape}. "
211 - f"If you are using vLLM's entrypoint, make sure your "
212 - f"supplied image input is consistent with "
213 - f"image_input_shape in engine args.")
214 -
215 - return data
216 -
239 def _validate_image_sizes(self, data: torch.Tensor) -> torch.Tensor:
240 if list(data.shape[1:]) != [2]:
241 raise ValueError(
@@ -223,14 +245,14 @@ class LlavaNextForConditionalGeneration(nn.Module, SupportsVision):
245 return data
246
247 def _parse_and_validate_image_input(
226 - self, **kwargs: object) -> Optional[LlavaNextImageInputs]:
248 + self, **kwargs: object) -> Optional[LlavaNextImagePixelInputs]:
249 pixel_values = kwargs.pop("pixel_values", None)
250 image_sizes = kwargs.pop("image_sizes", None)
251
230 - if pixel_values is None or image_sizes is None:
252 + if pixel_values is None:
253 return None
254
233 - if not isinstance(pixel_values, torch.Tensor):
255 + if not isinstance(pixel_values, (torch.Tensor, list)):
256 raise ValueError("Incorrect type of pixel values. "
257 f"Got type: {type(pixel_values)}")
258
@@ -240,7 +262,7 @@ class LlavaNextForConditionalGeneration(nn.Module, SupportsVision):
262
263 return LlavaNextImagePixelInputs(
264 type="pixel_values",
243 - data=self._validate_image_pixels(pixel_values),
265 + data=pixel_values,
266 image_sizes=self._validate_image_sizes(image_sizes),
267 )
268
@@ -267,15 +289,14 @@ class LlavaNextForConditionalGeneration(nn.Module, SupportsVision):
289 strategy=self.config.vision_feature_select_strategy,
290 )
291
292 + # Based on: https://github.com/haotian-liu/LLaVA/blob/main/llava/model/llava_arch.py
293 def _merge_image_patch_embeddings(self, image_size: torch.Tensor,
294 patch_embeddings: torch.Tensor, *,
295 strategy: str) -> torch.Tensor:
273 - # Based on: https://github.com/haotian-liu/LLaVA/blob/main/llava/model/llava_arch.py
296 if strategy == "flat":
297 return patch_embeddings.flatten(0, 1)
298
299 if strategy.startswith("spatial"):
278 - orig_width, orig_height = image_size
300 height = width = self.config.vision_config.image_size \
301 // self.config.vision_config.patch_size
302
@@ -289,13 +310,15 @@ class LlavaNextForConditionalGeneration(nn.Module, SupportsVision):
310 other_patch_embeds = patch_embeddings[1:]
311
312 # image_aspect_ratio == "anyres"
313 + # Note: We follow the "wrong" width/height order
314 + # [ref: PR huggingface/transformers#31588]
315 num_patch_width, num_patch_height = get_anyres_image_grid_shape(
293 - (orig_width, orig_height),
316 + image_size,
317 self.config.image_grid_pinpoints,
318 self.config.vision_config.image_size,
319 )
320 other_patch_embeds = other_patch_embeds \
298 - .view(num_patch_width, num_patch_height, height, width, -1)
321 + .view(num_patch_height, num_patch_width, height, width, -1)
322
323 if "unpad" in strategy:
324 other_patch_embeds = other_patch_embeds \
@@ -333,44 +356,53 @@ class LlavaNextForConditionalGeneration(nn.Module, SupportsVision):
356 raise ValueError(f"Unexpected patch merge strategy: {strategy}")
357
358 def _process_image_pixels(
336 - self, inputs: LlavaNextImagePixelInputs) -> torch.Tensor:
359 + self,
360 + inputs: LlavaNextImagePixelInputs,
361 + ) -> BatchedTensors:
362 assert self.vision_tower is not None
363
364 pixel_values = inputs["data"]
365
341 - b, num_patches, c, h, w = pixel_values.shape
342 - stacked_pixel_values = pixel_values.view(b * num_patches, c, h, w)
366 + if isinstance(pixel_values, torch.Tensor):
367 + b, num_patches, c, h, w = pixel_values.shape
368 + stacked_pixel_values = pixel_values.view(b * num_patches, c, h, w)
369 + stacked_image_features = self._image_pixels_to_features(
370 + self.vision_tower, stacked_pixel_values)
371 + stacked_patch_embeddings = self.multi_modal_projector(
372 + stacked_image_features)
373
374 + return stacked_patch_embeddings.view(
375 + b, num_patches, *stacked_patch_embeddings.shape[1:])
376 +
377 + num_patches_per_batch = [v.shape[0] for v in pixel_values]
378 + stacked_pixel_values = torch.cat(pixel_values)
379 stacked_image_features = self._image_pixels_to_features(
380 self.vision_tower, stacked_pixel_values)
381
347 - return stacked_image_features.view(b, num_patches,
348 - *stacked_image_features.shape[-2:])
382 + return [
383 + self.multi_modal_projector(image_features) for image_features in
384 + torch.split(stacked_image_features, num_patches_per_batch)
385 + ]
386
387 def _process_image_input(
351 - self, image_input: LlavaNextImageInputs) -> torch.Tensor:
352 - assert self.vision_tower is not None
353 - image_features = self._process_image_pixels(image_input)
354 -
355 - patch_embeddings = self.multi_modal_projector(image_features)
388 + self, image_input: LlavaNextImageInputs) -> BatchedTensors:
389 + patch_embeddings = self._process_image_pixels(image_input)
390
391 image_sizes = image_input.get("image_sizes")
392 if image_sizes is None:
359 - batch_size = image_input["data"].shape[0]
393 + batch_size = len(image_input["data"])
394 vision_config = self.config.vision_config
361 - default_width = default_height = vision_config.image_size
362 - image_sizes = torch.as_tensor([[default_width, default_height]
395 + default_height = default_width = vision_config.image_size
396 + image_sizes = torch.as_tensor([[default_height, default_width]
397 for _ in range(batch_size)])
398
365 - merged_patch_embeddings = [
399 + return [
400 self._merge_image_patch_embeddings(image_sizes[i],
367 - patch_features,
401 + patch_features_batch,
402 strategy="spatial_unpad")
369 - for i, patch_features in enumerate(patch_embeddings)
403 + for i, patch_features_batch in enumerate(patch_embeddings)
404 ]
405
372 - return torch.stack(merged_patch_embeddings, dim=0)
373 -
406 def forward(
407 self,
408 input_ids: torch.Tensor,
@@ -404,8 +436,8 @@ class LlavaNextForConditionalGeneration(nn.Module, SupportsVision):
436 input_ids: Flattened (concatenated) input_ids corresponding to a
437 batch.
438 pixel_values: The pixels in each grid patch for each input image.
407 - Expects a batch with shape `[1, num_patches, 3, 336, 336]`.
408 - image_sizes: The original `(width, height)` for each input image.
439 + Expects a batch with shape `[1, num_patches, 3, h, w]`.
440 + image_sizes: The original `(height, width)` for each input image.
441 Expects a batch with shape `[1, 2]`.
442
443 See also:
vllm/model_executor/models/phi3v.py
+157 -65
@@ -13,7 +13,9 @@
13 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 # See the License for the specific language governing permissions and
15 # limitations under the License.
16 -from typing import Dict, Iterable, List, Literal, Optional, Tuple, TypedDict
16 +import re
17 +from functools import lru_cache
18 +from typing import Iterable, List, Literal, Optional, Tuple, TypedDict
19
20 import numpy as np
21 import torch
@@ -22,8 +24,8 @@ from PIL import Image
24 from transformers import CLIPVisionConfig, PretrainedConfig
25
26 from vllm.attention import AttentionMetadata
25 -from vllm.config import CacheConfig, VisionLanguageConfig
26 -from vllm.inputs import INPUT_REGISTRY, InputContext
27 +from vllm.config import CacheConfig, ModelConfig, VisionLanguageConfig
28 +from vllm.inputs import INPUT_REGISTRY, InputContext, LLMInputs
29 from vllm.logger import init_logger
30 from vllm.model_executor.layers.logits_processor import LogitsProcessor
31 from vllm.model_executor.layers.quantization.base_config import (
@@ -34,10 +36,12 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
36 from vllm.model_executor.models.clip import CLIPVisionModel
37 from vllm.model_executor.models.llama import LlamaModel
38 from vllm.model_executor.sampling_metadata import SamplingMetadata
37 -from vllm.multimodal import MULTIMODAL_REGISTRY
39 +from vllm.multimodal import MULTIMODAL_REGISTRY, BatchedTensors
40 +from vllm.multimodal.image import cached_get_tokenizer
41 from vllm.sequence import IntermediateTensors, SamplerOutput
42
40 -from .clip import dummy_image_for_clip, dummy_seq_data_for_clip
43 +from .clip import (dummy_image_for_clip, dummy_seq_data_for_clip,
44 + input_processor_for_clip)
45 from .interfaces import SupportsVision
46
47 logger = init_logger(__name__)
@@ -251,50 +255,22 @@ class Phi3HDImageEmbedding(Phi3ImageEmbeddingBase):
255
256 class Phi3VImagePixelInputs(TypedDict):
257 type: Literal["pixel_values"]
254 - data: torch.Tensor
255 - """Shape: (batch_size, 1 + num_patches, num_channels, height, width)"""
256 -
257 - image_sizes: torch.Tensor
258 - """Shape: (batch_size, 2)"""
259 -
260 -
261 -def _get_phi3v_image_feature_size(
262 - *,
263 - input_height: int,
264 - input_width: int,
265 -) -> int:
266 - h, w = input_height, input_width
267 -
268 - # https://huggingface.co/microsoft/Phi-3-vision-128k-instruct/blob/main/image_processing_phi3_v.py#L178
269 - return (h // 336 * w // 336 + 1) * 144 + 1 + (h // 336 + 1) * 12
258 + data: BatchedTensors
259 + """
260 + Shape: `(batch_size, 1 + num_patches, num_channels, height, width)`
261
262 + Note that `num_patches` may be different for each batch.
263 + """
264
272 -def dummy_data_for_phi3v(ctx: InputContext, seq_len: int):
273 - multimodal_config = ctx.get_multimodal_config()
274 -
275 - #TODO: change the logic for dummy data to support dynamic shape
276 - _, _, dummy_height, dummy_width = multimodal_config.image_input_shape
277 - image_feature_size = _get_phi3v_image_feature_size(
278 - input_height=dummy_height,
279 - input_width=dummy_width,
280 - )
281 -
282 - seq_data = dummy_seq_data_for_clip(
283 - CLIP_VIT_LARGE_PATCH14_336_CONFIG,
284 - seq_len,
285 - image_token_id=32044,
286 - image_feature_size_override=image_feature_size,
287 - )
288 - mm_data = dummy_image_for_clip(
289 - CLIP_VIT_LARGE_PATCH14_336_CONFIG,
290 - image_width_override=dummy_width,
291 - image_height_override=dummy_height,
292 - )
265 + image_sizes: torch.Tensor
266 + """
267 + Shape: `(batch_size, 2)`
268
294 - return seq_data, mm_data
269 + This should be in `(height, width)` format.
270 + """
271
272
297 -# Based on https://huggingface.co/microsoft/Phi-3-vision-128k-instruct/blob/main/image_processing_phi3_v.py
273 +# Based on https://huggingface.co/microsoft/Phi-3-vision-128k-instruct/blob/main/image_processing_phi3_v.py#L57
274 def _calc_padded_size(*, width: int, height: int, padding_unit: int = 336):
275 target_height = int(np.ceil(height / padding_unit) * padding_unit)
276 top_padding = int((target_height - height) / 2)
@@ -304,7 +280,7 @@ def _calc_padded_size(*, width: int, height: int, padding_unit: int = 336):
280 return padded_width, padded_height
281
282
307 -# Based on https://huggingface.co/microsoft/Phi-3-vision-128k-instruct/blob/main/image_processing_phi3_v.py
283 +# Based on https://huggingface.co/microsoft/Phi-3-vision-128k-instruct/blob/main/image_processing_phi3_v.py#L90
284 def _calc_hd_transform_size(*, width: int, height: int, hd_num: int = 16):
285 transposed = False
286 if width < height:
@@ -329,27 +305,133 @@ def _calc_hd_transform_size(*, width: int, height: int, hd_num: int = 16):
305 return padded_width, padded_height
306
307
332 -def _image_processor(ctx: InputContext,
333 - image: object) -> Dict[str, torch.Tensor]:
308 +# Based on https://huggingface.co/microsoft/Phi-3-vision-128k-instruct/blob/main/image_processing_phi3_v.py#L181
309 +def get_phi3v_image_feature_size(
310 + hf_config: PretrainedConfig,
311 + *,
312 + input_height: int,
313 + input_width: int,
314 +) -> int:
315 + num_crops = getattr(hf_config, "num_crops", 16)
316 + new_width, new_height = _calc_hd_transform_size(width=input_width,
317 + height=input_height,
318 + hd_num=num_crops)
319
335 - if isinstance(image, Image.Image):
336 - # Temporary patch before dynamic number of image tokens is supported
337 - _, _, h, w = ctx.get_multimodal_config().image_input_shape
338 - if (w, h) != _calc_hd_transform_size(width=image.width,
339 - height=image.height):
340 - logger.warning(
341 - "Dynamic image shape is currently not supported. "
342 - "Resizing input image to (%d, %d).", w, h)
320 + return (new_height // 336 * new_width // 336 + 1) * 144 + 1 \
321 + + (new_height // 336 + 1) * 12
322
344 - image = image.resize((w, h))
323
346 - return MULTIMODAL_REGISTRY._get_plugin("image") \
347 - ._default_input_mapper(ctx, image)
348 - raise TypeError(f"Invalid type for 'image': {type(image)}")
324 +def dummy_data_for_phi3v(ctx: InputContext, seq_len: int):
325 + # Result in the max possible feature size (h:w = 16:1)
326 + dummy_height, dummy_width = 8000, 50
327 + image_feature_size = get_phi3v_image_feature_size(
328 + ctx.get_hf_config(PretrainedConfig),
329 + input_height=dummy_height,
330 + input_width=dummy_width,
331 + )
332 +
333 + seq_data = dummy_seq_data_for_clip(
334 + CLIP_VIT_LARGE_PATCH14_336_CONFIG,
335 + seq_len,
336 + image_token_id=32044,
337 + image_feature_size_override=image_feature_size,
338 + )
339 + mm_data = dummy_image_for_clip(
340 + CLIP_VIT_LARGE_PATCH14_336_CONFIG,
341 + image_width_override=dummy_width,
342 + image_height_override=dummy_height,
343 + )
344 +
345 + return seq_data, mm_data
346 +
347
348 +# Reserve this function to also handle placeholders for additional images
349 +# [ref: PR #5820]
350 +@lru_cache
351 +def _get_image_placeholder_token_ids(model_config: ModelConfig,
352 + idx: int) -> List[int]:
353 + assert idx > 0
354
351 -@MULTIMODAL_REGISTRY.register_image_input_mapper(_image_processor)
355 + tokenizer = cached_get_tokenizer(model_config.tokenizer)
356 +
357 + # We need to get the token for "<", not "▁<"
358 + # https://huggingface.co/microsoft/Phi-3-vision-128k-instruct/raw/main/tokenizer.json
359 + a_token_id, = tokenizer.encode("a", add_special_tokens=False)
360 + a_token_id_, *image_placeholder_token_ids = tokenizer.encode(
361 + f"a<|image_{idx}|>", add_special_tokens=False)
362 + assert a_token_id == a_token_id_
363 +
364 + return image_placeholder_token_ids
365 +
366 +
367 +def input_processor_for_phi3v(ctx: InputContext, llm_inputs: LLMInputs):
368 + multi_modal_data = llm_inputs.get("multi_modal_data")
369 + if multi_modal_data is None or "image" not in multi_modal_data:
370 + return llm_inputs
371 +
372 + model_config = ctx.model_config
373 + multimodal_config = ctx.get_multimodal_config()
374 + hf_config = ctx.get_hf_config(PretrainedConfig)
375 +
376 + image_data = multi_modal_data["image"]
377 + if isinstance(image_data, Image.Image):
378 + w, h = image_data.size
379 + w, h = _calc_hd_transform_size(width=w, height=h)
380 +
381 + image_feature_size = get_phi3v_image_feature_size(hf_config,
382 + input_width=w,
383 + input_height=h)
384 + elif isinstance(image_data, torch.Tensor):
385 + raise NotImplementedError("Embeddings input is not supported yet")
386 + else:
387 + raise TypeError(f"Invalid image type: {type(image_data)}")
388 +
389 + prompt = llm_inputs.get("prompt")
390 + if prompt is None:
391 + new_prompt = None
392 + else:
393 + if prompt.count("<|image|>") > 0:
394 + logger.warning("Please follow the prompt format that is "
395 + "documented on HuggingFace which does not involve "
396 + "repeating <|image|> tokens.")
397 + elif len(re.findall(r"(<\|image_\d+\|>)+", prompt)) > 1:
398 + logger.warning("Multiple image input is not supported yet, "
399 + "so any extra image tokens will be treated "
400 + "as plain text.")
401 +
402 + new_prompt = prompt
403 +
404 + prompt_token_ids = llm_inputs["prompt_token_ids"]
405 + image_1_token_ids = _get_image_placeholder_token_ids(model_config, idx=1)
406 +
407 + new_token_ids: List[int] = []
408 + for i in range(len(prompt_token_ids) - len(image_1_token_ids) + 1):
409 + if prompt_token_ids[i:i + len(image_1_token_ids)] == image_1_token_ids:
410 + new_token_ids.append(multimodal_config.image_token_id)
411 +
412 + # No need to further scan the list since we only replace once
413 + new_token_ids.extend(prompt_token_ids[i + len(image_1_token_ids):])
414 + break
415 + else:
416 + new_token_ids.append(prompt_token_ids[i])
417 +
418 + # NOTE: Create a defensive copy of the original inputs
419 + llm_inputs = LLMInputs(prompt_token_ids=new_token_ids,
420 + prompt=new_prompt,
421 + multi_modal_data=multi_modal_data)
422 +
423 + return input_processor_for_clip(
424 + model_config,
425 + CLIP_VIT_LARGE_PATCH14_336_CONFIG,
426 + llm_inputs,
427 + image_token_id=multimodal_config.image_token_id,
428 + image_feature_size_override=image_feature_size,
429 + )
430 +
431 +
432 +@MULTIMODAL_REGISTRY.register_image_input_mapper()
433 @INPUT_REGISTRY.register_dummy_data(dummy_data_for_phi3v)
434 +@INPUT_REGISTRY.register_input_processor(input_processor_for_phi3v)
435 class Phi3VForCausalLM(nn.Module, SupportsVision):
436
437 def __init__(self,
@@ -363,6 +445,8 @@ class Phi3VForCausalLM(nn.Module, SupportsVision):
445 self.vlm_config = vlm_config
446
447 self.model = LlamaModel(config, cache_config, quant_config)
448 +
449 + # TODO: Optionally initializes this for supporting embeddings.
450 self.vision_embed_tokens = Phi3HDImageEmbedding(
451 vlm_config, config, self.model.embed_tokens)
452 self.lm_head = ParallelLMHead(config.vocab_size,
@@ -376,12 +460,20 @@ class Phi3VForCausalLM(nn.Module, SupportsVision):
460 pixel_values = kwargs.pop("pixel_values", None)
461 image_sizes = kwargs.pop("image_sizes", None)
462
379 - if pixel_values is not None and image_sizes is not None:
380 - return Phi3VImagePixelInputs(type="pixel_values",
381 - data=pixel_values,
382 - image_sizes=image_sizes)
463 + if pixel_values is None:
464 + return None
465 +
466 + if not isinstance(pixel_values, (torch.Tensor, list)):
467 + raise ValueError("Incorrect type of pixel values. "
468 + f"Got type: {type(pixel_values)}")
469 +
470 + if not isinstance(image_sizes, torch.Tensor):
471 + raise ValueError("Incorrect type of image sizes. "
472 + f"Got type: {type(image_sizes)}")
473
384 - return None
474 + return Phi3VImagePixelInputs(type="pixel_values",
475 + data=pixel_values,
476 + image_sizes=image_sizes)
477
478 def forward(self,
479 input_ids: torch.Tensor,
vllm/model_executor/models/utils.py new
+41
@@ -0,0 +1,41 @@
1 +import torch
2 +
3 +from vllm.multimodal import BatchedTensors
4 +
5 +
6 +def merge_vision_embeddings(input_ids: torch.Tensor,
7 + inputs_embeds: torch.Tensor,
8 + vision_embeddings: BatchedTensors,
9 + image_token_id: int) -> torch.Tensor:
10 + """
11 + Merge `vision_embeddings` into `inputs_embeds` by overwriting the positions
12 + in `inputs_embeds` corresponding to placeholder image tokens in `input_ids`.
13 +
14 + Note:
15 + This updates `inputs_embeds` in place.
16 + """
17 + mask = (input_ids == image_token_id)
18 + num_expected_tokens = mask.sum()
19 +
20 + if isinstance(vision_embeddings, torch.Tensor):
21 + batch_size, batch_tokens, *_, embed_dim = vision_embeddings.shape
22 + total_tokens = batch_size * batch_tokens
23 + if num_expected_tokens != total_tokens:
24 + expr = f"{batch_size} x {batch_tokens}"
25 + raise ValueError(
26 + f"Attempted to assign {expr} = {total_tokens} "
27 + f"image tokens to {num_expected_tokens} placeholders")
28 +
29 + inputs_embeds[mask] = vision_embeddings.view(total_tokens, embed_dim)
30 + else:
31 + size_per_batch = [t.shape[0] for t in vision_embeddings]
32 + total_tokens = sum(size_per_batch)
33 + if num_expected_tokens != total_tokens:
34 + expr = ' + '.join(map(str, size_per_batch))
35 + raise ValueError(
36 + f"Attempted to assign {expr} = {total_tokens} "
37 + f"image tokens to {num_expected_tokens} placeholders")
38 +
39 + inputs_embeds[mask] = torch.cat(vision_embeddings)
40 +
41 + return inputs_embeds
vllm/multimodal/__init__.py
+5 -2
@@ -1,4 +1,5 @@
1 -from .base import MultiModalDataDict, MultiModalPlugin
1 +from .base import (BatchedTensors, MultiModalDataDict, MultiModalInputs,
2 + MultiModalPlugin)
3 from .registry import MultiModalRegistry
4
5 MULTIMODAL_REGISTRY = MultiModalRegistry()
@@ -11,8 +12,10 @@ See also:
12 """
13
14 __all__ = [
15 + "BatchedTensors",
16 + "MultiModalDataDict",
17 + "MultiModalInputs",
18 "MultiModalPlugin",
19 "MULTIMODAL_REGISTRY",
20 "MultiModalRegistry",
17 - "MultiModalDataDict",
21 ]
vllm/multimodal/base.py
+85 -16
@@ -1,23 +1,90 @@
1 +import sys
2 from abc import ABC, abstractmethod
2 -from typing import (TYPE_CHECKING, Any, Callable, Dict, Optional, Type,
3 - TypedDict, TypeVar, Union)
3 +from collections import UserDict, defaultdict
4 +from typing import (Any, Callable, Dict, List, Optional, Type, TypedDict,
5 + TypeVar, Union)
6 +
7 +import torch
8 +import torch.types
9 +from PIL import Image
10 +from torch import nn
11
12 from vllm.config import ModelConfig
13 from vllm.inputs import InputContext
14 from vllm.logger import init_logger
15
9 -if TYPE_CHECKING:
10 - import torch
11 - from PIL import Image
12 - from torch import nn
13 -
16 logger = init_logger(__name__)
17
16 -N = TypeVar("N", bound=Type["nn.Module"])
18 +BatchedTensors = Union[torch.Tensor, List[torch.Tensor]]
19 +"""
20 +If each input tensor in the batch has the same size, this is a single batched
21 +tensor; otherwise, this is a list of tensors with one element per batch.
22 +"""
23 +
24 +if sys.version_info < (3, 9):
25 + # UserDict cannot be subscripted
26 + class _MultiModalInputsBase(UserDict):
27 + pass
28 +else:
29 +
30 + class _MultiModalInputsBase(UserDict[str, torch.Tensor]):
31 + pass
32 +
33 +
34 +class MultiModalInputs(_MultiModalInputsBase):
35 + """
36 + A dictionary that represents the keyword arguments to
37 + :meth:`~torch.nn.Module.forward`.
38 + """
39 +
40 + @staticmethod
41 + def try_concat(
42 + tensors: List[torch.Tensor],
43 + *,
44 + device: torch.types.Device,
45 + ) -> BatchedTensors:
46 + # Avoid initializing CUDA too early
47 + import torch
48 +
49 + unbatched_shape = tensors[0].shape[1:]
50 +
51 + for tensor in tensors:
52 + if tensor.shape[1:] != unbatched_shape:
53 + return [
54 + tensor.squeeze(0).to(device=device) for tensor in tensors
55 + ]
56 +
57 + return torch.cat(tensors, dim=0).to(device=device)
58 +
59 + @staticmethod
60 + def batch(
61 + inputs_list: List["MultiModalInputs"],
62 + device: torch.types.Device,
63 + ) -> Dict[str, BatchedTensors]:
64 + """Batch multiple inputs together into a dictionary."""
65 + if len(inputs_list) == 0:
66 + return {}
67 +
68 + keys = inputs_list[0].keys()
69 +
70 + item_lists: Dict[str, List[torch.Tensor]] = defaultdict(list)
71 +
72 + for inputs in inputs_list:
73 + if inputs.keys() != keys:
74 + msg = f"Inputs do not share the same keys ({keys})"
75 + raise ValueError(msg)
76 +
77 + for k, v in inputs.items():
78 + item_lists[k].append(v)
79 +
80 + return {
81 + k: MultiModalInputs.try_concat(item_list, device=device)
82 + for k, item_list in item_lists.items()
83 + }
84
85
86 class MultiModalDataBuiltins(TypedDict, total=False):
20 - image: "Image.Image"
87 + image: Image.Image
88
89
90 MultiModalDataDict = Union[MultiModalDataBuiltins, Dict[str, Any]]
@@ -29,12 +96,13 @@ to the model by the corresponding mapper. By default, the mapper of
96 the corresponding plugin with the same modality key is applied.
97 """
98
32 -MultiModalInputMapper = Callable[[InputContext, object], Dict[str,
33 - "torch.Tensor"]]
99 +MultiModalInputMapper = Callable[[InputContext, object], MultiModalInputs]
100 """Return a dictionary to be passed as keyword arguments to
101 :meth:`~torch.nn.Module.forward`. This is similar in concept to tokenizers
102 and processors in HuggingFace Transformers."""
103
104 +N = TypeVar("N", bound=Type[nn.Module])
105 +
106
107 class MultiModalPlugin(ABC):
108 """
@@ -48,8 +116,7 @@ class MultiModalPlugin(ABC):
116 """
117
118 def __init__(self) -> None:
51 - self._input_mappers: Dict[Type["nn.Module"],
52 - MultiModalInputMapper] = {}
119 + self._input_mappers: Dict[Type[nn.Module], MultiModalInputMapper] = {}
120
121 @abstractmethod
122 def get_data_key(self) -> str:
@@ -60,7 +127,7 @@ class MultiModalPlugin(ABC):
127
128 @abstractmethod
129 def _default_input_mapper(self, ctx: InputContext,
63 - data: object) -> Dict[str, "torch.Tensor"]:
130 + data: object) -> MultiModalInputs:
131 """Return a dictionary to be passed as keyword arguments to
132 :meth:`~torch.nn.Module.forward`. This is similar in concept to
133 tokenizers and processors in HuggingFace Transformers.
@@ -80,6 +147,7 @@ class MultiModalPlugin(ABC):
147
148 See also:
149 :ref:`input_processing_pipeline`
150 + :ref:`adding_a_new_multimodal_model`
151 """
152
153 def wrapper(model_cls: N) -> N:
@@ -97,7 +165,7 @@ class MultiModalPlugin(ABC):
165 return wrapper
166
167 def map_input(self, model_config: ModelConfig,
100 - data: object) -> Dict[str, "torch.Tensor"]:
168 + data: object) -> MultiModalInputs:
169 """
170 Apply an input mapper to a data passed
171 to the model, transforming the data into a dictionary of model inputs.
@@ -106,7 +174,8 @@ class MultiModalPlugin(ABC):
174
175 The model is identified by ``model_config``.
176
109 - TODO: Add guide [ref: PR #5276]
177 + See also:
178 + :ref:`adding_a_new_multimodal_model`
179 """
180 # Avoid circular import
181 from vllm.model_executor.model_loader import get_model_architecture
vllm/multimodal/image.py
+94 -6
@@ -1,19 +1,102 @@
1 from functools import lru_cache
2 -from typing import Dict
2 +from typing import List, Optional, Tuple, TypeVar
3
4 import torch
5 from PIL import Image
6 +from transformers import PreTrainedTokenizerBase
7
8 from vllm.config import ModelConfig
9 from vllm.inputs.registry import InputContext
10 from vllm.logger import init_logger
11 from vllm.transformers_utils.image_processor import get_image_processor
12 +from vllm.transformers_utils.tokenizer import get_tokenizer
13
12 -from .base import MultiModalPlugin
14 +from .base import MultiModalInputs, MultiModalPlugin
15
16 logger = init_logger(__name__)
17
18 cached_get_image_processor = lru_cache(get_image_processor)
19 +cached_get_tokenizer = lru_cache(get_tokenizer)
20 +
21 +# Utilities for image input processors
22 +_T = TypeVar("_T", str, int)
23 +
24 +
25 +def repeat_and_pad_token(
26 + token: _T,
27 + *,
28 + repeat_count: int = 1,
29 + pad_token_left: Optional[_T] = None,
30 + pad_token_right: Optional[_T] = None,
31 +) -> List[_T]:
32 + replacement = [token] * repeat_count
33 + if pad_token_left is not None:
34 + replacement = [pad_token_left] + replacement
35 + if pad_token_right is not None:
36 + replacement = replacement + [pad_token_right]
37 +
38 + return replacement
39 +
40 +
41 +def repeat_and_pad_image_tokens(
42 + tokenizer: PreTrainedTokenizerBase,
43 + prompt: Optional[str],
44 + prompt_token_ids: List[int],
45 + *,
46 + image_token_id: int,
47 + repeat_count: int = 1,
48 + pad_token_left: Optional[int] = None,
49 + pad_token_right: Optional[int] = None,
50 +) -> Tuple[Optional[str], List[int]]:
51 + if prompt is None:
52 + new_prompt = None
53 + else:
54 + image_token_str = tokenizer.decode(image_token_id)
55 + pad_token_str_left = (None if pad_token_left is None else
56 + tokenizer.decode(pad_token_left))
57 + pad_token_str_right = (None if pad_token_right is None else
58 + tokenizer.decode(pad_token_right))
59 + replacement_str = "".join(
60 + repeat_and_pad_token(
61 + image_token_str,
62 + repeat_count=repeat_count,
63 + pad_token_left=pad_token_str_left,
64 + pad_token_right=pad_token_str_right,
65 + ))
66 +
67 + image_token_count = prompt.count(image_token_str)
68 + # This is an arbitrary number to distinguish between the two cases
69 + if image_token_count > 16:
70 + logger.warning(
71 + "Please follow the prompt format that is "
72 + "documented on HuggingFace which does not involve "
73 + "repeating %s tokens.", image_token_str)
74 + elif image_token_count > 1:
75 + logger.warning("Multiple image input is not supported yet, "
76 + "so any extra image tokens will be treated "
77 + "as plain text.")
78 +
79 + # The image tokens are removed to be consistent with HuggingFace
80 + new_prompt = prompt.replace(image_token_str, replacement_str, 1)
81 +
82 + new_token_ids: List[int] = []
83 + for i, token in enumerate(prompt_token_ids):
84 + if token == image_token_id:
85 + replacement_ids = repeat_and_pad_token(
86 + image_token_id,
87 + repeat_count=repeat_count,
88 + pad_token_left=pad_token_left,
89 + pad_token_right=pad_token_right,
90 + )
91 + new_token_ids.extend(replacement_ids)
92 +
93 + # No need to further scan the list since we only replace once
94 + new_token_ids.extend(prompt_token_ids[i + 1:])
95 + break
96 + else:
97 + new_token_ids.append(token)
98 +
99 + return new_prompt, new_token_ids
100
101
102 class ImagePlugin(MultiModalPlugin):
@@ -27,7 +110,7 @@ class ImagePlugin(MultiModalPlugin):
110 trust_remote_code=model_config.trust_remote_code)
111
112 def _default_input_mapper(self, ctx: InputContext,
30 - data: object) -> Dict[str, torch.Tensor]:
113 + data: object) -> MultiModalInputs:
114 model_config = ctx.model_config
115 if isinstance(data, Image.Image):
116 image_processor = self._get_hf_image_processor(model_config)
@@ -35,10 +118,15 @@ class ImagePlugin(MultiModalPlugin):
118 raise RuntimeError("No HuggingFace processor is available"
119 "to process the image object")
120 try:
38 - return image_processor.preprocess(data, return_tensors="pt") \
39 - .to(model_config.dtype).data
121 + batch_data = image_processor \
122 + .preprocess(data, return_tensors="pt") \
123 + .data
124 except Exception:
125 logger.error("Failed to process image (%s)", data)
126 raise
127
44 - raise TypeError(f"Invalid type for 'image': {type(data)}")
128 + return MultiModalInputs(batch_data)
129 + elif isinstance(data, torch.Tensor):
130 + raise NotImplementedError("Embeddings input is not supported yet")
131 +
132 + raise TypeError(f"Invalid image type: {type(data)}")
vllm/multimodal/registry.py
+22 -11
@@ -1,18 +1,17 @@
1 import functools
2 -from typing import Optional, Sequence, Type, TypeVar
2 +from typing import Dict, Optional, Sequence
3
4 -from torch import nn
4 +import torch
5
6 from vllm.config import ModelConfig
7 from vllm.logger import init_logger
8
9 -from .base import MultiModalDataDict, MultiModalInputMapper, MultiModalPlugin
9 +from .base import (MultiModalDataDict, MultiModalInputMapper, MultiModalInputs,
10 + MultiModalPlugin)
11 from .image import ImagePlugin
12
13 logger = init_logger(__name__)
14
14 -N = TypeVar("N", bound=Type[nn.Module])
15 -
15
16 class MultiModalRegistry:
17 """
@@ -61,7 +60,7 @@ class MultiModalRegistry:
60 return self.register_input_mapper("image", mapper)
61
62 def _process_input(self, key: str, value: object,
64 - model_config: ModelConfig):
63 + model_config: ModelConfig) -> MultiModalInputs:
64 plugin = self._plugins.get(key)
65 if plugin:
66 return plugin.map_input(model_config, value)
@@ -93,16 +92,28 @@ class MultiModalRegistry:
92 """
93 return self.register_input_mapper("image", mapper)
94
96 - def map_input(self, model_config: ModelConfig, data: MultiModalDataDict):
95 + def map_input(self, model_config: ModelConfig,
96 + data: MultiModalDataDict) -> MultiModalInputs:
97 """
98 Apply an input mapper to the data passed to the model.
99
100 See :meth:`MultiModalPlugin.map_input` for more details.
101 """
102 - result_list = [
103 - self._process_input(k, v, model_config) for k, v in data.items()
104 - ]
105 - return {k: v for d in result_list for k, v in d.items()}
102 + merged_dict: Dict[str, torch.Tensor] = {}
103 +
104 + for data_key, data_value in data.items():
105 + input_dict = self._process_input(data_key, data_value,
106 + model_config)
107 +
108 + for input_key, input_tensor in input_dict.items():
109 + if input_key in merged_dict:
110 + raise ValueError(f"The input mappers (keys={set(data)}) "
111 + f"resulted in a conflicting keyword "
112 + f"argument to `forward()`: {input_key}")
113 +
114 + merged_dict[input_key] = input_tensor
115 +
116 + return MultiModalInputs(merged_dict)
117
118 def create_input_mapper(self, model_config: ModelConfig):
119 """
vllm/multimodal/utils.py
+61 -34
@@ -4,11 +4,56 @@ from typing import Optional, Union
4 from urllib.parse import urlparse
5
6 import aiohttp
7 +import requests
8 from PIL import Image
9
9 -from vllm.config import ModelConfig
10 from vllm.envs import VLLM_IMAGE_FETCH_TIMEOUT
11 from vllm.multimodal.base import MultiModalDataDict
12 +from vllm.version import __version__ as VLLM_VERSION
13 +
14 +
15 +def _validate_remote_url(url: str, *, name: str):
16 + parsed_url = urlparse(url)
17 + if parsed_url.scheme not in ["http", "https"]:
18 + raise ValueError(f"Invalid '{name}': A valid '{name}' "
19 + "must have scheme 'http' or 'https'.")
20 +
21 +
22 +def _get_request_headers():
23 + return {"User-Agent": f"vLLM/{VLLM_VERSION}"}
24 +
25 +
26 +def _load_image_from_bytes(b: bytes):
27 + image = Image.open(BytesIO(b))
28 + image.load()
29 + return image
30 +
31 +
32 +def _load_image_from_data_url(image_url: str):
33 + # Only split once and assume the second part is the base64 encoded image
34 + _, image_base64 = image_url.split(",", 1)
35 + return load_image_from_base64(image_base64)
36 +
37 +
38 +def fetch_image(image_url: str) -> Image.Image:
39 + """Load PIL image from a url or base64 encoded openai GPT4V format"""
40 + if image_url.startswith('http'):
41 + _validate_remote_url(image_url, name="image_url")
42 +
43 + headers = _get_request_headers()
44 +
45 + with requests.get(url=image_url, headers=headers) as response:
46 + response.raise_for_status()
47 + image_raw = response.content
48 + image = _load_image_from_bytes(image_raw)
49 +
50 + elif image_url.startswith('data:image'):
51 + image = _load_image_from_data_url(image_url)
52 + else:
53 + raise ValueError("Invalid 'image_url': A valid 'image_url' must start "
54 + "with either 'data:image' or 'http'.")
55 +
56 + return image
57
58
59 class ImageFetchAiohttp:
@@ -29,34 +74,31 @@ class ImageFetchAiohttp:
74 """Load PIL image from a url or base64 encoded openai GPT4V format"""
75
76 if image_url.startswith('http'):
32 - parsed_url = urlparse(image_url)
33 - if parsed_url.scheme not in ["http", "https"]:
34 - raise ValueError("Invalid 'image_url': A valid 'image_url' "
35 - "must have scheme 'http' or 'https'.")
36 - # Avoid circular import
37 - from vllm import __version__ as VLLM_VERSION
77 + _validate_remote_url(image_url, name="image_url")
78
79 client = cls.get_aiohttp_client()
40 - headers = {"User-Agent": f"vLLM/{VLLM_VERSION}"}
80 + headers = _get_request_headers()
81
82 async with client.get(url=image_url, headers=headers) as response:
83 response.raise_for_status()
84 image_raw = await response.read()
45 - image = Image.open(BytesIO(image_raw))
85 + image = _load_image_from_bytes(image_raw)
86
47 - # Only split once and assume the second part is the base64 encoded image
87 elif image_url.startswith('data:image'):
49 - image = load_image_from_base64(image_url.split(',', 1)[1])
50 -
88 + image = _load_image_from_data_url(image_url)
89 else:
90 raise ValueError(
91 "Invalid 'image_url': A valid 'image_url' must start "
92 "with either 'data:image' or 'http'.")
93
56 - image.load()
94 return image
95
96
97 +async def async_get_and_parse_image(image_url: str) -> MultiModalDataDict:
98 + image = await ImageFetchAiohttp.fetch_image(image_url)
99 + return {"image": image}
100 +
101 +
102 def encode_image_base64(image: Image.Image, format: str = 'JPEG') -> str:
103 """Encode a pillow image to base64 format."""
104
@@ -69,26 +111,11 @@ def encode_image_base64(image: Image.Image, format: str = 'JPEG') -> str:
111
112 def load_image_from_base64(image: Union[bytes, str]) -> Image.Image:
113 """Load image from base64 format."""
72 - return Image.open(BytesIO(base64.b64decode(image)))
114 + return _load_image_from_bytes(base64.b64decode(image))
115
116
75 -# TODO(ywang96): move this to a model registry for preprocessing vision
76 -# language prompts based on the model type.
77 -def get_full_image_text_prompt(image_prompt: str, text_prompt: str,
78 - config: ModelConfig) -> str:
79 - """Combine image and text prompts for vision language model depending on
80 - the model architecture."""
81 -
82 - if config.hf_config.model_type in ("llava", "llava_next"):
83 - full_prompt = f"{image_prompt}\n{text_prompt}"
84 - elif config.hf_config.model_type == 'phi3_v':
85 - full_prompt = f"{image_prompt}<s>\n{text_prompt}"
86 - else:
87 - raise ValueError(
88 - f"Unsupported model type: {config.hf_config.model_type}")
89 - return full_prompt
90 -
91 -
92 -async def async_get_and_parse_image(image_url: str) -> MultiModalDataDict:
93 - image = await ImageFetchAiohttp.fetch_image(image_url)
94 - return {"image": image}
117 +def rescale_image_size(image: Image.Image, size_factor: float) -> Image.Image:
118 + """Rescale the dimensions of an image by a constant factor."""
119 + new_width = int(image.width * size_factor)
120 + new_height = int(image.height * size_factor)
121 + return image.resize((new_width, new_height))
vllm/sequence.py
+1 -1
@@ -457,7 +457,7 @@ class SequenceGroup:
457 return next(iter(self.seqs_dict.values())).prompt_token_ids
458
459 @property
460 - def multi_modal_data(self) -> Optional["MultiModalDataDict"]:
460 + def multi_modal_data(self) -> "MultiModalDataDict":
461 # All sequences in the group should have the same multi-modal data.
462 # We use the multi-modal data of an arbitrary sequence.
463 return next(iter(self.seqs_dict.values())).multi_modal_data
vllm/transformers_utils/image_processor.py
+9 -9
@@ -1,9 +1,4 @@
1 -from transformers import AutoImageProcessor
2 -from transformers.image_processing_utils import BaseImageProcessor
3 -
4 -from vllm.logger import init_logger
5 -
6 -logger = init_logger(__name__)
1 +from typing import cast
2
3
4 def get_image_processor(
@@ -11,10 +6,15 @@ def get_image_processor(
6 *args,
7 trust_remote_code: bool = False,
8 **kwargs,
14 -) -> BaseImageProcessor:
9 +):
10 """Gets an image processor for the given model name via HuggingFace."""
11 + # don't put this import at the top level
12 + # it will call torch.cuda.device_count()
13 + from transformers import AutoImageProcessor
14 + from transformers.image_processing_utils import BaseImageProcessor
15 +
16 try:
17 - processor: BaseImageProcessor = AutoImageProcessor.from_pretrained(
17 + processor = AutoImageProcessor.from_pretrained(
18 processor_name,
19 *args,
20 trust_remote_code=trust_remote_code,
@@ -34,4 +34,4 @@ def get_image_processor(
34 else:
35 raise e
36
37 - return processor
37 + return cast(BaseImageProcessor, processor)
vllm/worker/cpu_model_runner.py
+15 -19
@@ -1,6 +1,6 @@
1 -from collections import defaultdict
1 from dataclasses import dataclass
3 -from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Type, Union
2 +from typing import (TYPE_CHECKING, Any, Dict, List, Mapping, Optional, Tuple,
3 + Type, Union)
4
5 import torch
6 from torch import nn
@@ -12,7 +12,8 @@ from vllm.config import (CacheConfig, DeviceConfig, LoadConfig, LoRAConfig,
12 from vllm.logger import init_logger
13 from vllm.model_executor import SamplingMetadata
14 from vllm.model_executor.model_loader import get_model
15 -from vllm.multimodal import MULTIMODAL_REGISTRY
15 +from vllm.multimodal import (MULTIMODAL_REGISTRY, BatchedTensors,
16 + MultiModalInputs)
17 from vllm.sequence import (IntermediateTensors, SamplerOutput,
18 SequenceGroupMetadata)
19 from vllm.utils import make_tensor_with_pad
@@ -40,7 +41,7 @@ class CPUModelInput(ModelRunnerInputBase):
41 input_positions: Optional[torch.Tensor] = None
42 attn_metadata: Optional["AttentionMetadata"] = None
43 sampling_metadata: Optional["SamplingMetadata"] = None
43 - multi_modal_kwargs: Optional[Dict[str, torch.Tensor]] = None
44 + multi_modal_kwargs: Optional[Mapping[str, BatchedTensors]] = None
45
46 def as_broadcastable_tensor_dict(
47 self) -> Dict[str, Union[int, torch.Tensor]]:
@@ -132,15 +133,14 @@ class CPUModelRunner(ModelRunnerBase[CPUModelInput]):
133 def _prepare_prompt(
134 self,
135 seq_group_metadata_list: List[SequenceGroupMetadata],
135 - ) -> Tuple[torch.Tensor, torch.Tensor, AttentionMetadata, List[int], Dict[
136 - str, torch.Tensor]]:
136 + ) -> Tuple[torch.Tensor, torch.Tensor, AttentionMetadata, List[int],
137 + Mapping[str, BatchedTensors]]:
138 assert len(seq_group_metadata_list) > 0
139 input_tokens: List[int] = []
140 input_positions: List[int] = []
141 slot_mapping: List[int] = []
142 seq_lens: List[int] = []
142 - multi_modal_kwargs_list: Dict[str,
143 - List[torch.Tensor]] = defaultdict(list)
143 + multi_modal_inputs_list: List[MultiModalInputs] = []
144
145 for seq_group_metadata in seq_group_metadata_list:
146 assert seq_group_metadata.is_prompt
@@ -162,10 +162,9 @@ class CPUModelRunner(ModelRunnerBase[CPUModelInput]):
162 input_positions.extend(list(range(computed_len, seq_len)))
163
164 mm_data = seq_group_metadata.multi_modal_data
165 - if mm_data is not None:
165 + if mm_data:
166 mm_kwargs = self.multi_modal_input_mapper(mm_data)
167 - for k, v in mm_kwargs.items():
168 - multi_modal_kwargs_list[k].append(v)
167 + multi_modal_inputs_list.append(mm_kwargs)
168
169 # Compute the slot mapping.
170 block_table = seq_group_metadata.block_tables[seq_id]
@@ -189,11 +188,6 @@ class CPUModelRunner(ModelRunnerBase[CPUModelInput]):
188 slot = block_number * self.block_size + block_offset
189 slot_mapping.append(slot)
190
192 - multi_modal_kwargs = {
193 - k: torch.cat(v, dim=0).to(self.device)
194 - for k, v in multi_modal_kwargs_list.items()
195 - }
196 -
191 num_prompt_tokens = len(input_tokens)
192
193 input_tokens = torch.tensor(input_tokens,
@@ -217,6 +211,10 @@ class CPUModelRunner(ModelRunnerBase[CPUModelInput]):
211 block_tables=torch.tensor([]),
212 slot_mapping=slot_mapping,
213 )
214 +
215 + multi_modal_kwargs = MultiModalInputs.batch(multi_modal_inputs_list,
216 + device=self.device)
217 +
218 return (input_tokens, input_positions, attn_metadata, seq_lens,
219 multi_modal_kwargs)
220
@@ -367,10 +365,8 @@ class CPUModelRunner(ModelRunnerBase[CPUModelInput]):
365 "positions": model_input.input_positions,
366 "kv_caches": kv_caches,
367 "attn_metadata": model_input.attn_metadata,
368 + **(model_input.multi_modal_kwargs or {}),
369 }
371 - if (self.vision_language_config
372 - and model_input.multi_modal_kwargs is not None):
373 - execute_model_kwargs.update(model_input.multi_modal_kwargs)
370
371 hidden_states = model_executable(**execute_model_kwargs)
372
vllm/worker/embedding_model_runner.py
+2 -3
@@ -92,10 +92,9 @@ class EmbeddingModelRunner(
92 "positions": model_input.input_positions,
93 "kv_caches": kv_caches,
94 "attn_metadata": model_input.attn_metadata,
95 + **(model_input.multi_modal_kwargs or {}),
96 }
96 - if self.vision_language_config:
97 - multi_modal_kwargs = model_input.multi_modal_kwargs or {}
98 - execute_model_kwargs.update({"image_input": multi_modal_kwargs})
97 +
98 hidden_states = model_executable(**execute_model_kwargs)
99
100 # Only perform pooling in the driver worker.
vllm/worker/model_runner.py
+14 -13
@@ -3,8 +3,8 @@ import gc
3 import time
4 import warnings
5 from collections import defaultdict
6 -from typing import (TYPE_CHECKING, Any, Dict, List, Optional, Set, Tuple, Type,
7 - TypeVar, Union)
6 +from typing import (TYPE_CHECKING, Any, Dict, List, Mapping, Optional, Set,
7 + Tuple, Type, TypeVar, Union)
8
9 import numpy as np
10 import torch
@@ -37,7 +37,8 @@ from vllm.model_executor import SamplingMetadata
37 from vllm.model_executor.model_loader import get_model
38 from vllm.model_executor.model_loader.tensorizer import TensorizerConfig
39 from vllm.model_executor.models.interfaces import supports_lora
40 -from vllm.multimodal import MULTIMODAL_REGISTRY
40 +from vllm.multimodal import (MULTIMODAL_REGISTRY, BatchedTensors,
41 + MultiModalInputs)
42 from vllm.sampling_params import SamplingParams
43 from vllm.sequence import (IntermediateTensors, SamplerOutput,
44 SequenceGroupMetadata)
@@ -83,7 +84,7 @@ class ModelInputForGPU(ModelRunnerInputBase):
84 lora_mapping: Optional["LoRAMapping"] = None
85 lora_requests: Optional[Set[LoRARequest]] = None
86 attn_metadata: Optional["AttentionMetadata"] = None
86 - multi_modal_kwargs: Optional[Dict[str, torch.Tensor]] = None
87 + multi_modal_kwargs: Optional[Mapping[str, BatchedTensors]] = None
88 request_ids_to_seq_ids: Optional[Dict[str, List[int]]] = None
89 finished_requests_ids: Optional[List[str]] = None
90 virtual_engine: int = 0
@@ -356,8 +357,7 @@ class GPUModelRunnerBase(ModelRunnerBase[TModelInputForGPU]):
357 context_lens: List[int] = []
358 query_lens: List[int] = []
359 block_tables: List[List[int]] = []
359 - multi_modal_kwargs_list: Dict[str,
360 - List[torch.Tensor]] = defaultdict(list)
360 + multi_modal_inputs_list: List[MultiModalInputs] = []
361 request_ids_to_seq_ids: Dict[str, List[int]] = defaultdict(list)
362 decode_only = True
363 num_prefills = 0
@@ -528,8 +528,7 @@ class GPUModelRunnerBase(ModelRunnerBase[TModelInputForGPU]):
528 if mm_data:
529 # Process multi-modal data
530 mm_kwargs = self.multi_modal_input_mapper(mm_data)
531 - for k, v in mm_kwargs.items():
532 - multi_modal_kwargs_list[k].append(v)
531 + multi_modal_inputs_list.append(mm_kwargs)
532
533 is_profile_run = _is_block_tables_empty(
534 seq_group_metadata.block_tables)
@@ -746,10 +745,8 @@ class GPUModelRunnerBase(ModelRunnerBase[TModelInputForGPU]):
745 else:
746 lora_mapping = None
747
749 - multi_modal_kwargs = {
750 - k: torch.cat(v, dim=0).to(self.device)
751 - for k, v in multi_modal_kwargs_list.items()
752 - }
748 + multi_modal_kwargs = MultiModalInputs.batch(multi_modal_inputs_list,
749 + device=self.device)
750 request_ids_to_seq_ids = {
751 seq_group_metadata.request_id:
752 list(seq_group_metadata.seq_data.keys())
@@ -821,7 +818,11 @@ class GPUModelRunnerBase(ModelRunnerBase[TModelInputForGPU]):
818
819 seq_data, dummy_multi_modal_data = INPUT_REGISTRY \
820 .dummy_data_for_profiling(model_config, seq_len)
824 - assert len(seq_data.prompt_token_ids) == seq_len
821 +
822 + # Having more tokens is over-conservative but otherwise fine
823 + assert len(seq_data.prompt_token_ids) >= seq_len, (
824 + f"Expected at least {seq_len} dummy tokens for profiling, "
825 + f"but got: {len(seq_data.prompt_token_ids)}")
826
827 seq = SequenceGroupMetadata(
828 request_id=str(group_id),
vllm/worker/neuron_model_runner.py
+29 -6
@@ -1,5 +1,6 @@
1 from dataclasses import dataclass
2 -from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union
2 +from typing import (TYPE_CHECKING, Any, Dict, List, Mapping, Optional, Tuple,
3 + Union)
4
5 import torch
6 from torch import nn
@@ -9,6 +10,8 @@ from vllm.config import (DeviceConfig, ModelConfig, ParallelConfig,
10 from vllm.logger import init_logger
11 from vllm.model_executor import SamplingMetadata
12 from vllm.model_executor.model_loader.neuron import get_neuron_model
13 +from vllm.multimodal import (MULTIMODAL_REGISTRY, BatchedTensors,
14 + MultiModalInputs)
15 from vllm.sequence import (IntermediateTensors, SamplerOutput,
16 SequenceGroupMetadata)
17 from vllm.utils import is_pin_memory_available, make_tensor_with_pad
@@ -29,6 +32,7 @@ class ModelInputForNeuron(ModelRunnerInputBase):
32 input_positions: Optional[torch.Tensor] = None
33 input_block_ids: Optional[torch.Tensor] = None
34 sampling_metadata: Optional["SamplingMetadata"] = None
35 + multi_modal_kwargs: Optional[Mapping[str, BatchedTensors]] = None
36
37 def as_broadcastable_tensor_dict(
38 self) -> Dict[str, Union[int, torch.Tensor]]:
@@ -65,6 +69,10 @@ class NeuronModelRunner(ModelRunnerBase[ModelInputForNeuron]):
69 self.device = self.device_config.device
70 self.pin_memory = is_pin_memory_available()
71
72 + # Multi-modal data support
73 + self.multi_modal_input_mapper = MULTIMODAL_REGISTRY \
74 + .create_input_mapper(self.model_config)
75 +
76 # Lazy initialization.
77 self.model: nn.Module # initialize after load_model.
78
@@ -76,13 +84,15 @@ class NeuronModelRunner(ModelRunnerBase[ModelInputForNeuron]):
84 def _prepare_prompt(
85 self,
86 seq_group_metadata_list: List[SequenceGroupMetadata],
79 - ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor, List[int]]:
87 + ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor, List[int], Mapping[
88 + str, BatchedTensors]]:
89 assert len(seq_group_metadata_list) > 0
90 input_tokens: List[List[int]] = []
91 input_positions: List[List[int]] = []
92 input_block_ids: List[int] = []
93
94 seq_lens: List[int] = []
95 + multi_modal_inputs_list: List[MultiModalInputs] = []
96 for seq_group_metadata in seq_group_metadata_list:
97 assert seq_group_metadata.is_prompt
98 seq_ids = list(seq_group_metadata.seq_data.keys())
@@ -102,6 +112,12 @@ class NeuronModelRunner(ModelRunnerBase[ModelInputForNeuron]):
112 assert len(block_table) == 1
113 input_block_ids.append(block_table[0])
114
115 + mm_data = seq_group_metadata.multi_modal_data
116 + if mm_data:
117 + # Process multi-modal data
118 + mm_kwargs = self.multi_modal_input_mapper(mm_data)
119 + multi_modal_inputs_list.append(mm_kwargs)
120 +
121 max_seq_len = max(seq_lens)
122 assert max_seq_len > 0
123 input_tokens = make_tensor_with_pad(input_tokens,
@@ -118,7 +134,11 @@ class NeuronModelRunner(ModelRunnerBase[ModelInputForNeuron]):
134 dtype=torch.long,
135 device=self.device)
136
121 - return input_tokens, input_positions, input_block_ids, seq_lens
137 + multi_modal_kwargs = MultiModalInputs.batch(multi_modal_inputs_list,
138 + device=self.device)
139 +
140 + return (input_tokens, input_positions, input_block_ids, seq_lens,
141 + multi_modal_kwargs)
142
143 def _prepare_decode(
144 self,
@@ -184,8 +204,9 @@ class NeuronModelRunner(ModelRunnerBase[ModelInputForNeuron]):
204 is_prompt = seq_group_metadata_list[0].is_prompt
205 # Prepare input tensors.
206 if is_prompt:
187 - (input_tokens, input_positions, input_block_ids,
188 - seq_lens) = self._prepare_prompt(seq_group_metadata_list)
207 + (input_tokens, input_positions, input_block_ids, seq_lens,
208 + multi_modal_kwargs
209 + ) = self._prepare_prompt(seq_group_metadata_list)
210 else:
211 (input_tokens, input_positions,
212 input_block_ids) = self._prepare_decode(seq_group_metadata_list)
@@ -203,7 +224,8 @@ class NeuronModelRunner(ModelRunnerBase[ModelInputForNeuron]):
224 return ModelInputForNeuron(input_tokens=input_tokens,
225 input_positions=input_positions,
226 input_block_ids=input_block_ids,
206 - sampling_metadata=sampling_metadata)
227 + sampling_metadata=sampling_metadata,
228 + multi_modal_kwargs=multi_modal_kwargs)
229
230 @torch.inference_mode()
231 def execute_model(
@@ -221,6 +243,7 @@ class NeuronModelRunner(ModelRunnerBase[ModelInputForNeuron]):
243 input_ids=model_input.input_tokens,
244 positions=model_input.input_positions,
245 input_block_ids=model_input.input_block_ids,
246 + **(model_input.multi_modal_kwargs or {}),
247 )
248
249 # Compute the logits.
vllm/worker/openvino_model_runner.py
+26 -12
@@ -1,4 +1,4 @@
1 -from typing import List, NamedTuple, Optional, Tuple
1 +from typing import List, Mapping, NamedTuple, Optional, Tuple
2
3 import openvino as ov
4 import torch
@@ -12,6 +12,8 @@ from vllm.config import (CacheConfig, DeviceConfig, LoadConfig, LoRAConfig,
12 from vllm.logger import init_logger
13 from vllm.model_executor import SamplingMetadata
14 from vllm.model_executor.model_loader.openvino import get_model
15 +from vllm.multimodal import (MULTIMODAL_REGISTRY, BatchedTensors,
16 + MultiModalInputs)
17 from vllm.sequence import SamplerOutput, SequenceGroupMetadata
18
19 logger = init_logger(__name__)
@@ -23,7 +25,7 @@ class ModelInput(NamedTuple):
25 attn_metadata: Optional[OpenVINOAttentionMetadata]
26 seq_lens: List[int]
27 query_lens: List[int]
26 - multi_modal_input: Optional[torch.Tensor]
28 + multi_modal_kwargs: Mapping[str, BatchedTensors]
29
30 @classmethod
31 def empty(cls, device):
@@ -32,7 +34,7 @@ class ModelInput(NamedTuple):
34 attn_metadata=None,
35 seq_lens=[],
36 query_lens=[],
35 - multi_modal_input=None)
37 + multi_modal_kwargs={})
38
39
40 class OpenVINOModelRunner:
@@ -78,6 +80,10 @@ class OpenVINOModelRunner:
80 self.block_size,
81 )
82
83 + # Multi-modal data support
84 + self.multi_modal_input_mapper = MULTIMODAL_REGISTRY \
85 + .create_input_mapper(self.model_config)
86 +
87 # Lazy initialization.
88 self.model: nn.Module # Set after init_Model
89
@@ -108,6 +114,8 @@ class OpenVINOModelRunner:
114 seq_lens: List[int] = []
115 past_lens: List[int] = []
116 query_lens: List[int] = []
117 + multi_modal_inputs_list: List[MultiModalInputs] = []
118 +
119 subsequence_begins: List[int] = []
120 block_indices: List[int] = []
121 block_indices_begins: List[int] = []
@@ -160,6 +168,11 @@ class OpenVINOModelRunner:
168 and self.sliding_window is None
169 and is_prompt)
170
171 + mm_data = seq_group_metadata.multi_modal_data
172 + if mm_data:
173 + mm_kwargs = self.multi_modal_input_mapper(mm_data)
174 + multi_modal_inputs_list.append(mm_kwargs)
175 +
176 block_table = seq_group_metadata.block_tables[seq_id]
177 # TODO(sang): Combine chunked prefill and prefix caching by
178 # only allowing multiple of block_size chunk size.
@@ -251,22 +264,24 @@ class OpenVINOModelRunner:
264 block_indices_begins=block_indices_begins_tensor,
265 max_context_len=max_context_len_tensor,
266 )
267 +
268 + multi_modal_kwargs = MultiModalInputs.batch(multi_modal_inputs_list,
269 + device=self.device)
270 +
271 return ModelInput(
272 input_tokens,
273 input_positions,
274 attn_metadata,
275 seq_lens,
276 query_lens,
260 - None,
277 + multi_modal_kwargs=multi_modal_kwargs,
278 )
279
280 def prepare_input_tensors(
281 self,
282 seq_group_metadata_list: List[SequenceGroupMetadata],
283 ) -> Tuple[torch.Tensor, torch.Tensor, OpenVINOAttentionMetadata,
267 - SamplingMetadata, Optional[torch.Tensor], ]:
268 - multi_modal_input = None
269 -
284 + SamplingMetadata, Mapping[str, BatchedTensors]]:
285 # Prepare input tensors.
286 (
287 input_tokens,
@@ -274,7 +289,7 @@ class OpenVINOModelRunner:
289 attn_metadata,
290 seq_lens,
291 query_lens,
277 - multi_modal_input,
292 + multi_modal_kwargs,
293 ) = self._prepare_model_input(seq_group_metadata_list)
294
295 sampling_metadata = SamplingMetadata.prepare(
@@ -290,7 +305,7 @@ class OpenVINOModelRunner:
305 input_positions,
306 attn_metadata,
307 sampling_metadata,
293 - multi_modal_input,
308 + multi_modal_kwargs,
309 )
310
311 @torch.inference_mode()
@@ -304,7 +319,7 @@ class OpenVINOModelRunner:
319 input_positions,
320 attn_metadata,
321 sampling_metadata,
307 - multi_modal_input,
322 + multi_modal_kwargs,
323 ) = self.prepare_input_tensors(seq_group_metadata_list)
324
325 model_executable = self.model
@@ -313,9 +328,8 @@ class OpenVINOModelRunner:
328 "positions": input_positions,
329 "kv_caches": kv_caches,
330 "attn_metadata": attn_metadata,
331 + **(multi_modal_kwargs or {}),
332 }
317 - if self.vision_language_config:
318 - execute_model_kwargs.update({"image_input": multi_modal_input})
333
334 hidden_states = model_executable(**execute_model_kwargs)
335
vllm/worker/tpu_model_runner.py
+39 -5
@@ -1,5 +1,5 @@
1 import time
2 -from typing import List, Optional, Tuple
2 +from typing import List, Mapping, Optional, Tuple
3
4 import numpy as np
5 import torch
@@ -12,6 +12,8 @@ from vllm.config import (CacheConfig, DeviceConfig, LoadConfig, ModelConfig,
12 from vllm.logger import init_logger
13 from vllm.model_executor.model_loader import get_model
14 from vllm.model_executor.sampling_metadata import SamplingMetadata
15 +from vllm.multimodal import (MULTIMODAL_REGISTRY, BatchedTensors,
16 + MultiModalInputs)
17 from vllm.sequence import (CompletionSequenceGroupOutput, Logprob,
18 SamplerOutput, SequenceGroupMetadata,
19 SequenceOutput)
@@ -66,6 +68,10 @@ class TPUModelRunner:
68 False,
69 )
70
71 + # Multi-modal data support
72 + self.multi_modal_input_mapper = MULTIMODAL_REGISTRY \
73 + .create_input_mapper(self.model_config)
74 +
75 def load_model(self) -> None:
76 self.device = self.device_config.device
77
@@ -193,12 +199,14 @@ class TPUModelRunner:
199 def _prepare_prompt(
200 self,
201 seq_group_metadata_list: List[SequenceGroupMetadata],
196 - ):
202 + ) -> Tuple[torch.Tensor, torch.Tensor, AttentionMetadata, torch.Tensor,
203 + Mapping[str, BatchedTensors]]:
204 assert len(seq_group_metadata_list) > 0
205 input_tokens: List[List[int]] = []
206 input_positions: List[List[int]] = []
207 prompt_lens: List[int] = []
208 slot_mapping: List[List[int]] = []
209 + multi_modal_inputs_list: List[MultiModalInputs] = []
210
211 for seq_group_metadata in seq_group_metadata_list:
212 assert seq_group_metadata.is_prompt
@@ -224,6 +232,11 @@ class TPUModelRunner:
232 slot = block_number * self.block_size + block_offset
233 slot_mapping[-1].append(slot)
234
235 + mm_data = seq_group_metadata.multi_modal_data
236 + if mm_data:
237 + mm_kwargs = self.multi_modal_input_mapper(mm_data)
238 + multi_modal_inputs_list.append(mm_kwargs)
239 +
240 assert len(prompt_lens) > 0
241 num_prefills = len(prompt_lens)
242 num_prefill_tokens = sum(prompt_lens)
@@ -261,17 +274,24 @@ class TPUModelRunner:
274 block_tables=None,
275 context_lens=None,
276 )
264 - return input_tokens, input_positions, attn_metadata, prompt_lens
277 +
278 + multi_modal_kwargs = MultiModalInputs.batch(multi_modal_inputs_list,
279 + device=self.device)
280 +
281 + return (input_tokens, input_positions, attn_metadata, prompt_lens,
282 + multi_modal_kwargs)
283
284 def _prepare_decode(
285 self,
286 seq_group_metadata_list: List[SequenceGroupMetadata],
269 - ):
287 + ) -> Tuple[torch.Tensor, torch.Tensor, AttentionMetadata, torch.Tensor,
288 + Mapping[str, BatchedTensors]]:
289 assert len(seq_group_metadata_list) > 0
290 input_tokens: List[List[int]] = []
291 input_positions: List[List[int]] = []
292 slot_mapping: List[List[int]] = []
293 context_lens: List[int] = []
294 + multi_modal_inputs_list: List[MultiModalInputs] = []
295
296 batch_idx = 0
297 for seq_group_metadata in seq_group_metadata_list:
@@ -297,6 +317,11 @@ class TPUModelRunner:
317 slot = block_number * self.block_size + block_offset
318 slot_mapping.append([slot])
319
320 + mm_data = seq_group_metadata.multi_modal_data
321 + if mm_data:
322 + mm_kwargs = self.multi_modal_input_mapper(mm_data)
323 + multi_modal_inputs_list.append(mm_kwargs)
324 +
325 batch_size = _get_padded_batch_size(batch_idx)
326 num_paddings = batch_size - batch_idx
327 input_tokens = input_tokens + [[0]] * num_paddings
@@ -330,7 +355,12 @@ class TPUModelRunner:
355 block_tables=block_tables,
356 context_lens=context_lens,
357 )
333 - return input_tokens, input_positions, attn_metadata, input_lens
358 +
359 + multi_modal_kwargs = MultiModalInputs.batch(multi_modal_inputs_list,
360 + device=self.device)
361 +
362 + return (input_tokens, input_positions, attn_metadata, input_lens,
363 + multi_modal_kwargs)
364
365 def _prepare_sample(
366 self,
@@ -483,6 +513,7 @@ class ModelWrapper(nn.Module):
513 kv_caches: List[Tuple[Optional[torch.Tensor], Optional[torch.Tensor]]],
514 attn_metadata: AttentionMetadata,
515 input_lens: torch.Tensor,
516 + multi_modal_kwargs: Optional[Mapping[str, BatchedTensors]],
517 t: torch.Tensor,
518 p: torch.Tensor,
519 num_samples: int,
@@ -496,6 +527,8 @@ class ModelWrapper(nn.Module):
527 memory profiling at initialization.
528 attn_metadata: The Pallas attention metadata.
529 input_lens: The actual input lengths of shape [batch_size].
530 + multi_modal_kwargs: Keyword arguments from multi-modal data to
531 + pass to the model.
532 t: The sampling temperature of shape [batch_size].
533 p: The top-p probability of shape [batch_size].
534 """
@@ -540,6 +573,7 @@ class ModelWrapper(nn.Module):
573 position_ids,
574 kv_caches,
575 attn_metadata,
576 + **(multi_modal_kwargs or {}),
577 )
578 hidden_states = hidden_states.flatten(0, 1)
579 logits = self.model.compute_logits(hidden_states, sampling_metadata)
vllm/worker/xpu_model_runner.py
+44 -26
@@ -1,5 +1,6 @@
1 from dataclasses import dataclass
2 -from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Type, Union
2 +from typing import (TYPE_CHECKING, Any, Dict, List, Mapping, Optional, Tuple,
3 + Type, Union)
4
5 import torch
6 import torch.nn as nn
@@ -9,10 +10,13 @@ from vllm.config import (CacheConfig, DeviceConfig, LoadConfig, LoRAConfig,
10 ModelConfig, ParallelConfig, SchedulerConfig,
11 VisionLanguageConfig)
12 from vllm.distributed import broadcast_tensor_dict
13 +from vllm.inputs import INPUT_REGISTRY
14 from vllm.logger import init_logger
15 from vllm.model_executor.model_loader import get_model
16 +from vllm.multimodal import (MULTIMODAL_REGISTRY, BatchedTensors,
17 + MultiModalInputs)
18 from vllm.sampling_params import SamplingParams
15 -from vllm.sequence import (IntermediateTensors, SamplerOutput, SequenceData,
19 +from vllm.sequence import (IntermediateTensors, SamplerOutput,
20 SequenceGroupMetadata)
21 from vllm.utils import CudaMemoryProfiler, make_tensor_with_pad
22 from vllm.worker.model_runner import AttentionMetadata, SamplingMetadata
@@ -44,7 +48,7 @@ class ModelInputForXPU(ModelRunnerInputBase):
48 input_positions: Optional[torch.Tensor] = None
49 attn_metadata: Optional["AttentionMetadata"] = None
50 sampling_metadata: Optional["SamplingMetadata"] = None
47 - multi_modal_input: Optional[Dict[str, torch.Tensor]] = None
51 + multi_modal_kwargs: Optional[Mapping[str, BatchedTensors]] = None
52
53 def as_broadcastable_tensor_dict(
54 self) -> Dict[str, Union[int, torch.Tensor]]:
@@ -116,6 +120,10 @@ class XPUModelRunner(ModelRunnerBase[ModelInputForXPU]):
120 self.block_size,
121 )
122
123 + # Multi-modal data support
124 + self.multi_modal_input_mapper = MULTIMODAL_REGISTRY \
125 + .create_input_mapper(self.model_config)
126 +
127 # Lazy initialization.
128 self.model: nn.Module # Set after init_Model
129
@@ -156,12 +164,26 @@ class XPUModelRunner(ModelRunnerBase[ModelInputForXPU]):
164 # To exercise the worst scenario for GPU memory consumption,
165 # the number of seqs (batch_size) is chosen to maximize the number
166 # of images processed.
167 + model_config = self.model_config
168 + vlm_config = self.vision_language_config
169 +
170 + if vlm_config:
171 + max_num_seqs = min(
172 + max_num_seqs,
173 + int(max_num_batched_tokens / vlm_config.image_feature_size))
174 +
175 for group_id in range(max_num_seqs):
176 seq_len = (max_num_batched_tokens // max_num_seqs +
177 (group_id < max_num_batched_tokens % max_num_seqs))
178
163 - seq_data = SequenceData([0] * seq_len)
164 - dummy_multi_modal_data = None
179 + seq_data, dummy_multi_modal_data = INPUT_REGISTRY \
180 + .dummy_data_for_profiling(model_config, seq_len)
181 +
182 + # Having more tokens is over-conservative but otherwise fine
183 + assert len(seq_data.prompt_token_ids) >= seq_len, (
184 + f"Expected at least {seq_len} dummy tokens for profiling, "
185 + f"but got: {len(seq_data.prompt_token_ids)}")
186 +
187 seq = SequenceGroupMetadata(
188 request_id=str(group_id),
189 is_prompt=True,
@@ -194,7 +216,7 @@ class XPUModelRunner(ModelRunnerBase[ModelInputForXPU]):
216 virtual_engine: int = 0,
217 finished_requests_ids: Optional[List[str]] = None
218 ) -> ModelInputForXPU:
197 - multi_modal_input = None
219 + multi_modal_kwargs = None
220 if self.is_driver_worker:
221 # NOTE: We assume that all sequences in the group are all prompts or
222 # all decodes.
@@ -202,7 +224,7 @@ class XPUModelRunner(ModelRunnerBase[ModelInputForXPU]):
224 # Prepare input tensors.
225 if is_prompt:
226 (input_tokens, input_positions, attn_metadata, seq_lens,
205 - multi_modal_input
227 + multi_modal_kwargs
228 ) = self._prepare_prompt(seq_group_metadata_list)
229 else:
230 (input_tokens, input_positions,
@@ -223,6 +245,7 @@ class XPUModelRunner(ModelRunnerBase[ModelInputForXPU]):
245 "input_positions": input_positions,
246 "selected_token_indices":
247 sampling_metadata.selected_token_indices,
248 + "multi_modal_kwargs": multi_modal_kwargs,
249 }
250 metadata_dict.update(attn_metadata.asdict_zerocopy())
251 broadcast_tensor_dict(metadata_dict, src=0)
@@ -232,6 +255,7 @@ class XPUModelRunner(ModelRunnerBase[ModelInputForXPU]):
255 input_positions = metadata_dict.pop("input_positions")
256 selected_token_indices = metadata_dict.pop(
257 "selected_token_indices")
258 + multi_modal_kwargs = metadata_dict.pop("multi_modal_kwargs")
259 attn_metadata = self.attn_backend.make_metadata(**metadata_dict)
260 sampling_metadata = SamplingMetadata(
261 seq_groups=None,
@@ -244,7 +268,7 @@ class XPUModelRunner(ModelRunnerBase[ModelInputForXPU]):
268 input_positions=input_positions,
269 attn_metadata=attn_metadata,
270 sampling_metadata=sampling_metadata,
247 - multi_modal_input=multi_modal_input)
271 + multi_modal_kwargs=multi_modal_kwargs)
272
273 def _prepare_decode(
274 self,
@@ -350,10 +374,8 @@ class XPUModelRunner(ModelRunnerBase[ModelInputForXPU]):
374 "positions": model_input.input_positions,
375 "kv_caches": kv_caches,
376 "attn_metadata": model_input.attn_metadata,
377 + **(model_input.multi_modal_kwargs or {}),
378 }
354 - if self.vision_language_config:
355 - execute_model_kwargs.update(
356 - {"image_input": model_input.multi_modal_input})
379
380 hidden_states = model_executable(**execute_model_kwargs)
381
@@ -376,13 +398,13 @@ class XPUModelRunner(ModelRunnerBase[ModelInputForXPU]):
398 self,
399 seq_group_metadata_list: List[SequenceGroupMetadata],
400 ) -> Tuple[torch.Tensor, torch.Tensor, AttentionMetadata, List[int],
379 - Optional[torch.Tensor]]:
401 + Mapping[str, BatchedTensors]]:
402 assert len(seq_group_metadata_list) > 0
403 input_tokens: List[int] = []
404 input_positions: List[int] = []
405 slot_mapping: List[int] = []
406 seq_lens: List[int] = []
385 - multi_modal_input_list: List[torch.Tensor] = []
407 + multi_modal_inputs_list: List[MultiModalInputs] = []
408
409 for seq_group_metadata in seq_group_metadata_list:
410 assert seq_group_metadata.is_prompt
@@ -403,9 +425,10 @@ class XPUModelRunner(ModelRunnerBase[ModelInputForXPU]):
425 # is always the first token in the sequence.
426 input_positions.extend(list(range(computed_len, seq_len)))
427
406 - if seq_group_metadata.multi_modal_data:
407 - multi_modal_input_list.append(
408 - seq_group_metadata.multi_modal_data.data)
428 + mm_data = seq_group_metadata.multi_modal_data
429 + if mm_data:
430 + mm_kwargs = self.multi_modal_input_mapper(mm_data)
431 + multi_modal_inputs_list.append(mm_kwargs)
432
433 if seq_group_metadata.block_tables is None:
434 # During memory profiling, the block tables are not initialized
@@ -435,15 +458,6 @@ class XPUModelRunner(ModelRunnerBase[ModelInputForXPU]):
458 slot = block_number * self.block_size + block_offset
459 slot_mapping.append(slot)
460
438 - if multi_modal_input_list:
439 - assert self.vision_language_config, (
440 - "Multi-modal inputs are only supported by "
441 - "vision language models.")
442 - multi_modal_input = torch.cat(multi_modal_input_list,
443 - dim=0).to(self.device)
444 - else:
445 - multi_modal_input = None
446 -
461 num_prompt_tokens = len(input_tokens)
462
463 input_tokens = torch.tensor(input_tokens,
@@ -475,5 +489,9 @@ class XPUModelRunner(ModelRunnerBase[ModelInputForXPU]):
489 num_decode_tokens=0,
490 block_tables=torch.tensor([], device=self.device, dtype=torch.int),
491 )
492 +
493 + multi_modal_kwargs = MultiModalInputs.batch(multi_modal_inputs_list,
494 + device=self.device)
495 +
496 return (input_tokens, input_positions, attn_metadata, seq_lens,
479 - multi_modal_input)
497 + multi_modal_kwargs)