A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 22:00 UTC
[Core] Dynamic image size support for VLMs (#5276)
Signed-off-by: Xiaowei Jiang <xwjiang2010@gmail.com> Co-authored-by: Xiaowei Jiang <xwjiang2010@gmail.com> Co-authored-by: ywang96 <ywang@roblox.com> Co-authored-by: xwjiang2010 <87673679+xwjiang2010@users.noreply.github.com> Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
Cyrus Leung committed
Jul 3, 2024 at 11:34 UTC
9831aec49fd541a8fe6cb65f0c3a65b03eccffe0
38 files changed
+1455
-666
docs/source/dev/input_processing/model_inputs_index.rst
+1
-1
@@ -8,7 +8,7 @@ Input Processing
8
vLLM provides a mechanism for defining input processors for each model so that the inputs are processed
9
in :class:`~vllm.LLMEngine` before they are passed to model executors.
10
11
-Currently, this mechanism is only utilized in **multi-modal models** for preprocessing multi-modal input
11
+Currently, this mechanism is only utilized in :ref:`multi-modal models <multi_modality>` for preprocessing multi-modal input
12
data in addition to input prompt, but it can be extended to text-only language models when needed.
13
14
Guides
docs/source/dev/multimodal/adding_multimodal_model.rst
new
+124
@@ -0,0 +1,124 @@
1
+.. _adding_a_new_multimodal_model:
2
+
3
+Adding a New Multimodal Model
4
+=============================
5
+
6
+This document provides a high-level guide on integrating a :ref:`multi-modal model <multi_modality>` into vLLM.
7
+
8
+.. note::
9
+ The complexity of adding a new model depends heavily on the model's architecture.
10
+ The process is considerably straightforward if the model shares a similar architecture with an existing model in vLLM.
11
+ However, for models that include new operators (e.g., a new attention mechanism), the process can be a bit more complex.
12
+
13
+.. tip::
14
+ If you are encountering issues while integrating your model into vLLM, feel free to open an issue on our `GitHub <https://github.com/vllm-project/vllm/issues>`_ repository.
15
+ We will be happy to help you out!
16
+
17
+
18
+1. Set up the base vLLM model
19
+-----------------------------
20
+
21
+As usual, follow :ref:`these steps <adding_a_new_model>` to implement the model in vLLM, but note the following:
22
+
23
+- You should additionally implement the :class:`~vllm.model_executor.models.interfaces.SupportsVision` interface.
24
+
25
+ .. code-block:: diff
26
+
27
+ + from vllm.model_executor.models.interfaces import SupportsVision
28
+
29
+ - class YourModelForImage2Seq(nn.Module):
30
+ + class YourModelForImage2Seq(nn.Module, SupportsVision):
31
+
32
+ .. note::
33
+ The model class does not have to be named :code:`*ForCausalLM`.
34
+ Check out `the HuggingFace Transformers documentation <https://huggingface.co/docs/transformers/model_doc/auto#multimodal>`__ for some examples.
35
+
36
+- While implementing the :meth:`~torch.nn.Module.forward` method, reserve a keyword parameter
37
+ for each input tensor that corresponds to a multi-modal input, as shown in the following example:
38
+
39
+ .. code-block:: diff
40
+
41
+ def forward(
42
+ self,
43
+ input_ids: torch.Tensor,
44
+ positions: torch.Tensor,
45
+ kv_caches: List[torch.Tensor],
46
+ attn_metadata: AttentionMetadata,
47
+ + pixel_values: torch.Tensor,
48
+ ) -> SamplerOutput:
49
+
50
+
51
+2. Register input mappers
52
+-------------------------
53
+
54
+For each modality type to support, decorate the model class with :meth:`MULTIMODAL_REGISTRY.register_input_mapper <vllm.multimodal.MultiModalRegistry.register_input_mapper>`.
55
+This decorator accepts a function that maps multi-modal inputs to the keyword arguments you have previously defined in :meth:`~torch.nn.Module.forward`.
56
+
57
+.. code-block:: diff
58
+
59
+ from vllm.model_executor.models.interfaces import SupportsVision
60
+ + from vllm.multimodal import MULTIMODAL_REGISTRY
61
+
62
+ + @MULTIMODAL_REGISTRY.register_image_feature_input_mapper()
63
+ + @MULTIMODAL_REGISTRY.register_image_pixel_input_mapper()
64
+ class YourModelForImage2Seq(nn.Module, SupportsVision):
65
+
66
+A default mapper is available for each modality in the core vLLM library. This input mapper will be used if you do not provide your own function.
67
+
68
+.. seealso::
69
+ :ref:`input_processing_pipeline`
70
+
71
+
72
+3. (Optional) Register dummy data
73
+---------------------------------
74
+
75
+During startup, dummy data is passed to the vLLM model to allocate memory. This only consists of text input by default, which may not be applicable to multi-modal models.
76
+In such cases, you can define your own dummy data by registering a factory method via :meth:`INPUT_REGISTRY.register_dummy_data <vllm.inputs.registry.InputRegistry.register_dummy_data>`.
77
+
78
+.. code-block:: diff
79
+
80
+ from vllm.inputs import INPUT_REGISTRY
81
+ from vllm.model_executor.models.interfaces import SupportsVision
82
+ from vllm.multimodal import MULTIMODAL_REGISTRY
83
+
84
+ @MULTIMODAL_REGISTRY.register_image_feature_input_mapper()
85
+ @MULTIMODAL_REGISTRY.register_image_pixel_input_mapper()
86
+ + @INPUT_REGISTRY.register_dummy_data(<your_dummy_data_factory>)
87
+ class YourModelForImage2Seq(nn.Module, SupportsVision):
88
+
89
+Here are some examples:
90
+
91
+- Image inputs (static feature size): `LLaVA-1.5 Model <https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/llava.py>`__
92
+- Image inputs (dynamic feature size): `LLaVA-NeXT Model <https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/llava_next.py>`__
93
+
94
+.. seealso::
95
+ :ref:`input_processing_pipeline`
96
+
97
+
98
+4. (Optional) Register input processor
99
+--------------------------------------
100
+
101
+Sometimes, there is a need to process inputs at the :class:`~vllm.LLMEngine` level before they are passed to the model executor.
102
+This is often due to the fact that unlike implementations in HuggingFace Transformers, the reshaping and/or expansion of multi-modal embeddings needs to take place outside model's :meth:`~torch.nn.Module.forward` call.
103
+You can register input processors via :meth:`INPUT_REGISTRY.register_input_processor <vllm.inputs.registry.InputRegistry.register_input_processor>`.
104
+
105
+.. code-block:: diff
106
+
107
+ from vllm.inputs import INPUT_REGISTRY
108
+ from vllm.model_executor.models.interfaces import SupportsVision
109
+ from vllm.multimodal import MULTIMODAL_REGISTRY
110
+
111
+ @MULTIMODAL_REGISTRY.register_image_feature_input_mapper()
112
+ @MULTIMODAL_REGISTRY.register_image_pixel_input_mapper()
113
+ @INPUT_REGISTRY.register_dummy_data(<your_dummy_data_factory>)
114
+ + @INPUT_REGISTRY.register_input_processor(<your_input_processor>)
115
+ class YourModelForImage2Seq(nn.Module, SupportsVision):
116
+
117
+A common use case of input processors is inserting placeholder tokens to leverage the vLLM framework for attention mask generation.
118
+Here are some examples:
119
+
120
+- Insert static number of image tokens: `LLaVA-1.5 Model <https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/llava.py>`__
121
+- Insert dynamic number of image tokens: `LLaVA-NeXT Model <https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/llava_next.py>`__
122
+
123
+.. seealso::
124
+ :ref:`input_processing_pipeline`
docs/source/dev/multimodal/multimodal_index.rst
+15
-3
@@ -1,3 +1,5 @@
1
+.. _multi_modality:
2
+
3
Multi-Modality
4
==============
5
@@ -8,12 +10,18 @@ vLLM provides experimental support for multi-modal models through the :mod:`vllm
10
:class:`vllm.inputs.PromptStrictInputs` accepts an additional attribute ``multi_modal_data``
11
which allows you to pass in multi-modal input alongside text and token prompts.
12
11
-By default, vLLM models do not support multi-modal inputs. To enable multi-modal support for a model,
12
-you must decorate the model class with :meth:`InputRegistry.register_dummy_data <vllm.inputs.registry.InputRegistry.register_dummy_data>`,
13
-as well as :meth:`MULTIMODAL_REGISTRY.register_input_mapper <MultiModalRegistry.register_input_mapper>` for each modality type to support.
13
+By default, vLLM models do not support multi-modal inputs. To enable multi-modal support for a model, please follow :ref:`the guide for adding a new multimodal model. <adding_a_new_multimodal_model>`.
14
15
# TODO: Add more instructions on how to do that once embeddings is in.
16
17
+Guides
18
+++++++
19
+
20
+.. toctree::
21
+ :maxdepth: 1
22
+
23
+ adding_multimodal_model
24
+
25
Module Contents
26
+++++++++++++++
27
@@ -35,6 +43,10 @@ Base Classes
43
:members:
44
:show-inheritance:
45
46
+.. autoclass:: vllm.multimodal.MultiModalInputs
47
+ :members:
48
+ :show-inheritance:
49
+
50
.. autoclass:: vllm.multimodal.MultiModalPlugin
51
:members:
52
:show-inheritance:
docs/source/models/vlm.rst
+16
-8
@@ -23,7 +23,6 @@ The following :ref:`engine arguments <engine_args>` are specific to VLMs:
23
Currently, the support for vision language models on vLLM has the following limitations:
24
25
* Only single image input is supported per text prompt.
26
- * Dynamic ``image_input_shape`` is not supported: the input image will be resized to the static ``image_input_shape``. This means our LLaVA-NeXT output may not exactly match the huggingface implementation.
26
27
We are continuously improving user & developer experience for VLMs. Please `open an issue on GitHub <https://github.com/vllm-project/vllm/issues/new/choose>`_ if you have any feedback or feature requests.
28
@@ -42,12 +41,17 @@ To initialize a VLM, the aforementioned arguments must be passed to the ``LLM``
41
)
42
43
.. important::
44
+ Currently, you have to specify ``image_feature_size`` to support memory profiling.
45
+ To avoid OOM during runtime, you should set this to the maximum value supported by the model.
46
+ The calculation of feature size is specific to the model. For more details, please refer to
47
+ the function :code:`get_<model_name>_image_feature_size` inside the corresponding model file.
48
+
49
We will remove most of the vision-specific arguments in a future release as they can be inferred from the HuggingFace configuration.
50
51
52
To pass an image to the model, note the following in :class:`vllm.inputs.PromptStrictInputs`:
53
50
-* ``prompt``: The prompt should have a number of ``<image>`` tokens equal to ``image_feature_size``.
54
+* ``prompt``: The prompt should follow the format that is documented on HuggingFace.
55
* ``multi_modal_data``: This is a dictionary that follows the schema defined in :class:`vllm.multimodal.MultiModalDataDict`.
56
57
.. note::
@@ -57,8 +61,8 @@ To pass an image to the model, note the following in :class:`vllm.inputs.PromptS
61
62
.. code-block:: python
63
60
- prompt = "<image>" * 576 + (
61
- "\nUSER: What is the content of this image?\nASSISTANT:")
64
+ # Refer to the HuggingFace repo for the correct format to use
65
+ prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:"
66
67
# Load the image using PIL.Image
68
image = ...
@@ -74,8 +78,6 @@ To pass an image to the model, note the following in :class:`vllm.inputs.PromptS
78
79
A code example can be found in `examples/llava_example.py <https://github.com/vllm-project/vllm/blob/main/examples/llava_example.py>`_.
80
77
-.. important::
78
- We will remove the need to format image tokens in a future release. Afterwards, the input text will follow the same format as that for the original HuggingFace model.
81
82
Online OpenAI Vision API Compatible Inference
83
----------------------------------------------
@@ -103,6 +105,11 @@ Below is an example on how to launch the same ``llava-hf/llava-1.5-7b-hf`` with
105
--chat-template template_llava.jinja
106
107
.. important::
108
+ Currently, you have to specify ``image_feature_size`` to support memory profiling.
109
+ To avoid OOM during runtime, you should set this to the maximum value supported by the model.
110
+ The calculation of feature size is specific to the model. For more details, please refer to
111
+ the function :code:`get_<model_name>_image_feature_size` inside the corresponding model file.
112
+
113
We will remove most of the vision-specific arguments in a future release as they can be inferred from the HuggingFace configuration.
114
115
To consume the server, you can use the OpenAI client like in the example below:
@@ -121,6 +128,8 @@ To consume the server, you can use the OpenAI client like in the example below:
128
messages=[{
129
"role": "user",
130
"content": [
131
+ # NOTE: The prompt formatting with the image token `<image>` is not needed
132
+ # since the prompt will be processed automatically by the API server.
133
{"type": "text", "text": "What's in this image?"},
134
{
135
"type": "image_url",
@@ -144,5 +153,4 @@ A full code example can be found in `examples/openai_vision_api_client.py <https
153
export VLLM_IMAGE_FETCH_TIMEOUT=<timeout>
154
155
.. note::
147
- The prompt formatting with the image token ``<image>`` is not needed when serving VLMs with the API server since the prompt will be
148
- processed automatically by the server.
156
+ There is no need to format the prompt in the API request since it will be handled by the server.
examples/llava_example.py
+1
-2
@@ -17,8 +17,7 @@ def run_llava():
17
image_feature_size=576,
18
)
19
20
- prompt = "<image>" * 576 + (
21
- "\nUSER: What is the content of this image?\nASSISTANT:")
20
+ prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:"
21
22
image = Image.open("images/stop_sign.jpg")
23
examples/llava_next_example.py
+3
-8
@@ -5,22 +5,17 @@ from PIL import Image
5
6
from vllm import LLM, SamplingParams
7
8
-# Dynamic image input is currently not supported and therefore
9
-# a fixed image input shape and its corresponding feature size is required.
10
-# See https://github.com/vllm-project/vllm/pull/4199 for the complete
11
-# configuration matrix.
12
-
8
9
def run_llava_next():
10
llm = LLM(
11
model="llava-hf/llava-v1.6-mistral-7b-hf",
12
image_token_id=32000,
13
image_input_shape="1,3,336,336",
19
- image_feature_size=1176,
14
+ # Use the maximum possible value for memory profiling
15
+ image_feature_size=2928,
16
)
17
22
- prompt = "[INST] " + "<image>" * 1176 + (
23
- "\nWhat is shown in this image? [/INST]")
18
+ prompt = "[INST] <image>\nWhat is shown in this image? [/INST]"
19
url = "https://h2o-release.s3.amazonaws.com/h2ogpt/bigben.jpg"
20
image = Image.open(BytesIO(requests.get(url).content))
21
sampling_params = SamplingParams(temperature=0.8,
examples/phi3v_example.py
+5
-3
@@ -5,6 +5,9 @@ from PIL import Image
5
6
from vllm import LLM, SamplingParams
7
8
+# The assets are located at `s3://air-example-data-2/vllm_opensource_llava/`.
9
+# You can use `.buildkite/download-images.sh` to download them
10
+
11
12
def run_phi3v():
13
model_path = "microsoft/Phi-3-vision-128k-instruct"
@@ -18,7 +21,8 @@ def run_phi3v():
21
trust_remote_code=True,
22
image_token_id=32044,
23
image_input_shape="1,3,1008,1344",
21
- image_feature_size=1921,
24
+ # Use the maximum possible value for memory profiling
25
+ image_feature_size=2653,
26
max_num_seqs=5,
27
)
28
@@ -26,8 +30,6 @@ def run_phi3v():
30
31
# single-image prompt
32
prompt = "<|user|>\n<|image_1|>\nWhat is the season?<|end|>\n<|assistant|>\n" # noqa: E501
29
- prompt = prompt.replace("<|image_1|>", "<|image|>" * 1921 + "<s>")
30
-
33
sampling_params = SamplingParams(temperature=0, max_tokens=64)
34
35
outputs = llm.generate(
tests/conftest.py
+81
-33
@@ -1,12 +1,13 @@
1
import contextlib
2
import gc
3
import os
4
+import sys
5
from collections import UserList
6
from dataclasses import dataclass
7
from functools import cached_property
8
from pathlib import Path
8
-from typing import (TYPE_CHECKING, Any, Dict, List, Literal, Optional, Tuple,
9
- TypedDict, TypeVar)
9
+from typing import (Any, Dict, List, Literal, Optional, Tuple, TypedDict,
10
+ TypeVar)
11
12
import pytest
13
import torch
@@ -22,13 +23,10 @@ from vllm.distributed import (destroy_distributed_environment,
23
destroy_model_parallel)
24
from vllm.inputs import TextPrompt
25
from vllm.logger import init_logger
26
+from vllm.multimodal.utils import fetch_image
27
from vllm.sequence import SampleLogprobs
28
from vllm.utils import cuda_device_count_stateless, is_cpu
29
28
-if TYPE_CHECKING:
29
- # it will call torch.cuda.device_count()
30
- from vllm.multimodal import MultiModalDataDict
31
-
30
logger = init_logger(__name__)
31
32
_TEST_DIR = os.path.dirname(__file__)
@@ -47,30 +45,42 @@ def _read_prompts(filename: str) -> List[str]:
45
46
@dataclass(frozen=True)
47
class ImageAsset:
50
- name: Literal["stop_sign", "cherry_blossom"]
48
+ name: Literal["stop_sign", "cherry_blossom", "boardwalk"]
49
50
@cached_property
51
def pil_image(self) -> Image.Image:
54
- return Image.open(_IMAGE_DIR / f"{self.name}.jpg")
55
-
56
- def for_hf(self) -> Image.Image:
57
- return self.pil_image
52
+ if self.name == "boardwalk":
53
+ return fetch_image(
54
+ "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
55
+ )
56
59
- def for_vllm(self) -> Dict[str, Any]:
60
- return {"image": self.pil_image}
57
+ return Image.open(_IMAGE_DIR / f"{self.name}.jpg")
58
59
60
class _ImageAssetPrompts(TypedDict):
61
stop_sign: str
62
cherry_blossom: str
63
+ boardwalk: str
64
+
65
+
66
+if sys.version_info < (3, 9):
67
+ # UserList cannot be subscripted
68
+ class _ImageAssetsBase(UserList):
69
+ pass
70
+else:
71
72
+ class _ImageAssetsBase(UserList[ImageAsset]):
73
+ pass
74
68
-class _ImageAssets(UserList):
75
+
76
+class _ImageAssets(_ImageAssetsBase):
77
78
def __init__(self) -> None:
71
- super().__init__(
72
- [ImageAsset("stop_sign"),
73
- ImageAsset("cherry_blossom")])
79
+ super().__init__([
80
+ ImageAsset("stop_sign"),
81
+ ImageAsset("cherry_blossom"),
82
+ ImageAsset("boardwalk")
83
+ ])
84
85
def prompts(self, prompts: _ImageAssetPrompts) -> List[str]:
86
"""
@@ -79,7 +89,10 @@ class _ImageAssets(UserList):
89
The order of the returned prompts matches the order of the
90
assets when iterating through this object.
91
"""
82
- return [prompts["stop_sign"], prompts["cherry_blossom"]]
92
+ return [
93
+ prompts["stop_sign"], prompts["cherry_blossom"],
94
+ prompts["boardwalk"]
95
+ ]
96
97
98
IMAGE_ASSETS = _ImageAssets()
@@ -220,7 +233,7 @@ class HfRunner:
233
self,
234
prompts: List[str],
235
images: Optional[List[Image.Image]] = None,
223
- **kwargs,
236
+ **kwargs: Any,
237
) -> List[Tuple[List[List[int]], List[str]]]:
238
if images:
239
assert len(prompts) == len(images)
@@ -255,7 +268,7 @@ class HfRunner:
268
prompts: List[str],
269
max_tokens: int,
270
images: Optional[List[Image.Image]] = None,
258
- **kwargs,
271
+ **kwargs: Any,
272
) -> List[Tuple[List[int], str]]:
273
outputs = self.generate(prompts,
274
do_sample=False,
@@ -291,19 +304,30 @@ class HfRunner:
304
self,
305
prompts: List[str],
306
max_tokens: int,
307
+ images: Optional[List[Image.Image]] = None,
308
+ **kwargs: Any,
309
) -> List[List[torch.Tensor]]:
295
- all_logprobs = []
296
- for prompt in prompts:
297
- input_ids = self.tokenizer(prompt, return_tensors="pt").input_ids
310
+ all_logprobs: List[List[torch.Tensor]] = []
311
+ for i, prompt in enumerate(prompts):
312
+ processor_kwargs: Dict[str, Any] = {
313
+ "text": prompt,
314
+ "return_tensors": "pt",
315
+ }
316
+ if images is not None and images[i] is not None:
317
+ processor_kwargs["images"] = images[i]
318
+
319
+ inputs = self.processor(**processor_kwargs)
320
+
321
output = self.model.generate(
299
- self.wrap_device(input_ids),
322
+ **self.wrap_device(inputs),
323
use_cache=True,
324
do_sample=False,
325
max_new_tokens=max_tokens,
326
output_hidden_states=True,
327
return_dict_in_generate=True,
328
+ **kwargs,
329
)
306
- seq_logprobs = []
330
+ seq_logprobs: List[torch.Tensor] = []
331
for hidden_states in output.hidden_states:
332
last_hidden_states = hidden_states[-1][0]
333
logits = torch.matmul(
@@ -323,20 +347,32 @@ class HfRunner:
347
prompts: List[str],
348
max_tokens: int,
349
num_logprobs: int,
350
+ images: Optional[List[Image.Image]] = None,
351
+ **kwargs: Any,
352
) -> List[Tuple[List[int], str, List[Dict[int, float]]]]:
353
all_logprobs: List[List[Dict[int, float]]] = []
354
all_output_ids: List[List[int]] = []
355
all_output_strs: List[str] = []
356
331
- for prompt in prompts:
332
- input_ids = self.tokenizer(prompt, return_tensors="pt").input_ids
357
+ for i, prompt in enumerate(prompts):
358
+ processor_kwargs: Dict[str, Any] = {
359
+ "text": prompt,
360
+ "return_tensors": "pt",
361
+ }
362
+ if images is not None and images[i] is not None:
363
+ processor_kwargs["images"] = images[i]
364
+
365
+ inputs = self.processor(**processor_kwargs)
366
+ input_ids = inputs.input_ids
367
+
368
output = self.model.generate(
334
- self.wrap_device(input_ids),
369
+ **self.wrap_device(inputs),
370
use_cache=True,
371
do_sample=False,
372
max_new_tokens=max_tokens,
373
output_hidden_states=True,
374
return_dict_in_generate=True,
375
+ **kwargs,
376
)
377
378
seq_logprobs: List[torch.Tensor] = []
@@ -431,7 +467,7 @@ class VllmRunner:
467
self,
468
prompts: List[str],
469
sampling_params: SamplingParams,
434
- images: Optional[List["MultiModalDataDict"]] = None,
470
+ images: Optional[List[Image.Image]] = None,
471
) -> List[Tuple[List[List[int]], List[str]]]:
472
if images is not None:
473
assert len(prompts) == len(images)
@@ -439,7 +475,7 @@ class VllmRunner:
475
inputs = [TextPrompt(prompt=prompt) for prompt in prompts]
476
if images is not None:
477
for i, image in enumerate(images):
442
- inputs[i]["multi_modal_data"] = image
478
+ inputs[i]["multi_modal_data"] = {"image": image}
479
480
req_outputs = self.model.generate(inputs,
481
sampling_params=sampling_params)
@@ -462,10 +498,19 @@ class VllmRunner:
498
self,
499
prompts: List[str],
500
sampling_params: SamplingParams,
501
+ images: Optional[List[Image.Image]] = None,
502
) -> List[Tuple[List[int], str, Optional[SampleLogprobs]]]:
503
assert sampling_params.logprobs is not None
504
468
- req_outputs = self.model.generate(prompts,
505
+ if images is not None:
506
+ assert len(prompts) == len(images)
507
+
508
+ inputs = [TextPrompt(prompt=prompt) for prompt in prompts]
509
+ if images is not None:
510
+ for i, image in enumerate(images):
511
+ inputs[i]["multi_modal_data"] = {"image": image}
512
+
513
+ req_outputs = self.model.generate(inputs,
514
sampling_params=sampling_params)
515
outputs: List[Tuple[List[int], str, Optional[SampleLogprobs]]] = []
516
for req_output in req_outputs:
@@ -480,7 +525,7 @@ class VllmRunner:
525
self,
526
prompts: List[str],
527
max_tokens: int,
483
- images: Optional[List["MultiModalDataDict"]] = None,
528
+ images: Optional[List[Image.Image]] = None,
529
) -> List[Tuple[List[int], str]]:
530
greedy_params = SamplingParams(temperature=0.0, max_tokens=max_tokens)
531
outputs = self.generate(prompts, greedy_params, images=images)
@@ -492,11 +537,14 @@ class VllmRunner:
537
prompts: List[str],
538
max_tokens: int,
539
num_logprobs: int,
540
+ images: Optional[List[Image.Image]] = None,
541
) -> List[Tuple[List[int], str, Optional[SampleLogprobs]]]:
542
greedy_logprobs_params = SamplingParams(temperature=0.0,
543
max_tokens=max_tokens,
544
logprobs=num_logprobs)
499
- outputs = self.generate_w_logprobs(prompts, greedy_logprobs_params)
545
+ outputs = self.generate_w_logprobs(prompts,
546
+ greedy_logprobs_params,
547
+ images=images)
548
549
return [(output_ids, output_str, output_logprobs)
550
for output_ids, output_str, output_logprobs in outputs]
tests/distributed/test_multimodal_broadcast.py
+5
-2
@@ -30,9 +30,10 @@ else:
30
@pytest.mark.parametrize("tensor_parallel_size", [2])
31
@pytest.mark.parametrize("dtype", ["half"])
32
@pytest.mark.parametrize("max_tokens", [128])
33
+@pytest.mark.parametrize("num_logprobs", [5])
34
def test_models(hf_runner, vllm_runner, image_assets,
34
- tensor_parallel_size: int, dtype: str,
35
- max_tokens: int) -> None:
35
+ tensor_parallel_size: int, dtype: str, max_tokens: int,
36
+ num_logprobs: int) -> None:
37
if cuda_device_count_stateless() < tensor_parallel_size:
38
pytest.skip(
39
f"Need at least {tensor_parallel_size} GPUs to run the test.")
@@ -44,8 +45,10 @@ def test_models(hf_runner, vllm_runner, image_assets,
45
vllm_runner,
46
image_assets,
47
model_and_config=model_and_vl_config[0],
48
+ size_factors=[1.0],
49
dtype=dtype,
50
max_tokens=max_tokens,
51
+ num_logprobs=num_logprobs,
52
tensor_parallel_size=tensor_parallel_size,
53
distributed_executor_backend=distributed_executor_backend,
54
)
tests/models/test_llava.py
+71
-35
@@ -4,18 +4,21 @@ import pytest
4
from transformers import AutoTokenizer
5
6
from vllm.config import VisionLanguageConfig
7
+from vllm.multimodal.utils import rescale_image_size
8
+from vllm.sequence import SampleLogprobs
9
10
from ..conftest import IMAGE_ASSETS, HfRunner, VllmRunner, _ImageAssets
9
-from .utils import check_outputs_equal
11
+from .utils import check_logprobs_close
12
13
pytestmark = pytest.mark.vlm
14
13
-# The image token is placed before "user" on purpose so that the test can pass
15
HF_IMAGE_PROMPTS = IMAGE_ASSETS.prompts({
16
"stop_sign":
16
- "<image>\nUSER: What's the content of the image?\nASSISTANT:",
17
+ "USER: <image>\nWhat's the content of the image?\nASSISTANT:",
18
"cherry_blossom":
18
- "<image>\nUSER: What is the season?\nASSISTANT:",
19
+ "USER: <image>\nWhat is the season?\nASSISTANT:",
20
+ "boardwalk":
21
+ "USER: <image>\nWhat's in this image?\nASSISTANT:",
22
})
23
24
@@ -37,27 +40,34 @@ model_and_vl_config = [
40
]
41
42
40
-def vllm_to_hf_output(vllm_output: Tuple[List[int], str],
43
+def vllm_to_hf_output(vllm_output: Tuple[List[int], str,
44
+ Optional[SampleLogprobs]],
45
vlm_config: VisionLanguageConfig, model_id: str):
46
"""Sanitize vllm output to be comparable with hf output.
47
The function reduces `input_ids` from 1, 32000, 32000, ..., 32000,
48
x1, x2, x3 ... to 1, 32000, x1, x2, x3 ...
49
It also reduces `output_str` from "<image><image>bla" to "bla".
50
"""
47
- output_ids, output_str = vllm_output
51
+ output_ids, output_str, out_logprobs = vllm_output
52
image_token_id = vlm_config.image_token_id
53
54
tokenizer = AutoTokenizer.from_pretrained(model_id)
55
image_token_str = tokenizer.decode(image_token_id)
56
+ eos_token_id = tokenizer.eos_token_id
57
58
hf_output_ids = [
59
token_id for idx, token_id in enumerate(output_ids)
60
if token_id != image_token_id or output_ids[idx - 1] != image_token_id
61
]
62
+
63
hf_output_str = output_str \
64
.replace(image_token_str * vlm_config.image_feature_size, "")
65
+ assert hf_output_str[0] == " "
66
+ hf_output_str = hf_output_str[1:]
67
+ if hf_output_ids[-1] == eos_token_id:
68
+ hf_output_str = hf_output_str + tokenizer.decode(eos_token_id)
69
60
- return hf_output_ids, hf_output_str
70
+ return hf_output_ids, hf_output_str, out_logprobs
71
72
73
def run_test(
@@ -66,8 +76,10 @@ def run_test(
76
image_assets: _ImageAssets,
77
model_and_config: Tuple[str, VisionLanguageConfig],
78
*,
79
+ size_factors: List[float],
80
dtype: str,
81
max_tokens: int,
82
+ num_logprobs: int,
83
tensor_parallel_size: int,
84
distributed_executor_backend: Optional[str] = None,
85
):
@@ -81,61 +93,85 @@ def run_test(
93
The text output is sanitized to be able to compare with hf.
94
"""
95
model_id, vlm_config = model_and_config
84
- hf_images = [asset.for_hf() for asset in image_assets]
96
+ images = [asset.pil_image for asset in image_assets]
97
+
98
+ inputs_per_image = [(
99
+ [prompt for _ in size_factors],
100
+ [rescale_image_size(image, factor) for factor in size_factors],
101
+ ) for image, prompt in zip(images, HF_IMAGE_PROMPTS)]
102
103
# NOTE: take care of the order. run vLLM first, and then run HF.
104
# vLLM needs a fresh new process without cuda initialization.
105
# if we run HF first, the cuda initialization will be done and it
106
# will hurt multiprocessing backend with fork method (the default method).
107
108
+ # max_model_len should be greater than image_feature_size
109
with vllm_runner(model_id,
110
dtype=dtype,
111
tensor_parallel_size=tensor_parallel_size,
112
distributed_executor_backend=distributed_executor_backend,
113
enforce_eager=True,
114
**vlm_config.as_cli_args_dict()) as vllm_model:
97
-
98
- # NOTE: `asset.for_vllm` will call `torch.cuda.device_count()`
99
- # we must put it inside the vllm_runner context manager
100
- # i.e. after creating vLLM instance.
101
- vllm_images = [asset.for_vllm() for asset in image_assets]
102
-
103
- vllm_image_prompts = [
104
- p.replace("<image>", "<image>" * vlm_config.image_feature_size)
105
- for p in HF_IMAGE_PROMPTS
115
+ vllm_outputs_per_image = [
116
+ vllm_model.generate_greedy_logprobs(prompts,
117
+ max_tokens,
118
+ num_logprobs=num_logprobs,
119
+ images=images)
120
+ for prompts, images in inputs_per_image
121
]
122
108
- vllm_outputs = vllm_model.generate_greedy(vllm_image_prompts,
109
- max_tokens,
110
- images=vllm_images)
111
-
123
with hf_runner(model_id, dtype=dtype, is_vision_model=True) as hf_model:
113
- hf_outputs = hf_model.generate_greedy(HF_IMAGE_PROMPTS,
114
- max_tokens,
115
- images=hf_images)
116
-
117
- check_outputs_equal(
118
- hf_outputs,
119
- [
120
- vllm_to_hf_output(vllm_output, vlm_config, model_id)
121
- for vllm_output in vllm_outputs
122
- ],
123
- name_0="hf",
124
- name_1="vllm",
125
- )
124
+ hf_outputs_per_image = [
125
+ hf_model.generate_greedy_logprobs_limit(prompts,
126
+ max_tokens,
127
+ num_logprobs=num_logprobs,
128
+ images=images)
129
+ for prompts, images in inputs_per_image
130
+ ]
131
+
132
+ for hf_outputs, vllm_outputs in zip(hf_outputs_per_image,
133
+ vllm_outputs_per_image):
134
+ # TODO: Check whether using original CLIPVisionModel can improve
135
+ # consistency against HF
136
+ check_logprobs_close(
137
+ outputs_0_lst=hf_outputs,
138
+ outputs_1_lst=[
139
+ vllm_to_hf_output(vllm_output, vlm_config, model_id)
140
+ for vllm_output in vllm_outputs
141
+ ],
142
+ name_0="hf",
143
+ name_1="vllm",
144
+ )
145
146
147
@pytest.mark.parametrize("model_and_config", model_and_vl_config)
148
+@pytest.mark.parametrize(
149
+ "size_factors",
150
+ [
151
+ # No image
152
+ [],
153
+ # Single-scale
154
+ [1.0],
155
+ # Single-scale, batched
156
+ [1.0, 1.0, 1.0],
157
+ # Multi-scale
158
+ [0.25, 0.5, 1.0],
159
+ ],
160
+)
161
@pytest.mark.parametrize("dtype", ["half"])
162
@pytest.mark.parametrize("max_tokens", [128])
163
+@pytest.mark.parametrize("num_logprobs", [5])
164
def test_models(hf_runner, vllm_runner, image_assets, model_and_config,
132
- dtype: str, max_tokens: int) -> None:
165
+ size_factors, dtype: str, max_tokens: int,
166
+ num_logprobs: int) -> None:
167
run_test(
168
hf_runner,
169
vllm_runner,
170
image_assets,
171
model_and_config,
172
+ size_factors=size_factors,
173
dtype=dtype,
174
max_tokens=max_tokens,
175
+ num_logprobs=num_logprobs,
176
tensor_parallel_size=1,
177
)
tests/models/test_llava_next.py
+79
-50
@@ -1,12 +1,15 @@
1
-from typing import List, Tuple
1
+import re
2
+from typing import List, Optional, Tuple
3
4
import pytest
5
from transformers import AutoTokenizer
6
7
from vllm.config import VisionLanguageConfig
8
+from vllm.multimodal.utils import rescale_image_size
9
+from vllm.sequence import SampleLogprobs
10
11
from ..conftest import IMAGE_ASSETS
9
-from .utils import check_outputs_equal
12
+from .utils import check_logprobs_close
13
14
pytestmark = pytest.mark.vlm
15
@@ -15,21 +18,20 @@ _PREFACE = (
18
"The assistant gives helpful, detailed, and polite answers to the human's "
19
"questions.")
20
18
-# The image token is placed before "user" on purpose so that the test can pass
21
HF_IMAGE_PROMPTS = IMAGE_ASSETS.prompts({
22
"stop_sign":
21
- f"{_PREFACE} <image>\nUSER: What's the content of the image?\nASSISTANT:",
23
+ f"{_PREFACE} USER: <image>\nWhat's the content of the image? ASSISTANT:",
24
"cherry_blossom":
23
- f"{_PREFACE} <image>\nUSER: What is the season?\nASSISTANT:",
25
+ f"{_PREFACE} USER: <image>\nWhat is the season? ASSISTANT:",
26
+ "boardwalk":
27
+ f"{_PREFACE} USER: <image>\nWhat's in this image? ASSISTANT:",
28
})
29
30
31
def iter_llava_next_configs(model_name: str):
32
+ # Need to use the max possible feature size for profile_run
33
image_hw_to_feature_size = {
29
- (336, 336): 1176,
30
- (672, 672): 2928,
31
- (1344, 336): 1944,
32
- (336, 1344): 1890,
34
+ (336, 336): 2928,
35
}
36
37
for (h, w), f in image_hw_to_feature_size.items():
@@ -47,37 +49,55 @@ model_and_vl_config = [
49
]
50
51
50
-def vllm_to_hf_output(vllm_output: Tuple[List[int], str],
52
+def vllm_to_hf_output(vllm_output: Tuple[List[int], str,
53
+ Optional[SampleLogprobs]],
54
vlm_config: VisionLanguageConfig, model_id: str):
55
"""Sanitize vllm output to be comparable with hf output.
56
The function reduces `input_ids` from 1, 32000, 32000, ..., 32000,
57
x1, x2, x3 ... to 1, 32000, x1, x2, x3 ...
58
It also reduces `output_str` from "<image><image>bla" to "bla".
59
"""
57
- output_ids, output_str = vllm_output
60
+ output_ids, output_str, out_logprobs = vllm_output
61
image_token_id = vlm_config.image_token_id
62
63
tokenizer = AutoTokenizer.from_pretrained(model_id)
64
image_token_str = tokenizer.decode(image_token_id)
65
+ eos_token_id = tokenizer.eos_token_id
66
67
hf_output_ids = [
68
token_id for idx, token_id in enumerate(output_ids)
69
if token_id != image_token_id or output_ids[idx - 1] != image_token_id
70
]
67
- hf_output_str = output_str \
68
- .replace(image_token_str * vlm_config.image_feature_size, " ")
71
70
- return hf_output_ids, hf_output_str
72
+ hf_output_str = re.sub(fr"({image_token_str})+", "", output_str)
73
+ assert hf_output_str[0] == " "
74
+ hf_output_str = hf_output_str[1:]
75
+ if hf_output_ids[-1] == eos_token_id:
76
+ hf_output_str = hf_output_str + tokenizer.decode(eos_token_id)
77
+
78
+ return hf_output_ids, hf_output_str, out_logprobs
79
80
73
-@pytest.mark.xfail(
74
- reason="Inconsistent image processor being used due to lack "
75
- "of support for dynamic image token replacement")
81
@pytest.mark.parametrize("model_and_config", model_and_vl_config)
82
+@pytest.mark.parametrize(
83
+ "size_factors",
84
+ [
85
+ # No image
86
+ [],
87
+ # Single-scale
88
+ [1.0],
89
+ # Single-scale, batched
90
+ [1.0, 1.0, 1.0],
91
+ # Multi-scale
92
+ [0.25, 0.5, 1.0],
93
+ ],
94
+)
95
@pytest.mark.parametrize("dtype", ["half"])
96
@pytest.mark.parametrize("max_tokens", [128])
97
+@pytest.mark.parametrize("num_logprobs", [5])
98
def test_models(hf_runner, vllm_runner, image_assets, model_and_config,
80
- dtype: str, max_tokens: int) -> None:
99
+ size_factors, dtype: str, max_tokens: int,
100
+ num_logprobs: int) -> None:
101
"""Inference result should be the same between hf and vllm.
102
103
All the image fixtures for the test is under tests/images.
@@ -88,37 +108,46 @@ def test_models(hf_runner, vllm_runner, image_assets, model_and_config,
108
The text output is sanitized to be able to compare with hf.
109
"""
110
model_id, vlm_config = model_and_config
91
- hf_images = [asset.for_hf() for asset in image_assets]
92
- vllm_images = [asset.for_vllm() for asset in image_assets]
111
+ images = [asset.pil_image for asset in image_assets]
112
+
113
+ inputs_per_image = [(
114
+ [prompt for _ in size_factors],
115
+ [rescale_image_size(image, factor) for factor in size_factors],
116
+ ) for image, prompt in zip(images, HF_IMAGE_PROMPTS)]
117
+
118
+ # max_model_len should be greater than image_feature_size
119
+ with vllm_runner(model_id,
120
+ dtype=dtype,
121
+ max_model_len=4096,
122
+ enforce_eager=True,
123
+ **vlm_config.as_cli_args_dict()) as vllm_model:
124
+ vllm_outputs_per_image = [
125
+ vllm_model.generate_greedy_logprobs(prompts,
126
+ max_tokens,
127
+ num_logprobs=num_logprobs,
128
+ images=images)
129
+ for prompts, images in inputs_per_image
130
+ ]
131
132
with hf_runner(model_id, dtype=dtype, is_vision_model=True) as hf_model:
95
- hf_outputs = hf_model.generate_greedy(HF_IMAGE_PROMPTS,
96
- max_tokens,
97
- images=hf_images)
98
-
99
- vllm_image_prompts = [
100
- p.replace("<image>", "<image>" * vlm_config.image_feature_size)
101
- for p in HF_IMAGE_PROMPTS
102
- ]
103
-
104
- with vllm_runner(
105
- model_id,
106
- dtype=dtype,
107
- # should be greater than image_feature_size
108
- max_model_len=4096,
109
- enforce_eager=True,
110
- **vlm_config.as_cli_args_dict(),
111
- ) as vllm_model:
112
- vllm_outputs = vllm_model.generate_greedy(vllm_image_prompts,
113
- max_tokens,
114
- images=vllm_images)
115
-
116
- check_outputs_equal(
117
- hf_outputs,
118
- [
119
- vllm_to_hf_output(vllm_output, vlm_config, model_id)
120
- for vllm_output in vllm_outputs
121
- ],
122
- name_0="hf",
123
- name_1="vllm",
124
- )
133
+ hf_outputs_per_image = [
134
+ hf_model.generate_greedy_logprobs_limit(prompts,
135
+ max_tokens,
136
+ num_logprobs=num_logprobs,
137
+ images=images)
138
+ for prompts, images in inputs_per_image
139
+ ]
140
+
141
+ for hf_outputs, vllm_outputs in zip(hf_outputs_per_image,
142
+ vllm_outputs_per_image):
143
+ # TODO: Check whether using original CLIPVisionModel can improve
144
+ # consistency against HF
145
+ check_logprobs_close(
146
+ outputs_0_lst=hf_outputs,
147
+ outputs_1_lst=[
148
+ vllm_to_hf_output(vllm_output, vlm_config, model_id)
149
+ for vllm_output in vllm_outputs
150
+ ],
151
+ name_0="hf",
152
+ name_1="vllm",
153
+ )
tests/models/test_phi3v.py
+82
-58
@@ -1,29 +1,33 @@
1
+import re
2
from typing import List, Optional, Tuple, Type
3
4
import pytest
5
from transformers import AutoTokenizer
6
7
from vllm.config import VisionLanguageConfig
8
+from vllm.multimodal.utils import rescale_image_size
9
+from vllm.sequence import SampleLogprobs
10
from vllm.utils import is_cpu
11
12
from ..conftest import IMAGE_ASSETS, HfRunner, VllmRunner, _ImageAssets
10
-from .utils import check_outputs_equal
13
+from .utils import check_logprobs_close
14
15
pytestmark = pytest.mark.vlm
16
14
-# The image token is placed before "user" on purpose so that the test can pass
17
HF_IMAGE_PROMPTS = IMAGE_ASSETS.prompts({
18
"stop_sign":
19
"<|user|>\n<|image_1|>\nWhat's the content of the image?<|end|>\n<|assistant|>\n", # noqa: E501
20
"cherry_blossom":
19
- "<|user|>\n<|image_1|>\nWhat is the season?<|end|>\n<|assistant|>\n", # noqa: E501
21
+ "<|user|>\n<|image_1|>\nWhat is the season?<|end|>\n<|assistant|>\n",
22
+ "boardwalk":
23
+ "<|user|>\n<|image_1|>\nWhat's in this image?<|end|>\n<|assistant|>\n",
24
})
25
26
27
def iter_phi3v_configs(model_name: str):
28
+ # Need to use the max possible feature size for profile_run
29
image_hw_to_feature_size = {
25
- (1008, 1344): 1921,
26
- (2016, 2688): 1933,
30
+ (1008, 1344): 2653,
31
}
32
33
for (h, w), f in image_hw_to_feature_size.items():
@@ -39,29 +43,29 @@ model_and_vl_config = [
43
]
44
45
42
-def vllm_to_hf_output(vllm_output: Tuple[List[int], str],
46
+def vllm_to_hf_output(vllm_output: Tuple[List[int], str,
47
+ Optional[SampleLogprobs]],
48
vlm_config: VisionLanguageConfig, model_id: str):
49
"""Sanitize vllm output to be comparable with hf output.
50
The function reduces `input_ids` from 1, 32000, 32000, ..., 32000,
51
x1, x2, x3 ... to 1, 32000, x1, x2, x3 ...
52
It also reduces `output_str` from "<image><image>bla" to "bla".
53
"""
49
- output_ids, output_str = vllm_output
50
- image_token_id = vlm_config.image_token_id
54
+ output_ids, output_str, out_logprobs = vllm_output
55
52
- tokenizer = AutoTokenizer.from_pretrained(model_id)
53
- image_token_str = tokenizer.decode(image_token_id)
54
-
55
- hf_output_ids = [
56
- token_id if token_id != image_token_id else 0
57
- for idx, token_id in enumerate(output_ids)
58
- ]
59
- hf_output_str = output_str \
60
- .replace(image_token_str * vlm_config.image_feature_size, "") \
61
- .replace("<s>", " ").replace("<|user|>", "") \
56
+ output_str_without_image = re.sub(r"(<\|image_\d+\|>)+", "", output_str)
57
+ assert output_str_without_image[0] == " "
58
+ output_str_without_image = output_str_without_image[1:]
59
+
60
+ hf_output_str = output_str_without_image.replace("<|user|>", "") \
61
.replace("<|end|>\n<|assistant|>", " ")
62
64
- return hf_output_ids, hf_output_str
63
+ tokenizer = AutoTokenizer.from_pretrained(model_id)
64
+ hf_output_ids = tokenizer.encode(output_str_without_image)
65
+ assert hf_output_ids[0] == 1
66
+ hf_output_ids = hf_output_ids[1:]
67
+
68
+ return hf_output_ids, hf_output_str, out_logprobs
69
70
71
target_dtype = "half"
@@ -75,8 +79,10 @@ def run_test(
79
image_assets: _ImageAssets,
80
model_and_config: Tuple[str, VisionLanguageConfig],
81
*,
82
+ size_factors: List[float],
83
dtype: str,
84
max_tokens: int,
85
+ num_logprobs: int,
86
tensor_parallel_size: int,
87
distributed_executor_backend: Optional[str] = None,
88
):
@@ -90,73 +96,91 @@ def run_test(
96
The text output is sanitized to be able to compare with hf.
97
"""
98
model_id, vlm_config = model_and_config
93
- hf_images = [asset.for_hf() for asset in image_assets]
99
+ images = [asset.pil_image for asset in image_assets]
100
+
101
+ inputs_per_image = [(
102
+ [prompt for _ in size_factors],
103
+ [rescale_image_size(image, factor) for factor in size_factors],
104
+ ) for image, prompt in zip(images, HF_IMAGE_PROMPTS)]
105
106
# NOTE: take care of the order. run vLLM first, and then run HF.
107
# vLLM needs a fresh new process without cuda initialization.
108
# if we run HF first, the cuda initialization will be done and it
109
# will hurt multiprocessing backend with fork method (the default method).
110
111
+ # max_model_len should be greater than image_feature_size
112
with vllm_runner(model_id,
101
- max_model_len=2048,
113
+ max_model_len=4096,
114
dtype=dtype,
115
tensor_parallel_size=tensor_parallel_size,
104
- enforce_eager=True,
116
distributed_executor_backend=distributed_executor_backend,
117
+ enforce_eager=True,
118
**vlm_config.as_cli_args_dict()) as vllm_model:
107
- # NOTE: `asset.for_vllm` will call `torch.cuda.device_count()`
108
- # we must put it inside the vllm_runner context manager
109
- # i.e. after creating vLLM instance.
110
-
111
- vllm_images = [asset.for_vllm() for asset in image_assets]
112
-
113
- vllm_image_prompts = [
114
- p.replace("<|image_1|>",
115
- "<|image|>" * vlm_config.image_feature_size + "<s>")
116
- for p in HF_IMAGE_PROMPTS
119
+ vllm_outputs_per_image = [
120
+ vllm_model.generate_greedy_logprobs(prompts,
121
+ max_tokens,
122
+ num_logprobs=num_logprobs,
123
+ images=vllm_images)
124
+ for prompts, vllm_images in inputs_per_image
125
]
126
119
- vllm_outputs = vllm_model.generate_greedy(vllm_image_prompts,
120
- max_tokens,
121
- images=vllm_images)
122
-
127
# use eager mode for hf runner, since phi3_v didn't work with flash_attn
128
hf_model_kwargs = {"_attn_implementation": "eager"}
129
with hf_runner(model_id, dtype=dtype,
130
model_kwargs=hf_model_kwargs) as hf_model:
127
- hf_outputs = hf_model.generate_greedy(
128
- HF_IMAGE_PROMPTS,
129
- max_tokens,
130
- images=hf_images,
131
- eos_token_id=hf_model.processor.tokenizer.eos_token_id)
132
-
133
- check_outputs_equal(
134
- hf_outputs,
135
- [
136
- vllm_to_hf_output(vllm_output, vlm_config, model_id)
137
- for vllm_output in vllm_outputs
138
- ],
139
- name_0="hf",
140
- name_1="vllm",
141
- )
142
-
131
+ eos_token_id = hf_model.processor.tokenizer.eos_token_id
132
+ hf_outputs_per_image = [
133
+ hf_model.generate_greedy_logprobs_limit(prompts,
134
+ max_tokens,
135
+ num_logprobs=num_logprobs,
136
+ images=hf_images,
137
+ eos_token_id=eos_token_id)
138
+ for prompts, hf_images in inputs_per_image
139
+ ]
140
144
-# Since we use _attn_implementation="eager" for hf_runner, here is
145
-# numeric difference for longer context and test can't pass
146
-@pytest.mark.xfail(
147
- reason="Inconsistent image processor being used due to lack "
148
- "of support for dynamic image token replacement")
141
+ for hf_outputs, vllm_outputs in zip(hf_outputs_per_image,
142
+ vllm_outputs_per_image):
143
+ check_logprobs_close(
144
+ outputs_0_lst=hf_outputs,
145
+ outputs_1_lst=[
146
+ vllm_to_hf_output(vllm_output, vlm_config, model_id)
147
+ for vllm_output in vllm_outputs
148
+ ],
149
+ name_0="hf",
150
+ name_1="vllm",
151
+ )
152
+
153
+
154
+# Since we use _attn_implementation="eager" for hf_runner, there is more
155
+# significant numerical difference. The basic `logprobs=5` fails to pass.
156
@pytest.mark.parametrize("model_and_config", model_and_vl_config)
157
+@pytest.mark.parametrize(
158
+ "size_factors",
159
+ [
160
+ # No image
161
+ [],
162
+ # Single-scale
163
+ [1.0],
164
+ # Single-scale, batched
165
+ [1.0, 1.0, 1.0],
166
+ # Multi-scale
167
+ [0.25, 0.5, 1.0],
168
+ ],
169
+)
170
@pytest.mark.parametrize("dtype", [target_dtype])
171
@pytest.mark.parametrize("max_tokens", [128])
172
+@pytest.mark.parametrize("num_logprobs", [10])
173
def test_models(hf_runner, vllm_runner, image_assets, model_and_config,
153
- dtype: str, max_tokens: int) -> None:
174
+ size_factors, dtype: str, max_tokens: int,
175
+ num_logprobs: int) -> None:
176
run_test(
177
hf_runner,
178
vllm_runner,
179
image_assets,
180
model_and_config,
181
+ size_factors=size_factors,
182
dtype=dtype,
183
max_tokens=max_tokens,
184
+ num_logprobs=num_logprobs,
185
tensor_parallel_size=1,
186
)
tests/models/utils.py
+67
-21
@@ -1,11 +1,18 @@
1
-from typing import Dict, List, Tuple
1
+import warnings
2
+from typing import Dict, List, Optional, Sequence, Tuple, Union
3
+
4
+from vllm.sequence import SampleLogprobs
5
6
TokensText = Tuple[List[int], str]
7
8
6
-def check_outputs_equal(outputs_0_lst: List[TokensText],
7
- outputs_1_lst: List[TokensText], name_0: str,
8
- name_1: str):
9
+def check_outputs_equal(
10
+ *,
11
+ outputs_0_lst: Sequence[TokensText],
12
+ outputs_1_lst: Sequence[TokensText],
13
+ name_0: str,
14
+ name_1: str,
15
+):
16
"""
17
Compare the two sequences generated by different models,
18
which should be equal.
@@ -18,20 +25,28 @@ def check_outputs_equal(outputs_0_lst: List[TokensText],
25
output_ids_0, output_str_0 = outputs_0
26
output_ids_1, output_str_1 = outputs_1
27
21
- assert output_str_0 == output_str_1, (f"Test{prompt_idx}:"
22
- f"\n{name_0}:\t{output_str_0!r}"
23
- f"\n{name_1}:\t{output_str_1!r}")
24
- assert output_ids_0 == output_ids_1, (f"Test{prompt_idx}:"
25
- f"\n{name_0}:\t{output_str_0!r}"
26
- f"\n{name_1}:\t{output_str_1!r}")
28
+ # The text and token outputs should exactly match
29
+ fail_msg = (f"Test{prompt_idx}:"
30
+ f"\n{name_0}:\t{output_str_0!r}"
31
+ f"\n{name_1}:\t{output_str_1!r}")
32
+
33
+ assert output_str_0 == output_str_1, fail_msg
34
+ assert output_ids_0 == output_ids_1, fail_msg
35
36
29
-TokensTextLogprobs = Tuple[List[int], str, List[Dict[int, float]]]
37
+TokensTextLogprobs = Tuple[List[int], str, Optional[Union[List[Dict[int,
38
+ float]],
39
+ SampleLogprobs]]]
40
41
32
-def check_logprobs_close(outputs_0_lst: List[TokensTextLogprobs],
33
- outputs_1_lst: List[TokensTextLogprobs], name_0: str,
34
- name_1: str):
42
+def check_logprobs_close(
43
+ *,
44
+ outputs_0_lst: Sequence[TokensTextLogprobs],
45
+ outputs_1_lst: Sequence[TokensTextLogprobs],
46
+ name_0: str,
47
+ name_1: str,
48
+ warn_on_mismatch: bool = True,
49
+):
50
"""
51
Compare the logprobs of two sequences generated by different models,
52
which should be similar but not necessarily equal.
@@ -45,21 +60,52 @@ def check_logprobs_close(outputs_0_lst: List[TokensTextLogprobs],
60
output_ids_0, output_str_0, logprobs_0 = outputs_0
61
output_ids_1, output_str_1, logprobs_1 = outputs_1
62
63
+ if logprobs_0 is None:
64
+ logprobs_0 = [None] * len(output_ids_0)
65
+ if logprobs_1 is None:
66
+ logprobs_1 = [None] * len(output_ids_1)
67
+
68
# Loop through generated tokens.
69
for idx, (output_id_0,
70
output_id_1) in enumerate(zip(output_ids_0, output_ids_1)):
71
72
# If generated tokens don't match, then
73
if output_id_0 != output_id_1:
74
+ logprobs_elem_0 = logprobs_0[idx]
75
+ logprobs_elem_1 = logprobs_1[idx]
76
+
77
# Each predicted token must be in top N logprobs of the other
55
- assert output_id_0 in logprobs_1[idx], (
56
- f"Test{prompt_idx}:"
57
- f"\n{name_0}:\t{output_str_0!r}"
58
- f"\n{name_1}:\t{output_str_1!r}")
59
- assert output_id_1 in logprobs_0[idx], (
78
+ fail_msg = (
79
f"Test{prompt_idx}:"
61
- f"\n{name_0}:\t{output_str_0!r}"
62
- f"\n{name_1}:\t{output_str_1!r}")
80
+ f"\n{name_0}:\t{output_str_0!r}\t{logprobs_elem_0}"
81
+ f"\n{name_1}:\t{output_str_1!r}\t{logprobs_elem_1}")
82
+
83
+ assert logprobs_elem_0 is not None, fail_msg
84
+ assert logprobs_elem_1 is not None, fail_msg
85
+ assert output_id_0 in logprobs_elem_1, fail_msg
86
+ assert output_id_1 in logprobs_elem_0, fail_msg
87
+
88
+ if warn_on_mismatch:
89
+ with warnings.catch_warnings():
90
+ # This ensures that repeated warnings are shown
91
+ # in the output, not just the first occurrence
92
+ warnings.simplefilter("always")
93
+
94
+ warnings.warn(fail_msg, stacklevel=2)
95
96
# Break out since sequences will now diverge.
97
break
98
+ else:
99
+ if output_str_0 != output_str_1 and warn_on_mismatch:
100
+ # The token outputs exactly match,
101
+ # so the text outputs should exactly match as well
102
+ fail_msg = (f"Test{prompt_idx}:"
103
+ f"\n{name_0}:\t{output_str_0!r}"
104
+ f"\n{name_1}:\t{output_str_1!r}")
105
+
106
+ with warnings.catch_warnings():
107
+ # This ensures that repeated warnings are shown
108
+ # in the output, not just the first occurrence
109
+ warnings.simplefilter("always")
110
+
111
+ warnings.warn(fail_msg, stacklevel=2)
tests/multimodal/test_mapper.py
+16
-47
@@ -4,12 +4,12 @@ from transformers import CLIPImageProcessor, LlavaNextImageProcessor
4
5
from vllm.config import ModelConfig
6
from vllm.multimodal import MULTIMODAL_REGISTRY
7
-
8
-from ..conftest import _STR_DTYPE_TO_TORCH_DTYPE
7
+from vllm.multimodal.utils import rescale_image_size
8
9
10
@pytest.mark.parametrize("dtype", ["half", "float"])
12
-def test_clip_image_processor(image_assets, dtype):
11
+@pytest.mark.parametrize("size_factor", [0.25, 0.5, 1.0])
12
+def test_clip_image_processor(image_assets, dtype, size_factor):
13
MODEL_NAME = "llava-hf/llava-1.5-7b-hf"
14
15
hf_processor = CLIPImageProcessor.from_pretrained(MODEL_NAME)
@@ -26,13 +26,15 @@ def test_clip_image_processor(image_assets, dtype):
26
)
27
28
for asset in image_assets:
29
+ image = rescale_image_size(asset.pil_image, size_factor)
30
+
31
hf_result = hf_processor.preprocess(
30
- asset.pil_image,
32
+ image,
33
return_tensors="pt",
32
- ).to(dtype=_STR_DTYPE_TO_TORCH_DTYPE[dtype])
34
+ )
35
vllm_result = MULTIMODAL_REGISTRY.map_input(
36
model_config,
35
- {"image": asset.pil_image},
37
+ {"image": image},
38
)
39
40
assert hf_result.keys() == vllm_result.keys()
@@ -44,12 +46,10 @@ def test_clip_image_processor(image_assets, dtype):
46
assert np.allclose(hf_arr, vllm_arr), f"Failed for key={key}"
47
48
47
-@pytest.mark.xfail(
48
- reason="Inconsistent image processor being used due to lack "
49
- "of support for dynamic image token replacement")
49
@pytest.mark.parametrize("dtype", ["half", "float"])
51
-def test_llava_next_image_processor(image_assets, dtype):
52
- MODEL_NAME = "llava-hf/llava-v1.6-34b-hf"
50
+@pytest.mark.parametrize("size_factor", [0.25, 0.5, 1.0])
51
+def test_llava_next_image_processor(image_assets, dtype, size_factor):
52
+ MODEL_NAME = "llava-hf/llava-v1.6-vicuna-7b-hf"
53
54
hf_processor = LlavaNextImageProcessor.from_pretrained(MODEL_NAME)
55
assert isinstance(hf_processor, LlavaNextImageProcessor)
@@ -65,13 +65,15 @@ def test_llava_next_image_processor(image_assets, dtype):
65
)
66
67
for asset in image_assets:
68
+ image = rescale_image_size(asset.pil_image, size_factor)
69
+
70
hf_result = hf_processor.preprocess(
69
- asset.pil_image,
71
+ image,
72
return_tensors="pt",
71
- ).to(dtype=_STR_DTYPE_TO_TORCH_DTYPE[dtype])
73
+ )
74
vllm_result = MULTIMODAL_REGISTRY.map_input(
75
model_config,
74
- {"image": asset.pil_image},
76
+ {"image": image},
77
)
78
79
assert hf_result.keys() == vllm_result.keys()
@@ -81,36 +83,3 @@ def test_llava_next_image_processor(image_assets, dtype):
83
84
assert hf_arr.shape == vllm_arr.shape, f"Failed for key={key}"
85
assert np.allclose(hf_arr, vllm_arr), f"Failed for key={key}"
84
-
85
-
86
-@pytest.mark.xfail(
87
- reason="Example image pixels were not processed using HuggingFace")
88
-@pytest.mark.parametrize("dtype", ["float"])
89
-def test_image_pixel_types(image_assets, dtype):
90
- MODEL_NAME = "llava-hf/llava-1.5-7b-hf"
91
-
92
- model_config = ModelConfig(
93
- model=MODEL_NAME,
94
- tokenizer=MODEL_NAME,
95
- tokenizer_mode="auto",
96
- trust_remote_code=False,
97
- seed=0,
98
- dtype=dtype,
99
- revision=None,
100
- )
101
- for asset in image_assets:
102
- image_result = MULTIMODAL_REGISTRY.map_input(
103
- model_config,
104
- {"image": asset.pil_image},
105
- )
106
- tensor_result = MULTIMODAL_REGISTRY.map_input(
107
- model_config,
108
- {"image": asset.pil_image},
109
- )
110
-
111
- assert image_result.keys() == tensor_result.keys()
112
- for key, image_arr in image_result.items():
113
- tensor_arr: np.ndarray = tensor_result[key].numpy()
114
-
115
- assert image_arr.shape == tensor_arr.shape, f"Failed for key={key}"
116
- assert np.allclose(image_arr, tensor_arr), f"Failed for key={key}"
tests/multimodal/test_utils.py
+18
-11
@@ -5,10 +5,9 @@ from typing import Dict, Tuple
5
6
import numpy as np
7
import pytest
8
-import pytest_asyncio
8
from PIL import Image
9
11
-from vllm.multimodal.utils import ImageFetchAiohttp
10
+from vllm.multimodal.utils import ImageFetchAiohttp, fetch_image
11
12
# Test different image extensions (JPG/PNG) and formats (gray/RGB/RGBA)
13
TEST_IMAGE_URLS = [
@@ -19,12 +18,9 @@ TEST_IMAGE_URLS = [
18
]
19
20
22
-@pytest_asyncio.fixture(scope="session")
23
-async def url_images() -> Dict[str, Image.Image]:
24
- return {
25
- image_url: await ImageFetchAiohttp.fetch_image(image_url)
26
- for image_url in TEST_IMAGE_URLS
27
- }
21
+@pytest.fixture(scope="module")
22
+def url_images() -> Dict[str, Image.Image]:
23
+ return {image_url: fetch_image(image_url) for image_url in TEST_IMAGE_URLS}
24
25
26
def get_supported_suffixes() -> Tuple[str, ...]:
@@ -41,7 +37,15 @@ def _image_equals(a: Image.Image, b: Image.Image) -> bool:
37
return (np.asarray(a) == np.asarray(b.convert(a.mode))).all()
38
39
44
-@pytest.mark.asyncio
40
+@pytest.mark.asyncio(scope="module")
41
+@pytest.mark.parametrize("image_url", TEST_IMAGE_URLS)
42
+async def test_fetch_image_http(image_url: str):
43
+ image_sync = fetch_image(image_url)
44
+ image_async = await ImageFetchAiohttp.fetch_image(image_url)
45
+ assert _image_equals(image_sync, image_async)
46
+
47
+
48
+@pytest.mark.asyncio(scope="module")
49
@pytest.mark.parametrize("image_url", TEST_IMAGE_URLS)
50
@pytest.mark.parametrize("suffix", get_supported_suffixes())
51
async def test_fetch_image_base64(url_images: Dict[str, Image.Image],
@@ -68,8 +72,11 @@ async def test_fetch_image_base64(url_images: Dict[str, Image.Image],
72
base64_image = base64.b64encode(f.read()).decode("utf-8")
73
data_url = f"data:{mime_type};base64,{base64_image}"
74
71
- data_image = await ImageFetchAiohttp.fetch_image(data_url)
75
+ data_image_sync = fetch_image(data_url)
76
if _image_equals(url_image, Image.open(f)):
73
- assert _image_equals(url_image, data_image)
77
+ assert _image_equals(url_image, data_image_sync)
78
else:
79
pass # Lossy format; only check that image can be opened
80
+
81
+ data_image_async = await ImageFetchAiohttp.fetch_image(data_url)
82
+ assert _image_equals(data_image_sync, data_image_async)
vllm/config.py
+1
-11
@@ -5,7 +5,7 @@ from typing import (TYPE_CHECKING, Any, ClassVar, Dict, List, Optional, Tuple,
5
Union)
6
7
import torch
8
-from transformers import PretrainedConfig, PreTrainedTokenizerBase
8
+from transformers import PretrainedConfig
9
10
import vllm.envs as envs
11
from vllm.logger import init_logger
@@ -1303,16 +1303,6 @@ class VisionLanguageConfig:
1303
image_input_shape: tuple
1304
image_feature_size: int
1305
1306
- #TODO(ywang96): make this a cached property once we refactor the
1307
- # VisionLanguageConfig class.
1308
- def get_image_token_text(
1309
- self, tokenizer: PreTrainedTokenizerBase) -> Tuple[str, str]:
1310
- """Get the image token placeholder text to be inserted into the
1311
- text prompt and the string representation of the image token id.
1312
- """
1313
- image_token_str = tokenizer.decode(self.image_token_id)
1314
- return image_token_str * self.image_feature_size, image_token_str
1315
-
1306
def as_cli_args_dict(self) -> Dict[str, Any]:
1307
"""Flatten vision language config to pure args.
1308
vllm/entrypoints/openai/serving_chat.py
+51
-46
@@ -1,6 +1,7 @@
1
import codecs
2
import time
3
from dataclasses import dataclass, field
4
+from functools import cached_property
5
from typing import (AsyncGenerator, AsyncIterator, Awaitable, Dict, Iterable,
6
List, Optional)
7
from typing import Sequence as GenericSequence
@@ -10,7 +11,7 @@ from fastapi import Request
11
from openai.types.chat import (ChatCompletionContentPartImageParam,
12
ChatCompletionContentPartTextParam)
13
13
-from vllm.config import ModelConfig, VisionLanguageConfig
14
+from vllm.config import ModelConfig
15
from vllm.engine.async_llm_engine import AsyncLLMEngine
16
from vllm.entrypoints.openai.protocol import (
17
ChatCompletionContentPartParam, ChatCompletionLogProb,
@@ -27,8 +28,7 @@ from vllm.logger import init_logger
28
from vllm.model_executor.guided_decoding import (
29
get_guided_decoding_logits_processor)
30
from vllm.multimodal import MultiModalDataDict
30
-from vllm.multimodal.utils import (async_get_and_parse_image,
31
- get_full_image_text_prompt)
31
+from vllm.multimodal.utils import async_get_and_parse_image
32
from vllm.outputs import RequestOutput
33
from vllm.sequence import Logprob
34
from vllm.tracing import (contains_trace_headers, extract_trace_headers,
@@ -97,6 +97,36 @@ class OpenAIServingChat(OpenAIServing):
97
logger.warning(
98
"No chat template provided. Chat API will not work.")
99
100
+ @cached_property
101
+ def image_token_str(self) -> Optional[str]:
102
+ # TODO: Let user specify how to insert image tokens into prompt
103
+ # (similar to chat template)
104
+ model_type = self.model_config.hf_config.model_type
105
+ if model_type == "phi3_v":
106
+ # Workaround since this token is not defined in the tokenizer
107
+ return "<|image_1|>"
108
+ if model_type in ("blip-2", "chatglm", "fuyu", "minicpmv",
109
+ "paligemma"):
110
+ # These models do not use image tokens in the prompt
111
+ return None
112
+
113
+ # The default behaviour assumes that the image token is
114
+ # available to the tokenizer.
115
+ # (Suitable for LLaVA, Idefics2, DeepSeek-VL)
116
+ vlm_config = self.model_config.multimodal_config
117
+ if vlm_config is None:
118
+ raise ValueError(
119
+ "'image_url' input is not supported as the loaded "
120
+ "model is not multimodal.")
121
+
122
+ image_token_id = vlm_config.image_token_id
123
+ if vlm_config.image_token_id is None:
124
+ raise ValueError(
125
+ "'image_url' input is not supported as the loaded "
126
+ "model does not specify an image token.")
127
+
128
+ return self.tokenizer.decode(image_token_id)
129
+
130
def _parse_chat_message_content_parts(
131
self,
132
role: str,
@@ -105,21 +135,26 @@ class OpenAIServingChat(OpenAIServing):
135
texts: List[str] = []
136
mm_futures: List[Awaitable[MultiModalDataDict]] = []
137
108
- vlm_config: Optional[VisionLanguageConfig] = getattr(
109
- self.engine.engine, "vision_language_config", None)
110
- model_config = getattr(self.engine.engine, "model_config", None)
111
-
138
for part in parts:
139
part_type = part["type"]
140
if part_type == "text":
141
text = cast(ChatCompletionContentPartTextParam, part)["text"]
142
texts.append(text)
143
elif part_type == "image_url":
118
- if vlm_config is None:
119
- raise ValueError(
120
- "'image_url' input is not supported as the loaded "
121
- "model is not multimodal.")
122
- assert self.tokenizer is not None
144
+ if len(mm_futures) > 0:
145
+ raise NotImplementedError(
146
+ "Multiple 'image_url' input is currently not supported."
147
+ )
148
+
149
+ image_token_str = self.image_token_str
150
+ if image_token_str is not None:
151
+ if any(image_token_str in text for text in texts):
152
+ logger.warning(
153
+ "Detected image token string in the text prompt. "
154
+ "Skipping prompt formatting.")
155
+ else:
156
+ texts.append(image_token_str)
157
+
158
image_url = cast(ChatCompletionContentPartImageParam,
159
part)["image_url"]
160
@@ -128,43 +163,13 @@ class OpenAIServingChat(OpenAIServing):
163
"'image_url.detail' is currently not supported and "
164
"will be ignored.")
165
131
- mm_future = async_get_and_parse_image(image_url["url"])
132
- mm_futures.append(mm_future)
133
-
166
+ image_future = async_get_and_parse_image(image_url["url"])
167
+ mm_futures.append(image_future)
168
else:
169
raise NotImplementedError(f"Unknown part type: {part_type}")
170
171
text_prompt = "\n".join(texts)
138
-
139
- if vlm_config is not None and len(mm_futures):
140
-
141
- assert len(
142
- mm_futures
143
- ) == 1, "Multiple 'image_url' input is currently not supported."
144
- (image_token_prompt,
145
- image_token_str) = vlm_config.get_image_token_text(self.tokenizer)
146
-
147
- # NOTE: If image token string (e.g, <image>) is already present
148
- # in the text prompt, we assume it follows the same format required
149
- # by the engine.
150
- if image_token_str in text_prompt:
151
- logger.warning(
152
- "Detected image token string in the text prompt. "
153
- "Skipping prompt formatting.")
154
- messages = [
155
- ConversationMessage(role=role, content=text_prompt)
156
- ]
157
-
158
- else:
159
- full_prompt = get_full_image_text_prompt(
160
- image_prompt=image_token_prompt,
161
- text_prompt=text_prompt,
162
- config=model_config)
163
- messages = [
164
- ConversationMessage(role=role, content=full_prompt)
165
- ]
166
- else:
167
- messages = [ConversationMessage(role=role, content=text_prompt)]
172
+ messages = [ConversationMessage(role=role, content=text_prompt)]
173
174
return ChatMessageParseResult(messages=messages, mm_futures=mm_futures)
175
@@ -267,7 +272,7 @@ class OpenAIServingChat(OpenAIServing):
272
"prompt": prompt_text,
273
"prompt_token_ids": prompt_ids,
274
}
270
- if mm_data is not None:
275
+ if mm_data:
276
inputs["multi_modal_data"] = mm_data
277
278
is_tracing_enabled = await self.engine.is_tracing_enabled()
vllm/entrypoints/openai/serving_engine.py
+1
@@ -36,6 +36,7 @@ class OpenAIServing:
36
super().__init__()
37
38
self.engine = engine
39
+ self.model_config = model_config
40
self.max_model_len = model_config.max_model_len
41
42
# A separate tokenizer to map token IDs to strings.
vllm/inputs/registry.py
+2
-1
@@ -140,7 +140,8 @@ class InputRegistry:
140
141
The model is identified by ``model_config``.
142
143
- TODO: Add guide [ref: PR #5276]
143
+ See also:
144
+ :ref:`adding_a_new_multimodal_model`
145
"""
146
# Avoid circular import
147
from vllm.model_executor.model_loader import get_model_architecture
vllm/model_executor/models/clip.py
+37
@@ -8,10 +8,14 @@ from PIL import Image
8
from transformers import CLIPVisionConfig
9
from transformers.models.clip.modeling_clip import CLIPAttention
10
11
+from vllm.config import ModelConfig
12
+from vllm.inputs import LLMInputs
13
from vllm.model_executor.layers.activation import get_act_fn
14
from vllm.model_executor.layers.linear import (ColumnParallelLinear,
15
RowParallelLinear)
16
from vllm.model_executor.layers.quantization import QuantizationConfig
17
+from vllm.multimodal.image import (cached_get_tokenizer,
18
+ repeat_and_pad_image_tokens)
19
from vllm.sequence import SequenceData
20
21
@@ -64,6 +68,39 @@ def dummy_image_for_clip(
68
return {"image": image}
69
70
71
+def input_processor_for_clip(
72
+ model_config: ModelConfig,
73
+ hf_config: CLIPVisionConfig,
74
+ llm_inputs: LLMInputs,
75
+ *,
76
+ image_token_id: int,
77
+ image_feature_size_override: Optional[int] = None,
78
+):
79
+ multi_modal_data = llm_inputs.get("multi_modal_data")
80
+ if multi_modal_data is None or "image" not in multi_modal_data:
81
+ return llm_inputs
82
+
83
+ tokenizer = cached_get_tokenizer(model_config.tokenizer)
84
+
85
+ if image_feature_size_override is None:
86
+ image_feature_size = get_clip_image_feature_size(hf_config)
87
+ else:
88
+ image_feature_size = image_feature_size_override
89
+
90
+ new_prompt, new_token_ids = repeat_and_pad_image_tokens(
91
+ tokenizer,
92
+ llm_inputs.get("prompt"),
93
+ llm_inputs["prompt_token_ids"],
94
+ image_token_id=image_token_id,
95
+ repeat_count=image_feature_size,
96
+ )
97
+
98
+ # NOTE: Create a defensive copy of the original inputs
99
+ return LLMInputs(prompt_token_ids=new_token_ids,
100
+ prompt=new_prompt,
101
+ multi_modal_data=multi_modal_data)
102
+
103
+
104
# Adapted from https://github.com/huggingface/transformers/blob/v4.39.0/src/transformers/models/clip/modeling_clip.py#L164 # noqa
105
class CLIPVisionEmbeddings(nn.Module):
106
vllm/model_executor/models/llava.py
+27
-22
@@ -6,7 +6,7 @@ from transformers import CLIPVisionConfig, LlavaConfig
6
7
from vllm.attention import AttentionMetadata
8
from vllm.config import CacheConfig, VisionLanguageConfig
9
-from vllm.inputs import INPUT_REGISTRY, InputContext
9
+from vllm.inputs import INPUT_REGISTRY, InputContext, LLMInputs
10
from vllm.model_executor.layers.activation import get_act_fn
11
from vllm.model_executor.layers.logits_processor import LogitsProcessor
12
from vllm.model_executor.layers.quantization.base_config import (
@@ -20,8 +20,10 @@ from vllm.model_executor.sampling_metadata import SamplingMetadata
20
from vllm.multimodal import MULTIMODAL_REGISTRY
21
from vllm.sequence import IntermediateTensors, SamplerOutput
22
23
-from .clip import dummy_image_for_clip, dummy_seq_data_for_clip
23
+from .clip import (dummy_image_for_clip, dummy_seq_data_for_clip,
24
+ input_processor_for_clip)
25
from .interfaces import SupportsVision
26
+from .utils import merge_vision_embeddings
27
28
_KEYS_TO_MODIFY_MAPPING = {
29
"language_model.lm_head": "lm_head",
@@ -51,28 +53,10 @@ class LlavaMultiModalProjector(nn.Module):
53
return hidden_states
54
55
54
-def merge_vision_embeddings(input_ids: torch.Tensor,
55
- inputs_embeds: torch.Tensor,
56
- vision_embeddings: torch.Tensor,
57
- image_token_id: int) -> torch.Tensor:
58
- """In place merges in vision_embeddings with inputs_embeds."""
59
- mask = (input_ids == image_token_id)
60
-
61
- image_feature_size = vision_embeddings.shape[0] * vision_embeddings.shape[1]
62
- if mask.sum() != image_feature_size:
63
- raise ValueError(f"image_feature_size should be {image_feature_size}, "
64
- f"but found: {mask.sum()}")
65
-
66
- inputs_embeds[mask] = vision_embeddings.view(image_feature_size,
67
- vision_embeddings.shape[-1])
68
-
69
- return inputs_embeds
70
-
71
-
56
class LlavaImagePixelInputs(TypedDict):
57
type: Literal["pixel_values"]
58
data: torch.Tensor
75
- """Shape: (batch_size, num_channels, height, width)"""
59
+ """Shape: `(batch_size, num_channels, height, width)`"""
60
61
62
LlavaImageInputs = LlavaImagePixelInputs
@@ -96,8 +80,30 @@ def dummy_data_for_llava(ctx: InputContext, seq_len: int):
80
raise NotImplementedError(msg)
81
82
83
+def input_processor_for_llava(ctx: InputContext, llm_inputs: LLMInputs):
84
+ multi_modal_data = llm_inputs.get("multi_modal_data")
85
+ if multi_modal_data is None or "image" not in multi_modal_data:
86
+ return llm_inputs
87
+
88
+ model_config = ctx.model_config
89
+ hf_config = ctx.get_hf_config(LlavaConfig)
90
+ vision_config = hf_config.vision_config
91
+
92
+ if isinstance(vision_config, CLIPVisionConfig):
93
+ return input_processor_for_clip(
94
+ model_config,
95
+ vision_config,
96
+ llm_inputs,
97
+ image_token_id=hf_config.image_token_index,
98
+ )
99
+
100
+ msg = f"Unsupported vision config: {type(vision_config)}"
101
+ raise NotImplementedError(msg)
102
+
103
+
104
@MULTIMODAL_REGISTRY.register_image_input_mapper()
105
@INPUT_REGISTRY.register_dummy_data(dummy_data_for_llava)
106
+@INPUT_REGISTRY.register_input_processor(input_processor_for_llava)
107
class LlavaForConditionalGeneration(nn.Module, SupportsVision):
108
109
def __init__(self,
@@ -112,7 +118,6 @@ class LlavaForConditionalGeneration(nn.Module, SupportsVision):
118
119
# TODO: Optionally initializes this for supporting embeddings.
120
self.vision_tower = CLIPVisionModel(config.vision_config)
115
-
121
self.multi_modal_projector = LlavaMultiModalProjector(
122
vision_hidden_size=config.vision_config.hidden_size,
123
text_hidden_size=config.text_config.hidden_size,
vllm/model_executor/models/llava_next.py
+108
-76
@@ -1,4 +1,4 @@
1
-from typing import Dict, Iterable, List, Literal, Optional, Tuple, TypedDict
1
+from typing import Iterable, List, Literal, Optional, Tuple, TypedDict
2
3
import torch
4
import torch.nn as nn
@@ -10,7 +10,7 @@ from typing_extensions import NotRequired
10
11
from vllm.attention import AttentionMetadata
12
from vllm.config import CacheConfig, VisionLanguageConfig
13
-from vllm.inputs import INPUT_REGISTRY, InputContext
13
+from vllm.inputs import INPUT_REGISTRY, InputContext, LLMInputs
14
from vllm.logger import init_logger
15
from vllm.model_executor.layers.logits_processor import LogitsProcessor
16
from vllm.model_executor.layers.quantization.base_config import (
@@ -21,13 +21,14 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
21
from vllm.model_executor.models.clip import CLIPVisionModel
22
from vllm.model_executor.models.llama import LlamaModel
23
from vllm.model_executor.sampling_metadata import SamplingMetadata
24
-from vllm.multimodal import MULTIMODAL_REGISTRY
24
+from vllm.multimodal import MULTIMODAL_REGISTRY, BatchedTensors
25
from vllm.sequence import IntermediateTensors, SamplerOutput
26
27
from .clip import (dummy_image_for_clip, dummy_seq_data_for_clip,
28
- get_clip_patch_grid_length)
28
+ get_clip_patch_grid_length, input_processor_for_clip)
29
from .interfaces import SupportsVision
30
-from .llava import LlavaMultiModalProjector, merge_vision_embeddings
30
+from .llava import LlavaMultiModalProjector
31
+from .utils import merge_vision_embeddings
32
33
logger = init_logger(__name__)
34
@@ -39,16 +40,27 @@ _KEYS_TO_MODIFY_MAPPING = {
40
41
class LlavaNextImagePixelInputs(TypedDict):
42
type: Literal["pixel_values"]
42
- data: torch.Tensor
43
- """Shape: (batch_size, 1 + num_patches, num_channels, height, width)"""
43
+ data: BatchedTensors
44
+ """
45
+ Shape: `(batch_size, 1 + num_patches, num_channels, height, width)`
46
+
47
+ Note that `num_patches` may be different for each batch.
48
+ """
49
50
image_sizes: NotRequired[torch.Tensor]
46
- """Shape: (batch_size, 2)"""
51
+ """
52
+ Shape: `(batch_size, 2)`
53
+
54
+ This should be in `(height, width)` format.
55
+ """
56
57
58
LlavaNextImageInputs = LlavaNextImagePixelInputs
59
60
61
+# Taken from: https://github.com/huggingface/text-generation-inference/blob/v2.0.4/server/text_generation_server/models/vlm_causal_lm.py#L91
62
+# NOTE: new_height and new_width are further incremented to properly invert the
63
+# floordiv operation: https://github.com/huggingface/transformers/blob/v4.42.2/src/transformers/models/llava_next/modeling_llava_next.py#L133
64
def _get_llava_next_num_unpadded_features(
65
height: int,
66
width: int,
@@ -56,7 +68,6 @@ def _get_llava_next_num_unpadded_features(
68
num_patch_height: int,
69
num_patch_width: int,
70
) -> Tuple[int, int]:
59
- # Taken from: https://github.com/huggingface/text-generation-inference/blob/799a193b109662743bed1b18a09af1fdcd508c8b/server/text_generation_server/models/vlm_causal_lm.py#L111
71
current_height = npatches * num_patch_height
72
current_width = npatches * num_patch_width
73
@@ -64,9 +75,13 @@ def _get_llava_next_num_unpadded_features(
75
current_aspect_ratio: float = current_width / current_height
76
if aspect_ratio > current_aspect_ratio:
77
new_height = (height * current_width) // width
78
+ if new_height % 2 == 1:
79
+ new_height += 1
80
current_height = new_height
81
else:
82
new_width = (width * current_height) // height
83
+ if new_width % 2 == 1:
84
+ new_width += 1
85
current_width = new_width
86
87
unpadded_features = current_height * current_width
@@ -74,7 +89,8 @@ def _get_llava_next_num_unpadded_features(
89
return (unpadded_features, newline_features)
90
91
77
-def _get_llava_next_image_feature_size(
92
+# Based on: https://github.com/huggingface/text-generation-inference/blob/v2.0.4/server/text_generation_server/models/vlm_causal_lm.py#L111
93
+def get_llava_next_image_feature_size(
94
hf_config: LlavaNextConfig,
95
*,
96
input_height: int,
@@ -89,7 +105,9 @@ def _get_llava_next_image_feature_size(
105
)
106
base_feature_size = num_patches * num_patches
107
92
- num_patch_height, num_patch_width = get_anyres_image_grid_shape(
108
+ # Note: We follow the "wrong" width/height order
109
+ # [ref: PR huggingface/transformers#31588]
110
+ num_patch_width, num_patch_height = get_anyres_image_grid_shape(
111
image_size=(input_height, input_width),
112
grid_pinpoints=hf_config.image_grid_pinpoints,
113
patch_size=vision_config.image_size,
@@ -110,14 +128,16 @@ def _get_llava_next_image_feature_size(
128
129
130
def dummy_data_for_llava_next(ctx: InputContext, seq_len: int):
113
- multimodal_config = ctx.get_multimodal_config()
131
hf_config = ctx.get_hf_config(LlavaNextConfig)
132
vision_config = hf_config.vision_config
133
117
- #TODO: change the logic for dummy data to support dynamic shape
118
- _, _, dummy_height, dummy_width = multimodal_config.image_input_shape
119
- image_feature_size = _get_llava_next_image_feature_size(
120
- hf_config, input_height=dummy_height, input_width=dummy_width)
134
+ # Result in the max possible feature size (2x2 grid of 336x336px tiles)
135
+ dummy_height = dummy_width = 448
136
+ image_feature_size = get_llava_next_image_feature_size(
137
+ hf_config,
138
+ input_height=dummy_height,
139
+ input_width=dummy_width,
140
+ )
141
142
if isinstance(vision_config, CLIPVisionConfig):
143
seq_data = dummy_seq_data_for_clip(
@@ -139,27 +159,47 @@ def dummy_data_for_llava_next(ctx: InputContext, seq_len: int):
159
raise NotImplementedError(msg)
160
161
142
-def _pixel_mapper(ctx: InputContext, image: object) -> Dict[str, torch.Tensor]:
162
+def input_processor_for_llava_next(ctx: InputContext, llm_inputs: LLMInputs):
163
+ multi_modal_data = llm_inputs.get("multi_modal_data")
164
+ if multi_modal_data is None or "image" not in multi_modal_data:
165
+ return llm_inputs
166
144
- if isinstance(image, Image.Image):
167
+ model_config = ctx.model_config
168
+ hf_config = ctx.get_hf_config(LlavaNextConfig)
169
+ vision_config = hf_config.vision_config
170
146
- # Temporary patch before dynamic number of image tokens is supported
147
- _, _, h, w = ctx.get_multimodal_config().image_input_shape
148
- if (w, h) != (image.width, image.height):
149
- logger.warning(
150
- "Dynamic image shape is currently not supported. "
151
- "Resizing input image to (%d, %d).", w, h)
171
+ image_data = multi_modal_data["image"]
172
+ if isinstance(image_data, Image.Image):
173
+ width, height = image_data.size
174
+
175
+ image_feature_size = get_llava_next_image_feature_size(
176
+ hf_config,
177
+ input_height=height,
178
+ input_width=width,
179
+ )
180
+ elif isinstance(image_data, torch.Tensor):
181
+ raise NotImplementedError("Embeddings input is not supported yet")
182
+ else:
183
+ raise TypeError(f"Invalid image type: {type(image_data)}")
184
153
- image = image.resize((w, h))
185
+ vision_config = hf_config.vision_config
186
155
- return MULTIMODAL_REGISTRY._get_plugin("image") \
156
- ._default_input_mapper(ctx, image)
187
+ if isinstance(vision_config, CLIPVisionConfig):
188
+ return input_processor_for_clip(
189
+ model_config,
190
+ vision_config,
191
+ llm_inputs,
192
+ image_token_id=hf_config.image_token_index,
193
+ image_feature_size_override=image_feature_size,
194
+ )
195
158
- raise TypeError(f"Invalid type for 'image': {type(image)}")
196
+ msg = f"Unsupported vision config: {type(vision_config)}"
197
+ raise NotImplementedError(msg)
198
199
161
-@MULTIMODAL_REGISTRY.register_image_input_mapper(_pixel_mapper)
200
+@MULTIMODAL_REGISTRY.register_image_input_mapper()
201
@INPUT_REGISTRY.register_dummy_data(dummy_data_for_llava_next)
202
+@INPUT_REGISTRY.register_input_processor(input_processor_for_llava_next)
203
class LlavaNextForConditionalGeneration(nn.Module, SupportsVision):
204
205
def __init__(self,
@@ -172,8 +212,8 @@ class LlavaNextForConditionalGeneration(nn.Module, SupportsVision):
212
self.config = config
213
self.vlm_config = vlm_config
214
215
+ # TODO: Optionally initializes this for supporting embeddings.
216
self.vision_tower = CLIPVisionModel(config=config.vision_config)
176
-
217
self.multi_modal_projector = LlavaMultiModalProjector(
218
vision_hidden_size=config.vision_config.hidden_size,
219
text_hidden_size=config.text_config.hidden_size,
@@ -196,24 +236,6 @@ class LlavaNextForConditionalGeneration(nn.Module, SupportsVision):
236
self.image_newline = nn.Parameter(
237
torch.empty(config.text_config.hidden_size))
238
199
- def _validate_image_pixels(self, data: torch.Tensor) -> torch.Tensor:
200
- _, num_channels, _, _ = self.vlm_config.image_input_shape
201
-
202
- # Note that this is different from that of vLLM vision_language_config
203
- # since the image is resized by the HuggingFace preprocessor
204
- height = width = self.config.vision_config.image_size
205
-
206
- if list(data.shape[2:]) != [num_channels, height, width]:
207
- raise ValueError(
208
- f"The expected image tensor shape is batch dimension plus "
209
- f"num_patches plus {[num_channels, height, width]}. "
210
- f"You supplied {data.shape}. "
211
- f"If you are using vLLM's entrypoint, make sure your "
212
- f"supplied image input is consistent with "
213
- f"image_input_shape in engine args.")
214
-
215
- return data
216
-
239
def _validate_image_sizes(self, data: torch.Tensor) -> torch.Tensor:
240
if list(data.shape[1:]) != [2]:
241
raise ValueError(
@@ -223,14 +245,14 @@ class LlavaNextForConditionalGeneration(nn.Module, SupportsVision):
245
return data
246
247
def _parse_and_validate_image_input(
226
- self, **kwargs: object) -> Optional[LlavaNextImageInputs]:
248
+ self, **kwargs: object) -> Optional[LlavaNextImagePixelInputs]:
249
pixel_values = kwargs.pop("pixel_values", None)
250
image_sizes = kwargs.pop("image_sizes", None)
251
230
- if pixel_values is None or image_sizes is None:
252
+ if pixel_values is None:
253
return None
254
233
- if not isinstance(pixel_values, torch.Tensor):
255
+ if not isinstance(pixel_values, (torch.Tensor, list)):
256
raise ValueError("Incorrect type of pixel values. "
257
f"Got type: {type(pixel_values)}")
258
@@ -240,7 +262,7 @@ class LlavaNextForConditionalGeneration(nn.Module, SupportsVision):
262
263
return LlavaNextImagePixelInputs(
264
type="pixel_values",
243
- data=self._validate_image_pixels(pixel_values),
265
+ data=pixel_values,
266
image_sizes=self._validate_image_sizes(image_sizes),
267
)
268
@@ -267,15 +289,14 @@ class LlavaNextForConditionalGeneration(nn.Module, SupportsVision):
289
strategy=self.config.vision_feature_select_strategy,
290
)
291
292
+ # Based on: https://github.com/haotian-liu/LLaVA/blob/main/llava/model/llava_arch.py
293
def _merge_image_patch_embeddings(self, image_size: torch.Tensor,
294
patch_embeddings: torch.Tensor, *,
295
strategy: str) -> torch.Tensor:
273
- # Based on: https://github.com/haotian-liu/LLaVA/blob/main/llava/model/llava_arch.py
296
if strategy == "flat":
297
return patch_embeddings.flatten(0, 1)
298
299
if strategy.startswith("spatial"):
278
- orig_width, orig_height = image_size
300
height = width = self.config.vision_config.image_size \
301
// self.config.vision_config.patch_size
302
@@ -289,13 +310,15 @@ class LlavaNextForConditionalGeneration(nn.Module, SupportsVision):
310
other_patch_embeds = patch_embeddings[1:]
311
312
# image_aspect_ratio == "anyres"
313
+ # Note: We follow the "wrong" width/height order
314
+ # [ref: PR huggingface/transformers#31588]
315
num_patch_width, num_patch_height = get_anyres_image_grid_shape(
293
- (orig_width, orig_height),
316
+ image_size,
317
self.config.image_grid_pinpoints,
318
self.config.vision_config.image_size,
319
)
320
other_patch_embeds = other_patch_embeds \
298
- .view(num_patch_width, num_patch_height, height, width, -1)
321
+ .view(num_patch_height, num_patch_width, height, width, -1)
322
323
if "unpad" in strategy:
324
other_patch_embeds = other_patch_embeds \
@@ -333,44 +356,53 @@ class LlavaNextForConditionalGeneration(nn.Module, SupportsVision):
356
raise ValueError(f"Unexpected patch merge strategy: {strategy}")
357
358
def _process_image_pixels(
336
- self, inputs: LlavaNextImagePixelInputs) -> torch.Tensor:
359
+ self,
360
+ inputs: LlavaNextImagePixelInputs,
361
+ ) -> BatchedTensors:
362
assert self.vision_tower is not None
363
364
pixel_values = inputs["data"]
365
341
- b, num_patches, c, h, w = pixel_values.shape
342
- stacked_pixel_values = pixel_values.view(b * num_patches, c, h, w)
366
+ if isinstance(pixel_values, torch.Tensor):
367
+ b, num_patches, c, h, w = pixel_values.shape
368
+ stacked_pixel_values = pixel_values.view(b * num_patches, c, h, w)
369
+ stacked_image_features = self._image_pixels_to_features(
370
+ self.vision_tower, stacked_pixel_values)
371
+ stacked_patch_embeddings = self.multi_modal_projector(
372
+ stacked_image_features)
373
374
+ return stacked_patch_embeddings.view(
375
+ b, num_patches, *stacked_patch_embeddings.shape[1:])
376
+
377
+ num_patches_per_batch = [v.shape[0] for v in pixel_values]
378
+ stacked_pixel_values = torch.cat(pixel_values)
379
stacked_image_features = self._image_pixels_to_features(
380
self.vision_tower, stacked_pixel_values)
381
347
- return stacked_image_features.view(b, num_patches,
348
- *stacked_image_features.shape[-2:])
382
+ return [
383
+ self.multi_modal_projector(image_features) for image_features in
384
+ torch.split(stacked_image_features, num_patches_per_batch)
385
+ ]
386
387
def _process_image_input(
351
- self, image_input: LlavaNextImageInputs) -> torch.Tensor:
352
- assert self.vision_tower is not None
353
- image_features = self._process_image_pixels(image_input)
354
-
355
- patch_embeddings = self.multi_modal_projector(image_features)
388
+ self, image_input: LlavaNextImageInputs) -> BatchedTensors:
389
+ patch_embeddings = self._process_image_pixels(image_input)
390
391
image_sizes = image_input.get("image_sizes")
392
if image_sizes is None:
359
- batch_size = image_input["data"].shape[0]
393
+ batch_size = len(image_input["data"])
394
vision_config = self.config.vision_config
361
- default_width = default_height = vision_config.image_size
362
- image_sizes = torch.as_tensor([[default_width, default_height]
395
+ default_height = default_width = vision_config.image_size
396
+ image_sizes = torch.as_tensor([[default_height, default_width]
397
for _ in range(batch_size)])
398
365
- merged_patch_embeddings = [
399
+ return [
400
self._merge_image_patch_embeddings(image_sizes[i],
367
- patch_features,
401
+ patch_features_batch,
402
strategy="spatial_unpad")
369
- for i, patch_features in enumerate(patch_embeddings)
403
+ for i, patch_features_batch in enumerate(patch_embeddings)
404
]
405
372
- return torch.stack(merged_patch_embeddings, dim=0)
373
-
406
def forward(
407
self,
408
input_ids: torch.Tensor,
@@ -404,8 +436,8 @@ class LlavaNextForConditionalGeneration(nn.Module, SupportsVision):
436
input_ids: Flattened (concatenated) input_ids corresponding to a
437
batch.
438
pixel_values: The pixels in each grid patch for each input image.
407
- Expects a batch with shape `[1, num_patches, 3, 336, 336]`.
408
- image_sizes: The original `(width, height)` for each input image.
439
+ Expects a batch with shape `[1, num_patches, 3, h, w]`.
440
+ image_sizes: The original `(height, width)` for each input image.
441
Expects a batch with shape `[1, 2]`.
442
443
See also:
vllm/model_executor/models/phi3v.py
+157
-65
@@ -13,7 +13,9 @@
13
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14
# See the License for the specific language governing permissions and
15
# limitations under the License.
16
-from typing import Dict, Iterable, List, Literal, Optional, Tuple, TypedDict
16
+import re
17
+from functools import lru_cache
18
+from typing import Iterable, List, Literal, Optional, Tuple, TypedDict
19
20
import numpy as np
21
import torch
@@ -22,8 +24,8 @@ from PIL import Image
24
from transformers import CLIPVisionConfig, PretrainedConfig
25
26
from vllm.attention import AttentionMetadata
25
-from vllm.config import CacheConfig, VisionLanguageConfig
26
-from vllm.inputs import INPUT_REGISTRY, InputContext
27
+from vllm.config import CacheConfig, ModelConfig, VisionLanguageConfig
28
+from vllm.inputs import INPUT_REGISTRY, InputContext, LLMInputs
29
from vllm.logger import init_logger
30
from vllm.model_executor.layers.logits_processor import LogitsProcessor
31
from vllm.model_executor.layers.quantization.base_config import (
@@ -34,10 +36,12 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
36
from vllm.model_executor.models.clip import CLIPVisionModel
37
from vllm.model_executor.models.llama import LlamaModel
38
from vllm.model_executor.sampling_metadata import SamplingMetadata
37
-from vllm.multimodal import MULTIMODAL_REGISTRY
39
+from vllm.multimodal import MULTIMODAL_REGISTRY, BatchedTensors
40
+from vllm.multimodal.image import cached_get_tokenizer
41
from vllm.sequence import IntermediateTensors, SamplerOutput
42
40
-from .clip import dummy_image_for_clip, dummy_seq_data_for_clip
43
+from .clip import (dummy_image_for_clip, dummy_seq_data_for_clip,
44
+ input_processor_for_clip)
45
from .interfaces import SupportsVision
46
47
logger = init_logger(__name__)
@@ -251,50 +255,22 @@ class Phi3HDImageEmbedding(Phi3ImageEmbeddingBase):
255
256
class Phi3VImagePixelInputs(TypedDict):
257
type: Literal["pixel_values"]
254
- data: torch.Tensor
255
- """Shape: (batch_size, 1 + num_patches, num_channels, height, width)"""
256
-
257
- image_sizes: torch.Tensor
258
- """Shape: (batch_size, 2)"""
259
-
260
-
261
-def _get_phi3v_image_feature_size(
262
- *,
263
- input_height: int,
264
- input_width: int,
265
-) -> int:
266
- h, w = input_height, input_width
267
-
268
- # https://huggingface.co/microsoft/Phi-3-vision-128k-instruct/blob/main/image_processing_phi3_v.py#L178
269
- return (h // 336 * w // 336 + 1) * 144 + 1 + (h // 336 + 1) * 12
258
+ data: BatchedTensors
259
+ """
260
+ Shape: `(batch_size, 1 + num_patches, num_channels, height, width)`
261
262
+ Note that `num_patches` may be different for each batch.
263
+ """
264
272
-def dummy_data_for_phi3v(ctx: InputContext, seq_len: int):
273
- multimodal_config = ctx.get_multimodal_config()
274
-
275
- #TODO: change the logic for dummy data to support dynamic shape
276
- _, _, dummy_height, dummy_width = multimodal_config.image_input_shape
277
- image_feature_size = _get_phi3v_image_feature_size(
278
- input_height=dummy_height,
279
- input_width=dummy_width,
280
- )
281
-
282
- seq_data = dummy_seq_data_for_clip(
283
- CLIP_VIT_LARGE_PATCH14_336_CONFIG,
284
- seq_len,
285
- image_token_id=32044,
286
- image_feature_size_override=image_feature_size,
287
- )
288
- mm_data = dummy_image_for_clip(
289
- CLIP_VIT_LARGE_PATCH14_336_CONFIG,
290
- image_width_override=dummy_width,
291
- image_height_override=dummy_height,
292
- )
265
+ image_sizes: torch.Tensor
266
+ """
267
+ Shape: `(batch_size, 2)`
268
294
- return seq_data, mm_data
269
+ This should be in `(height, width)` format.
270
+ """
271
272
297
-# Based on https://huggingface.co/microsoft/Phi-3-vision-128k-instruct/blob/main/image_processing_phi3_v.py
273
+# Based on https://huggingface.co/microsoft/Phi-3-vision-128k-instruct/blob/main/image_processing_phi3_v.py#L57
274
def _calc_padded_size(*, width: int, height: int, padding_unit: int = 336):
275
target_height = int(np.ceil(height / padding_unit) * padding_unit)
276
top_padding = int((target_height - height) / 2)
@@ -304,7 +280,7 @@ def _calc_padded_size(*, width: int, height: int, padding_unit: int = 336):
280
return padded_width, padded_height
281
282
307
-# Based on https://huggingface.co/microsoft/Phi-3-vision-128k-instruct/blob/main/image_processing_phi3_v.py
283
+# Based on https://huggingface.co/microsoft/Phi-3-vision-128k-instruct/blob/main/image_processing_phi3_v.py#L90
284
def _calc_hd_transform_size(*, width: int, height: int, hd_num: int = 16):
285
transposed = False
286
if width < height:
@@ -329,27 +305,133 @@ def _calc_hd_transform_size(*, width: int, height: int, hd_num: int = 16):
305
return padded_width, padded_height
306
307
332
-def _image_processor(ctx: InputContext,
333
- image: object) -> Dict[str, torch.Tensor]:
308
+# Based on https://huggingface.co/microsoft/Phi-3-vision-128k-instruct/blob/main/image_processing_phi3_v.py#L181
309
+def get_phi3v_image_feature_size(
310
+ hf_config: PretrainedConfig,
311
+ *,
312
+ input_height: int,
313
+ input_width: int,
314
+) -> int:
315
+ num_crops = getattr(hf_config, "num_crops", 16)
316
+ new_width, new_height = _calc_hd_transform_size(width=input_width,
317
+ height=input_height,
318
+ hd_num=num_crops)
319
335
- if isinstance(image, Image.Image):
336
- # Temporary patch before dynamic number of image tokens is supported
337
- _, _, h, w = ctx.get_multimodal_config().image_input_shape
338
- if (w, h) != _calc_hd_transform_size(width=image.width,
339
- height=image.height):
340
- logger.warning(
341
- "Dynamic image shape is currently not supported. "
342
- "Resizing input image to (%d, %d).", w, h)
320
+ return (new_height // 336 * new_width // 336 + 1) * 144 + 1 \
321
+ + (new_height // 336 + 1) * 12
322
344
- image = image.resize((w, h))
323
346
- return MULTIMODAL_REGISTRY._get_plugin("image") \
347
- ._default_input_mapper(ctx, image)
348
- raise TypeError(f"Invalid type for 'image': {type(image)}")
324
+def dummy_data_for_phi3v(ctx: InputContext, seq_len: int):
325
+ # Result in the max possible feature size (h:w = 16:1)
326
+ dummy_height, dummy_width = 8000, 50
327
+ image_feature_size = get_phi3v_image_feature_size(
328
+ ctx.get_hf_config(PretrainedConfig),
329
+ input_height=dummy_height,
330
+ input_width=dummy_width,
331
+ )
332
+
333
+ seq_data = dummy_seq_data_for_clip(
334
+ CLIP_VIT_LARGE_PATCH14_336_CONFIG,
335
+ seq_len,
336
+ image_token_id=32044,
337
+ image_feature_size_override=image_feature_size,
338
+ )
339
+ mm_data = dummy_image_for_clip(
340
+ CLIP_VIT_LARGE_PATCH14_336_CONFIG,
341
+ image_width_override=dummy_width,
342
+ image_height_override=dummy_height,
343
+ )
344
+
345
+ return seq_data, mm_data
346
+
347
348
+# Reserve this function to also handle placeholders for additional images
349
+# [ref: PR #5820]
350
+@lru_cache
351
+def _get_image_placeholder_token_ids(model_config: ModelConfig,
352
+ idx: int) -> List[int]:
353
+ assert idx > 0
354
351
-@MULTIMODAL_REGISTRY.register_image_input_mapper(_image_processor)
355
+ tokenizer = cached_get_tokenizer(model_config.tokenizer)
356
+
357
+ # We need to get the token for "<", not "▁<"
358
+ # https://huggingface.co/microsoft/Phi-3-vision-128k-instruct/raw/main/tokenizer.json
359
+ a_token_id, = tokenizer.encode("a", add_special_tokens=False)
360
+ a_token_id_, *image_placeholder_token_ids = tokenizer.encode(
361
+ f"a<|image_{idx}|>", add_special_tokens=False)
362
+ assert a_token_id == a_token_id_
363
+
364
+ return image_placeholder_token_ids
365
+
366
+
367
+def input_processor_for_phi3v(ctx: InputContext, llm_inputs: LLMInputs):
368
+ multi_modal_data = llm_inputs.get("multi_modal_data")
369
+ if multi_modal_data is None or "image" not in multi_modal_data:
370
+ return llm_inputs
371
+
372
+ model_config = ctx.model_config
373
+ multimodal_config = ctx.get_multimodal_config()
374
+ hf_config = ctx.get_hf_config(PretrainedConfig)
375
+
376
+ image_data = multi_modal_data["image"]
377
+ if isinstance(image_data, Image.Image):
378
+ w, h = image_data.size
379
+ w, h = _calc_hd_transform_size(width=w, height=h)
380
+
381
+ image_feature_size = get_phi3v_image_feature_size(hf_config,
382
+ input_width=w,
383
+ input_height=h)
384
+ elif isinstance(image_data, torch.Tensor):
385
+ raise NotImplementedError("Embeddings input is not supported yet")
386
+ else:
387
+ raise TypeError(f"Invalid image type: {type(image_data)}")
388
+
389
+ prompt = llm_inputs.get("prompt")
390
+ if prompt is None:
391
+ new_prompt = None
392
+ else:
393
+ if prompt.count("<|image|>") > 0:
394
+ logger.warning("Please follow the prompt format that is "
395
+ "documented on HuggingFace which does not involve "
396
+ "repeating <|image|> tokens.")
397
+ elif len(re.findall(r"(<\|image_\d+\|>)+", prompt)) > 1:
398
+ logger.warning("Multiple image input is not supported yet, "
399
+ "so any extra image tokens will be treated "
400
+ "as plain text.")
401
+
402
+ new_prompt = prompt
403
+
404
+ prompt_token_ids = llm_inputs["prompt_token_ids"]
405
+ image_1_token_ids = _get_image_placeholder_token_ids(model_config, idx=1)
406
+
407
+ new_token_ids: List[int] = []
408
+ for i in range(len(prompt_token_ids) - len(image_1_token_ids) + 1):
409
+ if prompt_token_ids[i:i + len(image_1_token_ids)] == image_1_token_ids:
410
+ new_token_ids.append(multimodal_config.image_token_id)
411
+
412
+ # No need to further scan the list since we only replace once
413
+ new_token_ids.extend(prompt_token_ids[i + len(image_1_token_ids):])
414
+ break
415
+ else:
416
+ new_token_ids.append(prompt_token_ids[i])
417
+
418
+ # NOTE: Create a defensive copy of the original inputs
419
+ llm_inputs = LLMInputs(prompt_token_ids=new_token_ids,
420
+ prompt=new_prompt,
421
+ multi_modal_data=multi_modal_data)
422
+
423
+ return input_processor_for_clip(
424
+ model_config,
425
+ CLIP_VIT_LARGE_PATCH14_336_CONFIG,
426
+ llm_inputs,
427
+ image_token_id=multimodal_config.image_token_id,
428
+ image_feature_size_override=image_feature_size,
429
+ )
430
+
431
+
432
+@MULTIMODAL_REGISTRY.register_image_input_mapper()
433
@INPUT_REGISTRY.register_dummy_data(dummy_data_for_phi3v)
434
+@INPUT_REGISTRY.register_input_processor(input_processor_for_phi3v)
435
class Phi3VForCausalLM(nn.Module, SupportsVision):
436
437
def __init__(self,
@@ -363,6 +445,8 @@ class Phi3VForCausalLM(nn.Module, SupportsVision):
445
self.vlm_config = vlm_config
446
447
self.model = LlamaModel(config, cache_config, quant_config)
448
+
449
+ # TODO: Optionally initializes this for supporting embeddings.
450
self.vision_embed_tokens = Phi3HDImageEmbedding(
451
vlm_config, config, self.model.embed_tokens)
452
self.lm_head = ParallelLMHead(config.vocab_size,
@@ -376,12 +460,20 @@ class Phi3VForCausalLM(nn.Module, SupportsVision):
460
pixel_values = kwargs.pop("pixel_values", None)
461
image_sizes = kwargs.pop("image_sizes", None)
462
379
- if pixel_values is not None and image_sizes is not None:
380
- return Phi3VImagePixelInputs(type="pixel_values",
381
- data=pixel_values,
382
- image_sizes=image_sizes)
463
+ if pixel_values is None:
464
+ return None
465
+
466
+ if not isinstance(pixel_values, (torch.Tensor, list)):
467
+ raise ValueError("Incorrect type of pixel values. "
468
+ f"Got type: {type(pixel_values)}")
469
+
470
+ if not isinstance(image_sizes, torch.Tensor):
471
+ raise ValueError("Incorrect type of image sizes. "
472
+ f"Got type: {type(image_sizes)}")
473
384
- return None
474
+ return Phi3VImagePixelInputs(type="pixel_values",
475
+ data=pixel_values,
476
+ image_sizes=image_sizes)
477
478
def forward(self,
479
input_ids: torch.Tensor,
vllm/model_executor/models/utils.py
new
+41
@@ -0,0 +1,41 @@
1
+import torch
2
+
3
+from vllm.multimodal import BatchedTensors
4
+
5
+
6
+def merge_vision_embeddings(input_ids: torch.Tensor,
7
+ inputs_embeds: torch.Tensor,
8
+ vision_embeddings: BatchedTensors,
9
+ image_token_id: int) -> torch.Tensor:
10
+ """
11
+ Merge `vision_embeddings` into `inputs_embeds` by overwriting the positions
12
+ in `inputs_embeds` corresponding to placeholder image tokens in `input_ids`.
13
+
14
+ Note:
15
+ This updates `inputs_embeds` in place.
16
+ """
17
+ mask = (input_ids == image_token_id)
18
+ num_expected_tokens = mask.sum()
19
+
20
+ if isinstance(vision_embeddings, torch.Tensor):
21
+ batch_size, batch_tokens, *_, embed_dim = vision_embeddings.shape
22
+ total_tokens = batch_size * batch_tokens
23
+ if num_expected_tokens != total_tokens:
24
+ expr = f"{batch_size} x {batch_tokens}"
25
+ raise ValueError(
26
+ f"Attempted to assign {expr} = {total_tokens} "
27
+ f"image tokens to {num_expected_tokens} placeholders")
28
+
29
+ inputs_embeds[mask] = vision_embeddings.view(total_tokens, embed_dim)
30
+ else:
31
+ size_per_batch = [t.shape[0] for t in vision_embeddings]
32
+ total_tokens = sum(size_per_batch)
33
+ if num_expected_tokens != total_tokens:
34
+ expr = ' + '.join(map(str, size_per_batch))
35
+ raise ValueError(
36
+ f"Attempted to assign {expr} = {total_tokens} "
37
+ f"image tokens to {num_expected_tokens} placeholders")
38
+
39
+ inputs_embeds[mask] = torch.cat(vision_embeddings)
40
+
41
+ return inputs_embeds
vllm/multimodal/__init__.py
+5
-2
@@ -1,4 +1,5 @@
1
-from .base import MultiModalDataDict, MultiModalPlugin
1
+from .base import (BatchedTensors, MultiModalDataDict, MultiModalInputs,
2
+ MultiModalPlugin)
3
from .registry import MultiModalRegistry
4
5
MULTIMODAL_REGISTRY = MultiModalRegistry()
@@ -11,8 +12,10 @@ See also:
12
"""
13
14
__all__ = [
15
+ "BatchedTensors",
16
+ "MultiModalDataDict",
17
+ "MultiModalInputs",
18
"MultiModalPlugin",
19
"MULTIMODAL_REGISTRY",
20
"MultiModalRegistry",
17
- "MultiModalDataDict",
21
]
vllm/multimodal/base.py
+85
-16
@@ -1,23 +1,90 @@
1
+import sys
2
from abc import ABC, abstractmethod
2
-from typing import (TYPE_CHECKING, Any, Callable, Dict, Optional, Type,
3
- TypedDict, TypeVar, Union)
3
+from collections import UserDict, defaultdict
4
+from typing import (Any, Callable, Dict, List, Optional, Type, TypedDict,
5
+ TypeVar, Union)
6
+
7
+import torch
8
+import torch.types
9
+from PIL import Image
10
+from torch import nn
11
12
from vllm.config import ModelConfig
13
from vllm.inputs import InputContext
14
from vllm.logger import init_logger
15
9
-if TYPE_CHECKING:
10
- import torch
11
- from PIL import Image
12
- from torch import nn
13
-
16
logger = init_logger(__name__)
17
16
-N = TypeVar("N", bound=Type["nn.Module"])
18
+BatchedTensors = Union[torch.Tensor, List[torch.Tensor]]
19
+"""
20
+If each input tensor in the batch has the same size, this is a single batched
21
+tensor; otherwise, this is a list of tensors with one element per batch.
22
+"""
23
+
24
+if sys.version_info < (3, 9):
25
+ # UserDict cannot be subscripted
26
+ class _MultiModalInputsBase(UserDict):
27
+ pass
28
+else:
29
+
30
+ class _MultiModalInputsBase(UserDict[str, torch.Tensor]):
31
+ pass
32
+
33
+
34
+class MultiModalInputs(_MultiModalInputsBase):
35
+ """
36
+ A dictionary that represents the keyword arguments to
37
+ :meth:`~torch.nn.Module.forward`.
38
+ """
39
+
40
+ @staticmethod
41
+ def try_concat(
42
+ tensors: List[torch.Tensor],
43
+ *,
44
+ device: torch.types.Device,
45
+ ) -> BatchedTensors:
46
+ # Avoid initializing CUDA too early
47
+ import torch
48
+
49
+ unbatched_shape = tensors[0].shape[1:]
50
+
51
+ for tensor in tensors:
52
+ if tensor.shape[1:] != unbatched_shape:
53
+ return [
54
+ tensor.squeeze(0).to(device=device) for tensor in tensors
55
+ ]
56
+
57
+ return torch.cat(tensors, dim=0).to(device=device)
58
+
59
+ @staticmethod
60
+ def batch(
61
+ inputs_list: List["MultiModalInputs"],
62
+ device: torch.types.Device,
63
+ ) -> Dict[str, BatchedTensors]:
64
+ """Batch multiple inputs together into a dictionary."""
65
+ if len(inputs_list) == 0:
66
+ return {}
67
+
68
+ keys = inputs_list[0].keys()
69
+
70
+ item_lists: Dict[str, List[torch.Tensor]] = defaultdict(list)
71
+
72
+ for inputs in inputs_list:
73
+ if inputs.keys() != keys:
74
+ msg = f"Inputs do not share the same keys ({keys})"
75
+ raise ValueError(msg)
76
+
77
+ for k, v in inputs.items():
78
+ item_lists[k].append(v)
79
+
80
+ return {
81
+ k: MultiModalInputs.try_concat(item_list, device=device)
82
+ for k, item_list in item_lists.items()
83
+ }
84
85
86
class MultiModalDataBuiltins(TypedDict, total=False):
20
- image: "Image.Image"
87
+ image: Image.Image
88
89
90
MultiModalDataDict = Union[MultiModalDataBuiltins, Dict[str, Any]]
@@ -29,12 +96,13 @@ to the model by the corresponding mapper. By default, the mapper of
96
the corresponding plugin with the same modality key is applied.
97
"""
98
32
-MultiModalInputMapper = Callable[[InputContext, object], Dict[str,
33
- "torch.Tensor"]]
99
+MultiModalInputMapper = Callable[[InputContext, object], MultiModalInputs]
100
"""Return a dictionary to be passed as keyword arguments to
101
:meth:`~torch.nn.Module.forward`. This is similar in concept to tokenizers
102
and processors in HuggingFace Transformers."""
103
104
+N = TypeVar("N", bound=Type[nn.Module])
105
+
106
107
class MultiModalPlugin(ABC):
108
"""
@@ -48,8 +116,7 @@ class MultiModalPlugin(ABC):
116
"""
117
118
def __init__(self) -> None:
51
- self._input_mappers: Dict[Type["nn.Module"],
52
- MultiModalInputMapper] = {}
119
+ self._input_mappers: Dict[Type[nn.Module], MultiModalInputMapper] = {}
120
121
@abstractmethod
122
def get_data_key(self) -> str:
@@ -60,7 +127,7 @@ class MultiModalPlugin(ABC):
127
128
@abstractmethod
129
def _default_input_mapper(self, ctx: InputContext,
63
- data: object) -> Dict[str, "torch.Tensor"]:
130
+ data: object) -> MultiModalInputs:
131
"""Return a dictionary to be passed as keyword arguments to
132
:meth:`~torch.nn.Module.forward`. This is similar in concept to
133
tokenizers and processors in HuggingFace Transformers.
@@ -80,6 +147,7 @@ class MultiModalPlugin(ABC):
147
148
See also:
149
:ref:`input_processing_pipeline`
150
+ :ref:`adding_a_new_multimodal_model`
151
"""
152
153
def wrapper(model_cls: N) -> N:
@@ -97,7 +165,7 @@ class MultiModalPlugin(ABC):
165
return wrapper
166
167
def map_input(self, model_config: ModelConfig,
100
- data: object) -> Dict[str, "torch.Tensor"]:
168
+ data: object) -> MultiModalInputs:
169
"""
170
Apply an input mapper to a data passed
171
to the model, transforming the data into a dictionary of model inputs.
@@ -106,7 +174,8 @@ class MultiModalPlugin(ABC):
174
175
The model is identified by ``model_config``.
176
109
- TODO: Add guide [ref: PR #5276]
177
+ See also:
178
+ :ref:`adding_a_new_multimodal_model`
179
"""
180
# Avoid circular import
181
from vllm.model_executor.model_loader import get_model_architecture
vllm/multimodal/image.py
+94
-6
@@ -1,19 +1,102 @@
1
from functools import lru_cache
2
-from typing import Dict
2
+from typing import List, Optional, Tuple, TypeVar
3
4
import torch
5
from PIL import Image
6
+from transformers import PreTrainedTokenizerBase
7
8
from vllm.config import ModelConfig
9
from vllm.inputs.registry import InputContext
10
from vllm.logger import init_logger
11
from vllm.transformers_utils.image_processor import get_image_processor
12
+from vllm.transformers_utils.tokenizer import get_tokenizer
13
12
-from .base import MultiModalPlugin
14
+from .base import MultiModalInputs, MultiModalPlugin
15
16
logger = init_logger(__name__)
17
18
cached_get_image_processor = lru_cache(get_image_processor)
19
+cached_get_tokenizer = lru_cache(get_tokenizer)
20
+
21
+# Utilities for image input processors
22
+_T = TypeVar("_T", str, int)
23
+
24
+
25
+def repeat_and_pad_token(
26
+ token: _T,
27
+ *,
28
+ repeat_count: int = 1,
29
+ pad_token_left: Optional[_T] = None,
30
+ pad_token_right: Optional[_T] = None,
31
+) -> List[_T]:
32
+ replacement = [token] * repeat_count
33
+ if pad_token_left is not None:
34
+ replacement = [pad_token_left] + replacement
35
+ if pad_token_right is not None:
36
+ replacement = replacement + [pad_token_right]
37
+
38
+ return replacement
39
+
40
+
41
+def repeat_and_pad_image_tokens(
42
+ tokenizer: PreTrainedTokenizerBase,
43
+ prompt: Optional[str],
44
+ prompt_token_ids: List[int],
45
+ *,
46
+ image_token_id: int,
47
+ repeat_count: int = 1,
48
+ pad_token_left: Optional[int] = None,
49
+ pad_token_right: Optional[int] = None,
50
+) -> Tuple[Optional[str], List[int]]:
51
+ if prompt is None:
52
+ new_prompt = None
53
+ else:
54
+ image_token_str = tokenizer.decode(image_token_id)
55
+ pad_token_str_left = (None if pad_token_left is None else
56
+ tokenizer.decode(pad_token_left))
57
+ pad_token_str_right = (None if pad_token_right is None else
58
+ tokenizer.decode(pad_token_right))
59
+ replacement_str = "".join(
60
+ repeat_and_pad_token(
61
+ image_token_str,
62
+ repeat_count=repeat_count,
63
+ pad_token_left=pad_token_str_left,
64
+ pad_token_right=pad_token_str_right,
65
+ ))
66
+
67
+ image_token_count = prompt.count(image_token_str)
68
+ # This is an arbitrary number to distinguish between the two cases
69
+ if image_token_count > 16:
70
+ logger.warning(
71
+ "Please follow the prompt format that is "
72
+ "documented on HuggingFace which does not involve "
73
+ "repeating %s tokens.", image_token_str)
74
+ elif image_token_count > 1:
75
+ logger.warning("Multiple image input is not supported yet, "
76
+ "so any extra image tokens will be treated "
77
+ "as plain text.")
78
+
79
+ # The image tokens are removed to be consistent with HuggingFace
80
+ new_prompt = prompt.replace(image_token_str, replacement_str, 1)
81
+
82
+ new_token_ids: List[int] = []
83
+ for i, token in enumerate(prompt_token_ids):
84
+ if token == image_token_id:
85
+ replacement_ids = repeat_and_pad_token(
86
+ image_token_id,
87
+ repeat_count=repeat_count,
88
+ pad_token_left=pad_token_left,
89
+ pad_token_right=pad_token_right,
90
+ )
91
+ new_token_ids.extend(replacement_ids)
92
+
93
+ # No need to further scan the list since we only replace once
94
+ new_token_ids.extend(prompt_token_ids[i + 1:])
95
+ break
96
+ else:
97
+ new_token_ids.append(token)
98
+
99
+ return new_prompt, new_token_ids
100
101
102
class ImagePlugin(MultiModalPlugin):
@@ -27,7 +110,7 @@ class ImagePlugin(MultiModalPlugin):
110
trust_remote_code=model_config.trust_remote_code)
111
112
def _default_input_mapper(self, ctx: InputContext,
30
- data: object) -> Dict[str, torch.Tensor]:
113
+ data: object) -> MultiModalInputs:
114
model_config = ctx.model_config
115
if isinstance(data, Image.Image):
116
image_processor = self._get_hf_image_processor(model_config)
@@ -35,10 +118,15 @@ class ImagePlugin(MultiModalPlugin):
118
raise RuntimeError("No HuggingFace processor is available"
119
"to process the image object")
120
try:
38
- return image_processor.preprocess(data, return_tensors="pt") \
39
- .to(model_config.dtype).data
121
+ batch_data = image_processor \
122
+ .preprocess(data, return_tensors="pt") \
123
+ .data
124
except Exception:
125
logger.error("Failed to process image (%s)", data)
126
raise
127
44
- raise TypeError(f"Invalid type for 'image': {type(data)}")
128
+ return MultiModalInputs(batch_data)
129
+ elif isinstance(data, torch.Tensor):
130
+ raise NotImplementedError("Embeddings input is not supported yet")
131
+
132
+ raise TypeError(f"Invalid image type: {type(data)}")
vllm/multimodal/registry.py
+22
-11
@@ -1,18 +1,17 @@
1
import functools
2
-from typing import Optional, Sequence, Type, TypeVar
2
+from typing import Dict, Optional, Sequence
3
4
-from torch import nn
4
+import torch
5
6
from vllm.config import ModelConfig
7
from vllm.logger import init_logger
8
9
-from .base import MultiModalDataDict, MultiModalInputMapper, MultiModalPlugin
9
+from .base import (MultiModalDataDict, MultiModalInputMapper, MultiModalInputs,
10
+ MultiModalPlugin)
11
from .image import ImagePlugin
12
13
logger = init_logger(__name__)
14
14
-N = TypeVar("N", bound=Type[nn.Module])
15
-
15
16
class MultiModalRegistry:
17
"""
@@ -61,7 +60,7 @@ class MultiModalRegistry:
60
return self.register_input_mapper("image", mapper)
61
62
def _process_input(self, key: str, value: object,
64
- model_config: ModelConfig):
63
+ model_config: ModelConfig) -> MultiModalInputs:
64
plugin = self._plugins.get(key)
65
if plugin:
66
return plugin.map_input(model_config, value)
@@ -93,16 +92,28 @@ class MultiModalRegistry:
92
"""
93
return self.register_input_mapper("image", mapper)
94
96
- def map_input(self, model_config: ModelConfig, data: MultiModalDataDict):
95
+ def map_input(self, model_config: ModelConfig,
96
+ data: MultiModalDataDict) -> MultiModalInputs:
97
"""
98
Apply an input mapper to the data passed to the model.
99
100
See :meth:`MultiModalPlugin.map_input` for more details.
101
"""
102
- result_list = [
103
- self._process_input(k, v, model_config) for k, v in data.items()
104
- ]
105
- return {k: v for d in result_list for k, v in d.items()}
102
+ merged_dict: Dict[str, torch.Tensor] = {}
103
+
104
+ for data_key, data_value in data.items():
105
+ input_dict = self._process_input(data_key, data_value,
106
+ model_config)
107
+
108
+ for input_key, input_tensor in input_dict.items():
109
+ if input_key in merged_dict:
110
+ raise ValueError(f"The input mappers (keys={set(data)}) "
111
+ f"resulted in a conflicting keyword "
112
+ f"argument to `forward()`: {input_key}")
113
+
114
+ merged_dict[input_key] = input_tensor
115
+
116
+ return MultiModalInputs(merged_dict)
117
118
def create_input_mapper(self, model_config: ModelConfig):
119
"""
vllm/multimodal/utils.py
+61
-34
@@ -4,11 +4,56 @@ from typing import Optional, Union
4
from urllib.parse import urlparse
5
6
import aiohttp
7
+import requests
8
from PIL import Image
9
9
-from vllm.config import ModelConfig
10
from vllm.envs import VLLM_IMAGE_FETCH_TIMEOUT
11
from vllm.multimodal.base import MultiModalDataDict
12
+from vllm.version import __version__ as VLLM_VERSION
13
+
14
+
15
+def _validate_remote_url(url: str, *, name: str):
16
+ parsed_url = urlparse(url)
17
+ if parsed_url.scheme not in ["http", "https"]:
18
+ raise ValueError(f"Invalid '{name}': A valid '{name}' "
19
+ "must have scheme 'http' or 'https'.")
20
+
21
+
22
+def _get_request_headers():
23
+ return {"User-Agent": f"vLLM/{VLLM_VERSION}"}
24
+
25
+
26
+def _load_image_from_bytes(b: bytes):
27
+ image = Image.open(BytesIO(b))
28
+ image.load()
29
+ return image
30
+
31
+
32
+def _load_image_from_data_url(image_url: str):
33
+ # Only split once and assume the second part is the base64 encoded image
34
+ _, image_base64 = image_url.split(",", 1)
35
+ return load_image_from_base64(image_base64)
36
+
37
+
38
+def fetch_image(image_url: str) -> Image.Image:
39
+ """Load PIL image from a url or base64 encoded openai GPT4V format"""
40
+ if image_url.startswith('http'):
41
+ _validate_remote_url(image_url, name="image_url")
42
+
43
+ headers = _get_request_headers()
44
+
45
+ with requests.get(url=image_url, headers=headers) as response:
46
+ response.raise_for_status()
47
+ image_raw = response.content
48
+ image = _load_image_from_bytes(image_raw)
49
+
50
+ elif image_url.startswith('data:image'):
51
+ image = _load_image_from_data_url(image_url)
52
+ else:
53
+ raise ValueError("Invalid 'image_url': A valid 'image_url' must start "
54
+ "with either 'data:image' or 'http'.")
55
+
56
+ return image
57
58
59
class ImageFetchAiohttp:
@@ -29,34 +74,31 @@ class ImageFetchAiohttp:
74
"""Load PIL image from a url or base64 encoded openai GPT4V format"""
75
76
if image_url.startswith('http'):
32
- parsed_url = urlparse(image_url)
33
- if parsed_url.scheme not in ["http", "https"]:
34
- raise ValueError("Invalid 'image_url': A valid 'image_url' "
35
- "must have scheme 'http' or 'https'.")
36
- # Avoid circular import
37
- from vllm import __version__ as VLLM_VERSION
77
+ _validate_remote_url(image_url, name="image_url")
78
79
client = cls.get_aiohttp_client()
40
- headers = {"User-Agent": f"vLLM/{VLLM_VERSION}"}
80
+ headers = _get_request_headers()
81
82
async with client.get(url=image_url, headers=headers) as response:
83
response.raise_for_status()
84
image_raw = await response.read()
45
- image = Image.open(BytesIO(image_raw))
85
+ image = _load_image_from_bytes(image_raw)
86
47
- # Only split once and assume the second part is the base64 encoded image
87
elif image_url.startswith('data:image'):
49
- image = load_image_from_base64(image_url.split(',', 1)[1])
50
-
88
+ image = _load_image_from_data_url(image_url)
89
else:
90
raise ValueError(
91
"Invalid 'image_url': A valid 'image_url' must start "
92
"with either 'data:image' or 'http'.")
93
56
- image.load()
94
return image
95
96
97
+async def async_get_and_parse_image(image_url: str) -> MultiModalDataDict:
98
+ image = await ImageFetchAiohttp.fetch_image(image_url)
99
+ return {"image": image}
100
+
101
+
102
def encode_image_base64(image: Image.Image, format: str = 'JPEG') -> str:
103
"""Encode a pillow image to base64 format."""
104
@@ -69,26 +111,11 @@ def encode_image_base64(image: Image.Image, format: str = 'JPEG') -> str:
111
112
def load_image_from_base64(image: Union[bytes, str]) -> Image.Image:
113
"""Load image from base64 format."""
72
- return Image.open(BytesIO(base64.b64decode(image)))
114
+ return _load_image_from_bytes(base64.b64decode(image))
115
116
75
-# TODO(ywang96): move this to a model registry for preprocessing vision
76
-# language prompts based on the model type.
77
-def get_full_image_text_prompt(image_prompt: str, text_prompt: str,
78
- config: ModelConfig) -> str:
79
- """Combine image and text prompts for vision language model depending on
80
- the model architecture."""
81
-
82
- if config.hf_config.model_type in ("llava", "llava_next"):
83
- full_prompt = f"{image_prompt}\n{text_prompt}"
84
- elif config.hf_config.model_type == 'phi3_v':
85
- full_prompt = f"{image_prompt}<s>\n{text_prompt}"
86
- else:
87
- raise ValueError(
88
- f"Unsupported model type: {config.hf_config.model_type}")
89
- return full_prompt
90
-
91
-
92
-async def async_get_and_parse_image(image_url: str) -> MultiModalDataDict:
93
- image = await ImageFetchAiohttp.fetch_image(image_url)
94
- return {"image": image}
117
+def rescale_image_size(image: Image.Image, size_factor: float) -> Image.Image:
118
+ """Rescale the dimensions of an image by a constant factor."""
119
+ new_width = int(image.width * size_factor)
120
+ new_height = int(image.height * size_factor)
121
+ return image.resize((new_width, new_height))
vllm/sequence.py
+1
-1
@@ -457,7 +457,7 @@ class SequenceGroup:
457
return next(iter(self.seqs_dict.values())).prompt_token_ids
458
459
@property
460
- def multi_modal_data(self) -> Optional["MultiModalDataDict"]:
460
+ def multi_modal_data(self) -> "MultiModalDataDict":
461
# All sequences in the group should have the same multi-modal data.
462
# We use the multi-modal data of an arbitrary sequence.
463
return next(iter(self.seqs_dict.values())).multi_modal_data
vllm/transformers_utils/image_processor.py
+9
-9
@@ -1,9 +1,4 @@
1
-from transformers import AutoImageProcessor
2
-from transformers.image_processing_utils import BaseImageProcessor
3
-
4
-from vllm.logger import init_logger
5
-
6
-logger = init_logger(__name__)
1
+from typing import cast
2
3
4
def get_image_processor(
@@ -11,10 +6,15 @@ def get_image_processor(
6
*args,
7
trust_remote_code: bool = False,
8
**kwargs,
14
-) -> BaseImageProcessor:
9
+):
10
"""Gets an image processor for the given model name via HuggingFace."""
11
+ # don't put this import at the top level
12
+ # it will call torch.cuda.device_count()
13
+ from transformers import AutoImageProcessor
14
+ from transformers.image_processing_utils import BaseImageProcessor
15
+
16
try:
17
- processor: BaseImageProcessor = AutoImageProcessor.from_pretrained(
17
+ processor = AutoImageProcessor.from_pretrained(
18
processor_name,
19
*args,
20
trust_remote_code=trust_remote_code,
@@ -34,4 +34,4 @@ def get_image_processor(
34
else:
35
raise e
36
37
- return processor
37
+ return cast(BaseImageProcessor, processor)
vllm/worker/cpu_model_runner.py
+15
-19
@@ -1,6 +1,6 @@
1
-from collections import defaultdict
1
from dataclasses import dataclass
3
-from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Type, Union
2
+from typing import (TYPE_CHECKING, Any, Dict, List, Mapping, Optional, Tuple,
3
+ Type, Union)
4
5
import torch
6
from torch import nn
@@ -12,7 +12,8 @@ from vllm.config import (CacheConfig, DeviceConfig, LoadConfig, LoRAConfig,
12
from vllm.logger import init_logger
13
from vllm.model_executor import SamplingMetadata
14
from vllm.model_executor.model_loader import get_model
15
-from vllm.multimodal import MULTIMODAL_REGISTRY
15
+from vllm.multimodal import (MULTIMODAL_REGISTRY, BatchedTensors,
16
+ MultiModalInputs)
17
from vllm.sequence import (IntermediateTensors, SamplerOutput,
18
SequenceGroupMetadata)
19
from vllm.utils import make_tensor_with_pad
@@ -40,7 +41,7 @@ class CPUModelInput(ModelRunnerInputBase):
41
input_positions: Optional[torch.Tensor] = None
42
attn_metadata: Optional["AttentionMetadata"] = None
43
sampling_metadata: Optional["SamplingMetadata"] = None
43
- multi_modal_kwargs: Optional[Dict[str, torch.Tensor]] = None
44
+ multi_modal_kwargs: Optional[Mapping[str, BatchedTensors]] = None
45
46
def as_broadcastable_tensor_dict(
47
self) -> Dict[str, Union[int, torch.Tensor]]:
@@ -132,15 +133,14 @@ class CPUModelRunner(ModelRunnerBase[CPUModelInput]):
133
def _prepare_prompt(
134
self,
135
seq_group_metadata_list: List[SequenceGroupMetadata],
135
- ) -> Tuple[torch.Tensor, torch.Tensor, AttentionMetadata, List[int], Dict[
136
- str, torch.Tensor]]:
136
+ ) -> Tuple[torch.Tensor, torch.Tensor, AttentionMetadata, List[int],
137
+ Mapping[str, BatchedTensors]]:
138
assert len(seq_group_metadata_list) > 0
139
input_tokens: List[int] = []
140
input_positions: List[int] = []
141
slot_mapping: List[int] = []
142
seq_lens: List[int] = []
142
- multi_modal_kwargs_list: Dict[str,
143
- List[torch.Tensor]] = defaultdict(list)
143
+ multi_modal_inputs_list: List[MultiModalInputs] = []
144
145
for seq_group_metadata in seq_group_metadata_list:
146
assert seq_group_metadata.is_prompt
@@ -162,10 +162,9 @@ class CPUModelRunner(ModelRunnerBase[CPUModelInput]):
162
input_positions.extend(list(range(computed_len, seq_len)))
163
164
mm_data = seq_group_metadata.multi_modal_data
165
- if mm_data is not None:
165
+ if mm_data:
166
mm_kwargs = self.multi_modal_input_mapper(mm_data)
167
- for k, v in mm_kwargs.items():
168
- multi_modal_kwargs_list[k].append(v)
167
+ multi_modal_inputs_list.append(mm_kwargs)
168
169
# Compute the slot mapping.
170
block_table = seq_group_metadata.block_tables[seq_id]
@@ -189,11 +188,6 @@ class CPUModelRunner(ModelRunnerBase[CPUModelInput]):
188
slot = block_number * self.block_size + block_offset
189
slot_mapping.append(slot)
190
192
- multi_modal_kwargs = {
193
- k: torch.cat(v, dim=0).to(self.device)
194
- for k, v in multi_modal_kwargs_list.items()
195
- }
196
-
191
num_prompt_tokens = len(input_tokens)
192
193
input_tokens = torch.tensor(input_tokens,
@@ -217,6 +211,10 @@ class CPUModelRunner(ModelRunnerBase[CPUModelInput]):
211
block_tables=torch.tensor([]),
212
slot_mapping=slot_mapping,
213
)
214
+
215
+ multi_modal_kwargs = MultiModalInputs.batch(multi_modal_inputs_list,
216
+ device=self.device)
217
+
218
return (input_tokens, input_positions, attn_metadata, seq_lens,
219
multi_modal_kwargs)
220
@@ -367,10 +365,8 @@ class CPUModelRunner(ModelRunnerBase[CPUModelInput]):
365
"positions": model_input.input_positions,
366
"kv_caches": kv_caches,
367
"attn_metadata": model_input.attn_metadata,
368
+ **(model_input.multi_modal_kwargs or {}),
369
}
371
- if (self.vision_language_config
372
- and model_input.multi_modal_kwargs is not None):
373
- execute_model_kwargs.update(model_input.multi_modal_kwargs)
370
371
hidden_states = model_executable(**execute_model_kwargs)
372
vllm/worker/embedding_model_runner.py
+2
-3
@@ -92,10 +92,9 @@ class EmbeddingModelRunner(
92
"positions": model_input.input_positions,
93
"kv_caches": kv_caches,
94
"attn_metadata": model_input.attn_metadata,
95
+ **(model_input.multi_modal_kwargs or {}),
96
}
96
- if self.vision_language_config:
97
- multi_modal_kwargs = model_input.multi_modal_kwargs or {}
98
- execute_model_kwargs.update({"image_input": multi_modal_kwargs})
97
+
98
hidden_states = model_executable(**execute_model_kwargs)
99
100
# Only perform pooling in the driver worker.
vllm/worker/model_runner.py
+14
-13
@@ -3,8 +3,8 @@ import gc
3
import time
4
import warnings
5
from collections import defaultdict
6
-from typing import (TYPE_CHECKING, Any, Dict, List, Optional, Set, Tuple, Type,
7
- TypeVar, Union)
6
+from typing import (TYPE_CHECKING, Any, Dict, List, Mapping, Optional, Set,
7
+ Tuple, Type, TypeVar, Union)
8
9
import numpy as np
10
import torch
@@ -37,7 +37,8 @@ from vllm.model_executor import SamplingMetadata
37
from vllm.model_executor.model_loader import get_model
38
from vllm.model_executor.model_loader.tensorizer import TensorizerConfig
39
from vllm.model_executor.models.interfaces import supports_lora
40
-from vllm.multimodal import MULTIMODAL_REGISTRY
40
+from vllm.multimodal import (MULTIMODAL_REGISTRY, BatchedTensors,
41
+ MultiModalInputs)
42
from vllm.sampling_params import SamplingParams
43
from vllm.sequence import (IntermediateTensors, SamplerOutput,
44
SequenceGroupMetadata)
@@ -83,7 +84,7 @@ class ModelInputForGPU(ModelRunnerInputBase):
84
lora_mapping: Optional["LoRAMapping"] = None
85
lora_requests: Optional[Set[LoRARequest]] = None
86
attn_metadata: Optional["AttentionMetadata"] = None
86
- multi_modal_kwargs: Optional[Dict[str, torch.Tensor]] = None
87
+ multi_modal_kwargs: Optional[Mapping[str, BatchedTensors]] = None
88
request_ids_to_seq_ids: Optional[Dict[str, List[int]]] = None
89
finished_requests_ids: Optional[List[str]] = None
90
virtual_engine: int = 0
@@ -356,8 +357,7 @@ class GPUModelRunnerBase(ModelRunnerBase[TModelInputForGPU]):
357
context_lens: List[int] = []
358
query_lens: List[int] = []
359
block_tables: List[List[int]] = []
359
- multi_modal_kwargs_list: Dict[str,
360
- List[torch.Tensor]] = defaultdict(list)
360
+ multi_modal_inputs_list: List[MultiModalInputs] = []
361
request_ids_to_seq_ids: Dict[str, List[int]] = defaultdict(list)
362
decode_only = True
363
num_prefills = 0
@@ -528,8 +528,7 @@ class GPUModelRunnerBase(ModelRunnerBase[TModelInputForGPU]):
528
if mm_data:
529
# Process multi-modal data
530
mm_kwargs = self.multi_modal_input_mapper(mm_data)
531
- for k, v in mm_kwargs.items():
532
- multi_modal_kwargs_list[k].append(v)
531
+ multi_modal_inputs_list.append(mm_kwargs)
532
533
is_profile_run = _is_block_tables_empty(
534
seq_group_metadata.block_tables)
@@ -746,10 +745,8 @@ class GPUModelRunnerBase(ModelRunnerBase[TModelInputForGPU]):
745
else:
746
lora_mapping = None
747
749
- multi_modal_kwargs = {
750
- k: torch.cat(v, dim=0).to(self.device)
751
- for k, v in multi_modal_kwargs_list.items()
752
- }
748
+ multi_modal_kwargs = MultiModalInputs.batch(multi_modal_inputs_list,
749
+ device=self.device)
750
request_ids_to_seq_ids = {
751
seq_group_metadata.request_id:
752
list(seq_group_metadata.seq_data.keys())
@@ -821,7 +818,11 @@ class GPUModelRunnerBase(ModelRunnerBase[TModelInputForGPU]):
818
819
seq_data, dummy_multi_modal_data = INPUT_REGISTRY \
820
.dummy_data_for_profiling(model_config, seq_len)
824
- assert len(seq_data.prompt_token_ids) == seq_len
821
+
822
+ # Having more tokens is over-conservative but otherwise fine
823
+ assert len(seq_data.prompt_token_ids) >= seq_len, (
824
+ f"Expected at least {seq_len} dummy tokens for profiling, "
825
+ f"but got: {len(seq_data.prompt_token_ids)}")
826
827
seq = SequenceGroupMetadata(
828
request_id=str(group_id),
vllm/worker/neuron_model_runner.py
+29
-6
@@ -1,5 +1,6 @@
1
from dataclasses import dataclass
2
-from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union
2
+from typing import (TYPE_CHECKING, Any, Dict, List, Mapping, Optional, Tuple,
3
+ Union)
4
5
import torch
6
from torch import nn
@@ -9,6 +10,8 @@ from vllm.config import (DeviceConfig, ModelConfig, ParallelConfig,
10
from vllm.logger import init_logger
11
from vllm.model_executor import SamplingMetadata
12
from vllm.model_executor.model_loader.neuron import get_neuron_model
13
+from vllm.multimodal import (MULTIMODAL_REGISTRY, BatchedTensors,
14
+ MultiModalInputs)
15
from vllm.sequence import (IntermediateTensors, SamplerOutput,
16
SequenceGroupMetadata)
17
from vllm.utils import is_pin_memory_available, make_tensor_with_pad
@@ -29,6 +32,7 @@ class ModelInputForNeuron(ModelRunnerInputBase):
32
input_positions: Optional[torch.Tensor] = None
33
input_block_ids: Optional[torch.Tensor] = None
34
sampling_metadata: Optional["SamplingMetadata"] = None
35
+ multi_modal_kwargs: Optional[Mapping[str, BatchedTensors]] = None
36
37
def as_broadcastable_tensor_dict(
38
self) -> Dict[str, Union[int, torch.Tensor]]:
@@ -65,6 +69,10 @@ class NeuronModelRunner(ModelRunnerBase[ModelInputForNeuron]):
69
self.device = self.device_config.device
70
self.pin_memory = is_pin_memory_available()
71
72
+ # Multi-modal data support
73
+ self.multi_modal_input_mapper = MULTIMODAL_REGISTRY \
74
+ .create_input_mapper(self.model_config)
75
+
76
# Lazy initialization.
77
self.model: nn.Module # initialize after load_model.
78
@@ -76,13 +84,15 @@ class NeuronModelRunner(ModelRunnerBase[ModelInputForNeuron]):
84
def _prepare_prompt(
85
self,
86
seq_group_metadata_list: List[SequenceGroupMetadata],
79
- ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor, List[int]]:
87
+ ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor, List[int], Mapping[
88
+ str, BatchedTensors]]:
89
assert len(seq_group_metadata_list) > 0
90
input_tokens: List[List[int]] = []
91
input_positions: List[List[int]] = []
92
input_block_ids: List[int] = []
93
94
seq_lens: List[int] = []
95
+ multi_modal_inputs_list: List[MultiModalInputs] = []
96
for seq_group_metadata in seq_group_metadata_list:
97
assert seq_group_metadata.is_prompt
98
seq_ids = list(seq_group_metadata.seq_data.keys())
@@ -102,6 +112,12 @@ class NeuronModelRunner(ModelRunnerBase[ModelInputForNeuron]):
112
assert len(block_table) == 1
113
input_block_ids.append(block_table[0])
114
115
+ mm_data = seq_group_metadata.multi_modal_data
116
+ if mm_data:
117
+ # Process multi-modal data
118
+ mm_kwargs = self.multi_modal_input_mapper(mm_data)
119
+ multi_modal_inputs_list.append(mm_kwargs)
120
+
121
max_seq_len = max(seq_lens)
122
assert max_seq_len > 0
123
input_tokens = make_tensor_with_pad(input_tokens,
@@ -118,7 +134,11 @@ class NeuronModelRunner(ModelRunnerBase[ModelInputForNeuron]):
134
dtype=torch.long,
135
device=self.device)
136
121
- return input_tokens, input_positions, input_block_ids, seq_lens
137
+ multi_modal_kwargs = MultiModalInputs.batch(multi_modal_inputs_list,
138
+ device=self.device)
139
+
140
+ return (input_tokens, input_positions, input_block_ids, seq_lens,
141
+ multi_modal_kwargs)
142
143
def _prepare_decode(
144
self,
@@ -184,8 +204,9 @@ class NeuronModelRunner(ModelRunnerBase[ModelInputForNeuron]):
204
is_prompt = seq_group_metadata_list[0].is_prompt
205
# Prepare input tensors.
206
if is_prompt:
187
- (input_tokens, input_positions, input_block_ids,
188
- seq_lens) = self._prepare_prompt(seq_group_metadata_list)
207
+ (input_tokens, input_positions, input_block_ids, seq_lens,
208
+ multi_modal_kwargs
209
+ ) = self._prepare_prompt(seq_group_metadata_list)
210
else:
211
(input_tokens, input_positions,
212
input_block_ids) = self._prepare_decode(seq_group_metadata_list)
@@ -203,7 +224,8 @@ class NeuronModelRunner(ModelRunnerBase[ModelInputForNeuron]):
224
return ModelInputForNeuron(input_tokens=input_tokens,
225
input_positions=input_positions,
226
input_block_ids=input_block_ids,
206
- sampling_metadata=sampling_metadata)
227
+ sampling_metadata=sampling_metadata,
228
+ multi_modal_kwargs=multi_modal_kwargs)
229
230
@torch.inference_mode()
231
def execute_model(
@@ -221,6 +243,7 @@ class NeuronModelRunner(ModelRunnerBase[ModelInputForNeuron]):
243
input_ids=model_input.input_tokens,
244
positions=model_input.input_positions,
245
input_block_ids=model_input.input_block_ids,
246
+ **(model_input.multi_modal_kwargs or {}),
247
)
248
249
# Compute the logits.
vllm/worker/openvino_model_runner.py
+26
-12
@@ -1,4 +1,4 @@
1
-from typing import List, NamedTuple, Optional, Tuple
1
+from typing import List, Mapping, NamedTuple, Optional, Tuple
2
3
import openvino as ov
4
import torch
@@ -12,6 +12,8 @@ from vllm.config import (CacheConfig, DeviceConfig, LoadConfig, LoRAConfig,
12
from vllm.logger import init_logger
13
from vllm.model_executor import SamplingMetadata
14
from vllm.model_executor.model_loader.openvino import get_model
15
+from vllm.multimodal import (MULTIMODAL_REGISTRY, BatchedTensors,
16
+ MultiModalInputs)
17
from vllm.sequence import SamplerOutput, SequenceGroupMetadata
18
19
logger = init_logger(__name__)
@@ -23,7 +25,7 @@ class ModelInput(NamedTuple):
25
attn_metadata: Optional[OpenVINOAttentionMetadata]
26
seq_lens: List[int]
27
query_lens: List[int]
26
- multi_modal_input: Optional[torch.Tensor]
28
+ multi_modal_kwargs: Mapping[str, BatchedTensors]
29
30
@classmethod
31
def empty(cls, device):
@@ -32,7 +34,7 @@ class ModelInput(NamedTuple):
34
attn_metadata=None,
35
seq_lens=[],
36
query_lens=[],
35
- multi_modal_input=None)
37
+ multi_modal_kwargs={})
38
39
40
class OpenVINOModelRunner:
@@ -78,6 +80,10 @@ class OpenVINOModelRunner:
80
self.block_size,
81
)
82
83
+ # Multi-modal data support
84
+ self.multi_modal_input_mapper = MULTIMODAL_REGISTRY \
85
+ .create_input_mapper(self.model_config)
86
+
87
# Lazy initialization.
88
self.model: nn.Module # Set after init_Model
89
@@ -108,6 +114,8 @@ class OpenVINOModelRunner:
114
seq_lens: List[int] = []
115
past_lens: List[int] = []
116
query_lens: List[int] = []
117
+ multi_modal_inputs_list: List[MultiModalInputs] = []
118
+
119
subsequence_begins: List[int] = []
120
block_indices: List[int] = []
121
block_indices_begins: List[int] = []
@@ -160,6 +168,11 @@ class OpenVINOModelRunner:
168
and self.sliding_window is None
169
and is_prompt)
170
171
+ mm_data = seq_group_metadata.multi_modal_data
172
+ if mm_data:
173
+ mm_kwargs = self.multi_modal_input_mapper(mm_data)
174
+ multi_modal_inputs_list.append(mm_kwargs)
175
+
176
block_table = seq_group_metadata.block_tables[seq_id]
177
# TODO(sang): Combine chunked prefill and prefix caching by
178
# only allowing multiple of block_size chunk size.
@@ -251,22 +264,24 @@ class OpenVINOModelRunner:
264
block_indices_begins=block_indices_begins_tensor,
265
max_context_len=max_context_len_tensor,
266
)
267
+
268
+ multi_modal_kwargs = MultiModalInputs.batch(multi_modal_inputs_list,
269
+ device=self.device)
270
+
271
return ModelInput(
272
input_tokens,
273
input_positions,
274
attn_metadata,
275
seq_lens,
276
query_lens,
260
- None,
277
+ multi_modal_kwargs=multi_modal_kwargs,
278
)
279
280
def prepare_input_tensors(
281
self,
282
seq_group_metadata_list: List[SequenceGroupMetadata],
283
) -> Tuple[torch.Tensor, torch.Tensor, OpenVINOAttentionMetadata,
267
- SamplingMetadata, Optional[torch.Tensor], ]:
268
- multi_modal_input = None
269
-
284
+ SamplingMetadata, Mapping[str, BatchedTensors]]:
285
# Prepare input tensors.
286
(
287
input_tokens,
@@ -274,7 +289,7 @@ class OpenVINOModelRunner:
289
attn_metadata,
290
seq_lens,
291
query_lens,
277
- multi_modal_input,
292
+ multi_modal_kwargs,
293
) = self._prepare_model_input(seq_group_metadata_list)
294
295
sampling_metadata = SamplingMetadata.prepare(
@@ -290,7 +305,7 @@ class OpenVINOModelRunner:
305
input_positions,
306
attn_metadata,
307
sampling_metadata,
293
- multi_modal_input,
308
+ multi_modal_kwargs,
309
)
310
311
@torch.inference_mode()
@@ -304,7 +319,7 @@ class OpenVINOModelRunner:
319
input_positions,
320
attn_metadata,
321
sampling_metadata,
307
- multi_modal_input,
322
+ multi_modal_kwargs,
323
) = self.prepare_input_tensors(seq_group_metadata_list)
324
325
model_executable = self.model
@@ -313,9 +328,8 @@ class OpenVINOModelRunner:
328
"positions": input_positions,
329
"kv_caches": kv_caches,
330
"attn_metadata": attn_metadata,
331
+ **(multi_modal_kwargs or {}),
332
}
317
- if self.vision_language_config:
318
- execute_model_kwargs.update({"image_input": multi_modal_input})
333
334
hidden_states = model_executable(**execute_model_kwargs)
335
vllm/worker/tpu_model_runner.py
+39
-5
@@ -1,5 +1,5 @@
1
import time
2
-from typing import List, Optional, Tuple
2
+from typing import List, Mapping, Optional, Tuple
3
4
import numpy as np
5
import torch
@@ -12,6 +12,8 @@ from vllm.config import (CacheConfig, DeviceConfig, LoadConfig, ModelConfig,
12
from vllm.logger import init_logger
13
from vllm.model_executor.model_loader import get_model
14
from vllm.model_executor.sampling_metadata import SamplingMetadata
15
+from vllm.multimodal import (MULTIMODAL_REGISTRY, BatchedTensors,
16
+ MultiModalInputs)
17
from vllm.sequence import (CompletionSequenceGroupOutput, Logprob,
18
SamplerOutput, SequenceGroupMetadata,
19
SequenceOutput)
@@ -66,6 +68,10 @@ class TPUModelRunner:
68
False,
69
)
70
71
+ # Multi-modal data support
72
+ self.multi_modal_input_mapper = MULTIMODAL_REGISTRY \
73
+ .create_input_mapper(self.model_config)
74
+
75
def load_model(self) -> None:
76
self.device = self.device_config.device
77
@@ -193,12 +199,14 @@ class TPUModelRunner:
199
def _prepare_prompt(
200
self,
201
seq_group_metadata_list: List[SequenceGroupMetadata],
196
- ):
202
+ ) -> Tuple[torch.Tensor, torch.Tensor, AttentionMetadata, torch.Tensor,
203
+ Mapping[str, BatchedTensors]]:
204
assert len(seq_group_metadata_list) > 0
205
input_tokens: List[List[int]] = []
206
input_positions: List[List[int]] = []
207
prompt_lens: List[int] = []
208
slot_mapping: List[List[int]] = []
209
+ multi_modal_inputs_list: List[MultiModalInputs] = []
210
211
for seq_group_metadata in seq_group_metadata_list:
212
assert seq_group_metadata.is_prompt
@@ -224,6 +232,11 @@ class TPUModelRunner:
232
slot = block_number * self.block_size + block_offset
233
slot_mapping[-1].append(slot)
234
235
+ mm_data = seq_group_metadata.multi_modal_data
236
+ if mm_data:
237
+ mm_kwargs = self.multi_modal_input_mapper(mm_data)
238
+ multi_modal_inputs_list.append(mm_kwargs)
239
+
240
assert len(prompt_lens) > 0
241
num_prefills = len(prompt_lens)
242
num_prefill_tokens = sum(prompt_lens)
@@ -261,17 +274,24 @@ class TPUModelRunner:
274
block_tables=None,
275
context_lens=None,
276
)
264
- return input_tokens, input_positions, attn_metadata, prompt_lens
277
+
278
+ multi_modal_kwargs = MultiModalInputs.batch(multi_modal_inputs_list,
279
+ device=self.device)
280
+
281
+ return (input_tokens, input_positions, attn_metadata, prompt_lens,
282
+ multi_modal_kwargs)
283
284
def _prepare_decode(
285
self,
286
seq_group_metadata_list: List[SequenceGroupMetadata],
269
- ):
287
+ ) -> Tuple[torch.Tensor, torch.Tensor, AttentionMetadata, torch.Tensor,
288
+ Mapping[str, BatchedTensors]]:
289
assert len(seq_group_metadata_list) > 0
290
input_tokens: List[List[int]] = []
291
input_positions: List[List[int]] = []
292
slot_mapping: List[List[int]] = []
293
context_lens: List[int] = []
294
+ multi_modal_inputs_list: List[MultiModalInputs] = []
295
296
batch_idx = 0
297
for seq_group_metadata in seq_group_metadata_list:
@@ -297,6 +317,11 @@ class TPUModelRunner:
317
slot = block_number * self.block_size + block_offset
318
slot_mapping.append([slot])
319
320
+ mm_data = seq_group_metadata.multi_modal_data
321
+ if mm_data:
322
+ mm_kwargs = self.multi_modal_input_mapper(mm_data)
323
+ multi_modal_inputs_list.append(mm_kwargs)
324
+
325
batch_size = _get_padded_batch_size(batch_idx)
326
num_paddings = batch_size - batch_idx
327
input_tokens = input_tokens + [[0]] * num_paddings
@@ -330,7 +355,12 @@ class TPUModelRunner:
355
block_tables=block_tables,
356
context_lens=context_lens,
357
)
333
- return input_tokens, input_positions, attn_metadata, input_lens
358
+
359
+ multi_modal_kwargs = MultiModalInputs.batch(multi_modal_inputs_list,
360
+ device=self.device)
361
+
362
+ return (input_tokens, input_positions, attn_metadata, input_lens,
363
+ multi_modal_kwargs)
364
365
def _prepare_sample(
366
self,
@@ -483,6 +513,7 @@ class ModelWrapper(nn.Module):
513
kv_caches: List[Tuple[Optional[torch.Tensor], Optional[torch.Tensor]]],
514
attn_metadata: AttentionMetadata,
515
input_lens: torch.Tensor,
516
+ multi_modal_kwargs: Optional[Mapping[str, BatchedTensors]],
517
t: torch.Tensor,
518
p: torch.Tensor,
519
num_samples: int,
@@ -496,6 +527,8 @@ class ModelWrapper(nn.Module):
527
memory profiling at initialization.
528
attn_metadata: The Pallas attention metadata.
529
input_lens: The actual input lengths of shape [batch_size].
530
+ multi_modal_kwargs: Keyword arguments from multi-modal data to
531
+ pass to the model.
532
t: The sampling temperature of shape [batch_size].
533
p: The top-p probability of shape [batch_size].
534
"""
@@ -540,6 +573,7 @@ class ModelWrapper(nn.Module):
573
position_ids,
574
kv_caches,
575
attn_metadata,
576
+ **(multi_modal_kwargs or {}),
577
)
578
hidden_states = hidden_states.flatten(0, 1)
579
logits = self.model.compute_logits(hidden_states, sampling_metadata)
vllm/worker/xpu_model_runner.py
+44
-26
@@ -1,5 +1,6 @@
1
from dataclasses import dataclass
2
-from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Type, Union
2
+from typing import (TYPE_CHECKING, Any, Dict, List, Mapping, Optional, Tuple,
3
+ Type, Union)
4
5
import torch
6
import torch.nn as nn
@@ -9,10 +10,13 @@ from vllm.config import (CacheConfig, DeviceConfig, LoadConfig, LoRAConfig,
10
ModelConfig, ParallelConfig, SchedulerConfig,
11
VisionLanguageConfig)
12
from vllm.distributed import broadcast_tensor_dict
13
+from vllm.inputs import INPUT_REGISTRY
14
from vllm.logger import init_logger
15
from vllm.model_executor.model_loader import get_model
16
+from vllm.multimodal import (MULTIMODAL_REGISTRY, BatchedTensors,
17
+ MultiModalInputs)
18
from vllm.sampling_params import SamplingParams
15
-from vllm.sequence import (IntermediateTensors, SamplerOutput, SequenceData,
19
+from vllm.sequence import (IntermediateTensors, SamplerOutput,
20
SequenceGroupMetadata)
21
from vllm.utils import CudaMemoryProfiler, make_tensor_with_pad
22
from vllm.worker.model_runner import AttentionMetadata, SamplingMetadata
@@ -44,7 +48,7 @@ class ModelInputForXPU(ModelRunnerInputBase):
48
input_positions: Optional[torch.Tensor] = None
49
attn_metadata: Optional["AttentionMetadata"] = None
50
sampling_metadata: Optional["SamplingMetadata"] = None
47
- multi_modal_input: Optional[Dict[str, torch.Tensor]] = None
51
+ multi_modal_kwargs: Optional[Mapping[str, BatchedTensors]] = None
52
53
def as_broadcastable_tensor_dict(
54
self) -> Dict[str, Union[int, torch.Tensor]]:
@@ -116,6 +120,10 @@ class XPUModelRunner(ModelRunnerBase[ModelInputForXPU]):
120
self.block_size,
121
)
122
123
+ # Multi-modal data support
124
+ self.multi_modal_input_mapper = MULTIMODAL_REGISTRY \
125
+ .create_input_mapper(self.model_config)
126
+
127
# Lazy initialization.
128
self.model: nn.Module # Set after init_Model
129
@@ -156,12 +164,26 @@ class XPUModelRunner(ModelRunnerBase[ModelInputForXPU]):
164
# To exercise the worst scenario for GPU memory consumption,
165
# the number of seqs (batch_size) is chosen to maximize the number
166
# of images processed.
167
+ model_config = self.model_config
168
+ vlm_config = self.vision_language_config
169
+
170
+ if vlm_config:
171
+ max_num_seqs = min(
172
+ max_num_seqs,
173
+ int(max_num_batched_tokens / vlm_config.image_feature_size))
174
+
175
for group_id in range(max_num_seqs):
176
seq_len = (max_num_batched_tokens // max_num_seqs +
177
(group_id < max_num_batched_tokens % max_num_seqs))
178
163
- seq_data = SequenceData([0] * seq_len)
164
- dummy_multi_modal_data = None
179
+ seq_data, dummy_multi_modal_data = INPUT_REGISTRY \
180
+ .dummy_data_for_profiling(model_config, seq_len)
181
+
182
+ # Having more tokens is over-conservative but otherwise fine
183
+ assert len(seq_data.prompt_token_ids) >= seq_len, (
184
+ f"Expected at least {seq_len} dummy tokens for profiling, "
185
+ f"but got: {len(seq_data.prompt_token_ids)}")
186
+
187
seq = SequenceGroupMetadata(
188
request_id=str(group_id),
189
is_prompt=True,
@@ -194,7 +216,7 @@ class XPUModelRunner(ModelRunnerBase[ModelInputForXPU]):
216
virtual_engine: int = 0,
217
finished_requests_ids: Optional[List[str]] = None
218
) -> ModelInputForXPU:
197
- multi_modal_input = None
219
+ multi_modal_kwargs = None
220
if self.is_driver_worker:
221
# NOTE: We assume that all sequences in the group are all prompts or
222
# all decodes.
@@ -202,7 +224,7 @@ class XPUModelRunner(ModelRunnerBase[ModelInputForXPU]):
224
# Prepare input tensors.
225
if is_prompt:
226
(input_tokens, input_positions, attn_metadata, seq_lens,
205
- multi_modal_input
227
+ multi_modal_kwargs
228
) = self._prepare_prompt(seq_group_metadata_list)
229
else:
230
(input_tokens, input_positions,
@@ -223,6 +245,7 @@ class XPUModelRunner(ModelRunnerBase[ModelInputForXPU]):
245
"input_positions": input_positions,
246
"selected_token_indices":
247
sampling_metadata.selected_token_indices,
248
+ "multi_modal_kwargs": multi_modal_kwargs,
249
}
250
metadata_dict.update(attn_metadata.asdict_zerocopy())
251
broadcast_tensor_dict(metadata_dict, src=0)
@@ -232,6 +255,7 @@ class XPUModelRunner(ModelRunnerBase[ModelInputForXPU]):
255
input_positions = metadata_dict.pop("input_positions")
256
selected_token_indices = metadata_dict.pop(
257
"selected_token_indices")
258
+ multi_modal_kwargs = metadata_dict.pop("multi_modal_kwargs")
259
attn_metadata = self.attn_backend.make_metadata(**metadata_dict)
260
sampling_metadata = SamplingMetadata(
261
seq_groups=None,
@@ -244,7 +268,7 @@ class XPUModelRunner(ModelRunnerBase[ModelInputForXPU]):
268
input_positions=input_positions,
269
attn_metadata=attn_metadata,
270
sampling_metadata=sampling_metadata,
247
- multi_modal_input=multi_modal_input)
271
+ multi_modal_kwargs=multi_modal_kwargs)
272
273
def _prepare_decode(
274
self,
@@ -350,10 +374,8 @@ class XPUModelRunner(ModelRunnerBase[ModelInputForXPU]):
374
"positions": model_input.input_positions,
375
"kv_caches": kv_caches,
376
"attn_metadata": model_input.attn_metadata,
377
+ **(model_input.multi_modal_kwargs or {}),
378
}
354
- if self.vision_language_config:
355
- execute_model_kwargs.update(
356
- {"image_input": model_input.multi_modal_input})
379
380
hidden_states = model_executable(**execute_model_kwargs)
381
@@ -376,13 +398,13 @@ class XPUModelRunner(ModelRunnerBase[ModelInputForXPU]):
398
self,
399
seq_group_metadata_list: List[SequenceGroupMetadata],
400
) -> Tuple[torch.Tensor, torch.Tensor, AttentionMetadata, List[int],
379
- Optional[torch.Tensor]]:
401
+ Mapping[str, BatchedTensors]]:
402
assert len(seq_group_metadata_list) > 0
403
input_tokens: List[int] = []
404
input_positions: List[int] = []
405
slot_mapping: List[int] = []
406
seq_lens: List[int] = []
385
- multi_modal_input_list: List[torch.Tensor] = []
407
+ multi_modal_inputs_list: List[MultiModalInputs] = []
408
409
for seq_group_metadata in seq_group_metadata_list:
410
assert seq_group_metadata.is_prompt
@@ -403,9 +425,10 @@ class XPUModelRunner(ModelRunnerBase[ModelInputForXPU]):
425
# is always the first token in the sequence.
426
input_positions.extend(list(range(computed_len, seq_len)))
427
406
- if seq_group_metadata.multi_modal_data:
407
- multi_modal_input_list.append(
408
- seq_group_metadata.multi_modal_data.data)
428
+ mm_data = seq_group_metadata.multi_modal_data
429
+ if mm_data:
430
+ mm_kwargs = self.multi_modal_input_mapper(mm_data)
431
+ multi_modal_inputs_list.append(mm_kwargs)
432
433
if seq_group_metadata.block_tables is None:
434
# During memory profiling, the block tables are not initialized
@@ -435,15 +458,6 @@ class XPUModelRunner(ModelRunnerBase[ModelInputForXPU]):
458
slot = block_number * self.block_size + block_offset
459
slot_mapping.append(slot)
460
438
- if multi_modal_input_list:
439
- assert self.vision_language_config, (
440
- "Multi-modal inputs are only supported by "
441
- "vision language models.")
442
- multi_modal_input = torch.cat(multi_modal_input_list,
443
- dim=0).to(self.device)
444
- else:
445
- multi_modal_input = None
446
-
461
num_prompt_tokens = len(input_tokens)
462
463
input_tokens = torch.tensor(input_tokens,
@@ -475,5 +489,9 @@ class XPUModelRunner(ModelRunnerBase[ModelInputForXPU]):
489
num_decode_tokens=0,
490
block_tables=torch.tensor([], device=self.device, dtype=torch.int),
491
)
492
+
493
+ multi_modal_kwargs = MultiModalInputs.batch(multi_modal_inputs_list,
494
+ device=self.device)
495
+
496
return (input_tokens, input_positions, attn_metadata, seq_lens,
479
- multi_modal_input)
497
+ multi_modal_kwargs)