A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 17:00 UTC
[Doc] Indicate more information about supported modalities (#8181)
Cyrus Leung committed
Sep 5, 2024 at 18:51 UTC
288a938872cc3c6150a486aaa15a3b5dcadf42cc
7 files changed
+206
-51
.buildkite/test-pipeline.yaml
+1
@@ -158,6 +158,7 @@ steps:
158
- python3 offline_inference_with_prefix.py
159
- python3 llm_engine_example.py
160
- python3 offline_inference_vision_language.py
161
+ - python3 offline_inference_vision_language_multi_image.py
162
- python3 tensorize_vllm_model.py --model facebook/opt-125m serialize --serialized-directory /tmp/ --suffix v1 && python3 tensorize_vllm_model.py --model facebook/opt-125m deserialize --path-to-tensors /tmp/vllm/facebook/opt-125m/v1/model.tensors
163
- python3 offline_inference_encoder_decoder.py
164
docs/source/getting_started/debugging.rst
+1
-1
@@ -21,7 +21,7 @@ If you have already taken care of the above issues, but the vLLM instance still
21
22
With more logging, hopefully you can find the root cause of the issue.
23
24
-If it crashes, and the error trace shows somewhere around ``self.graph.replay()`` in ``vllm/worker/model_runner.py``, it is a cuda error inside cudagraph. To know the particular cuda operation that causes the error, you can add ``--enforce-eager`` to the command line, or ``enforce_eager=True`` to the ``LLM`` class, to disable the cudagraph optimization. This way, you can locate the exact cuda operation that causes the error.
24
+If it crashes, and the error trace shows somewhere around ``self.graph.replay()`` in ``vllm/worker/model_runner.py``, it is a cuda error inside cudagraph. To know the particular cuda operation that causes the error, you can add ``--enforce-eager`` to the command line, or ``enforce_eager=True`` to the :class:`~vllm.LLM` class, to disable the cudagraph optimization. This way, you can locate the exact cuda operation that causes the error.
25
26
Here are some common issues that can cause hangs:
27
docs/source/getting_started/quickstart.rst
+4
-2
@@ -24,7 +24,9 @@ Offline Batched Inference
24
25
We first show an example of using vLLM for offline batched inference on a dataset. In other words, we use vLLM to generate texts for a list of input prompts.
26
27
-Import ``LLM`` and ``SamplingParams`` from vLLM. The ``LLM`` class is the main class for running offline inference with vLLM engine. The ``SamplingParams`` class specifies the parameters for the sampling process.
27
+Import :class:`~vllm.LLM` and :class:`~vllm.SamplingParams` from vLLM.
28
+The :class:`~vllm.LLM` class is the main class for running offline inference with vLLM engine.
29
+The :class:`~vllm.SamplingParams` class specifies the parameters for the sampling process.
30
31
.. code-block:: python
32
@@ -42,7 +44,7 @@ Define the list of input prompts and the sampling parameters for generation. The
44
]
45
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
46
45
-Initialize vLLM's engine for offline inference with the ``LLM`` class and the `OPT-125M model <https://arxiv.org/abs/2205.01068>`_. The list of supported models can be found at :ref:`supported models <supported_models>`.
47
+Initialize vLLM's engine for offline inference with the :class:`~vllm.LLM` class and the `OPT-125M model <https://arxiv.org/abs/2205.01068>`_. The list of supported models can be found at :ref:`supported models <supported_models>`.
48
49
.. code-block:: python
50
docs/source/models/supported_models.rst
+12
-9
@@ -194,12 +194,12 @@ Multimodal Language Models
194
195
* - Architecture
196
- Models
197
- - Supported Modalities
197
+ - Modalities
198
- Example HuggingFace Models
199
- :ref:`LoRA <lora>`
200
* - :code:`Blip2ForConditionalGeneration`
201
- BLIP-2
202
- - Image
202
+ - Image\ :sup:`E`
203
- :code:`Salesforce/blip2-opt-2.7b`, :code:`Salesforce/blip2-opt-6.7b`, etc.
204
-
205
* - :code:`ChameleonForConditionalGeneration`
@@ -214,40 +214,43 @@ Multimodal Language Models
214
-
215
* - :code:`InternVLChatModel`
216
- InternVL2
217
- - Image
217
+ - Image\ :sup:`E`
218
- :code:`OpenGVLab/InternVL2-4B`, :code:`OpenGVLab/InternVL2-8B`, etc.
219
-
220
* - :code:`LlavaForConditionalGeneration`
221
- LLaVA-1.5
222
- - Image
222
+ - Image\ :sup:`E`
223
- :code:`llava-hf/llava-1.5-7b-hf`, :code:`llava-hf/llava-1.5-13b-hf`, etc.
224
-
225
* - :code:`LlavaNextForConditionalGeneration`
226
- LLaVA-NeXT
227
- - Image
227
+ - Image\ :sup:`E+`
228
- :code:`llava-hf/llava-v1.6-mistral-7b-hf`, :code:`llava-hf/llava-v1.6-vicuna-7b-hf`, etc.
229
-
230
* - :code:`PaliGemmaForConditionalGeneration`
231
- PaliGemma
232
- - Image
232
+ - Image\ :sup:`E`
233
- :code:`google/paligemma-3b-pt-224`, :code:`google/paligemma-3b-mix-224`, etc.
234
-
235
* - :code:`Phi3VForCausalLM`
236
- Phi-3-Vision, Phi-3.5-Vision
237
- - Image
237
+ - Image\ :sup:`E+`
238
- :code:`microsoft/Phi-3-vision-128k-instruct`, :code:`microsoft/Phi-3.5-vision-instruct` etc.
239
-
240
* - :code:`MiniCPMV`
241
- MiniCPM-V
242
- - Image
242
+ - Image\ :sup:`+`
243
- :code:`openbmb/MiniCPM-V-2` (see note), :code:`openbmb/MiniCPM-Llama3-V-2_5`, :code:`openbmb/MiniCPM-V-2_6`, etc.
244
-
245
* - :code:`UltravoxModel`
246
- Ultravox
247
- - Audio
247
+ - Audio\ :sup:`E+`
248
- :code:`fixie-ai/ultravox-v0_3`
249
-
250
251
+| :sup:`E` Pre-computed embeddings can be inputted for this modality.
252
+| :sup:`+` Multiple items can be inputted per text prompt for this modality.
253
+
254
.. note::
255
For :code:`openbmb/MiniCPM-V-2`, the official repo doesn't work yet, so we need to use a fork (:code:`HwwwH/MiniCPM-V-2`) for now.
256
For more details, please see: https://github.com/vllm-project/vllm/pull/4087#issuecomment-2250397630
docs/source/models/vlm.rst
+88
-35
@@ -9,26 +9,23 @@ This document shows you how to run and serve these models using vLLM.
9
.. important::
10
We are actively iterating on VLM support. Expect breaking changes to VLM usage and development in upcoming releases without prior deprecation.
11
12
- Currently, the support for vision language models on vLLM has the following limitations:
13
-
14
- * Only single image input is supported per text prompt.
15
-
12
We are continuously improving user & developer experience for VLMs. Please `open an issue on GitHub <https://github.com/vllm-project/vllm/issues/new/choose>`_ if you have any feedback or feature requests.
13
18
-Offline Batched Inference
19
--------------------------
14
+Offline Inference
15
+-----------------
16
+
17
+Single-image input
18
+^^^^^^^^^^^^^^^^^^
19
21
-To initialize a VLM, the aforementioned arguments must be passed to the ``LLM`` class for instantiating the engine.
20
+The :class:`~vllm.LLM` class can be instantiated in much the same way as language-only models.
21
22
.. code-block:: python
23
24
llm = LLM(model="llava-hf/llava-1.5-7b-hf")
25
27
-.. important::
26
+.. note::
27
We have removed all vision language related CLI args in the ``0.5.1`` release. **This is a breaking change**, so please update your code to follow
29
- the above snippet. Specifically, ``image_feature_size`` is no longer required to be specified as we now calculate that
30
- internally for each model.
31
-
28
+ the above snippet. Specifically, ``image_feature_size`` can no longer be specified as we now calculate that internally for each model.
29
30
To pass an image to the model, note the following in :class:`vllm.inputs.PromptInputs`:
31
@@ -86,61 +83,117 @@ To pass an image to the model, note the following in :class:`vllm.inputs.PromptI
83
84
A code example can be found in `examples/offline_inference_vision_language.py <https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_vision_language.py>`_.
85
86
+Multi-image input
87
+^^^^^^^^^^^^^^^^^
88
90
-Online OpenAI Vision API Compatible Inference
91
-----------------------------------------------
89
+Multi-image input is only supported for a subset of VLMs, as shown :ref:`here <supported_vlms>`.
90
93
-You can serve vision language models with vLLM's HTTP server that is compatible with `OpenAI Vision API <https://platform.openai.com/docs/guides/vision>`_.
91
+To enable multiple multi-modal items per text prompt, you have to set ``limit_mm_per_prompt`` for the :class:`~vllm.LLM` class.
92
95
-.. note::
96
- Currently, vLLM supports only **single** ``image_url`` input per ``messages``. Support for multi-image inputs will be
97
- added in the future.
93
+.. code-block:: python
94
99
-Below is an example on how to launch the same ``llava-hf/llava-1.5-7b-hf`` with vLLM API server.
95
+ llm = LLM(
96
+ model="microsoft/Phi-3.5-vision-instruct",
97
+ trust_remote_code=True, # Required to load Phi-3.5-vision
98
+ max_model_len=4096, # Otherwise, it may not fit in smaller GPUs
99
+ limit_mm_per_prompt={"image": 2}, # The maximum number to accept
100
+ )
101
101
-.. important::
102
- Since OpenAI Vision API is based on `Chat <https://platform.openai.com/docs/api-reference/chat>`_ API, a chat template
103
- is **required** to launch the API server if the model's tokenizer does not come with one. In this example, we use the
104
- HuggingFace Llava chat template that you can find in the example folder `here <https://github.com/vllm-project/vllm/blob/main/examples/template_llava.jinja>`_.
102
+Instead of passing in a single image, you can pass in a list of images.
103
+
104
+.. code-block:: python
105
+
106
+ # Refer to the HuggingFace repo for the correct format to use
107
+ prompt = "<|user|>\n<image_1>\n<image_2>\nWhat is the content of each image?<|end|>\n<|assistant|>\n"
108
+
109
+ # Load the images using PIL.Image
110
+ image1 = PIL.Image.open(...)
111
+ image2 = PIL.Image.open(...)
112
+
113
+ outputs = llm.generate({
114
+ "prompt": prompt,
115
+ "multi_modal_data": {
116
+ "image": [image1, image2]
117
+ },
118
+ })
119
+
120
+ for o in outputs:
121
+ generated_text = o.outputs[0].text
122
+ print(generated_text)
123
+
124
+A code example can be found in `examples/offline_inference_vision_language_multi_image.py <https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_vision_language_multi_image.py>`_.
125
+
126
+Online Inference
127
+----------------
128
+
129
+OpenAI Vision API
130
+^^^^^^^^^^^^^^^^^
131
+
132
+You can serve vision language models with vLLM's HTTP server that is compatible with `OpenAI Vision API <https://platform.openai.com/docs/guides/vision>`_.
133
+
134
+Below is an example on how to launch the same ``microsoft/Phi-3.5-vision-instruct`` with vLLM's OpenAI-compatible API server.
135
136
.. code-block:: bash
137
108
- vllm serve llava-hf/llava-1.5-7b-hf --chat-template template_llava.jinja
138
+ vllm serve microsoft/Phi-3.5-vision-instruct --max-model-len 4096 \
139
+ --trust-remote-code --limit-mm-per-prompt image=2
140
141
.. important::
111
- We have removed all vision language related CLI args in the ``0.5.1`` release. **This is a breaking change**, so please update your code to follow
112
- the above snippet. Specifically, ``image_feature_size`` is no longer required to be specified as we now calculate that
113
- internally for each model.
142
+ Since OpenAI Vision API is based on `Chat Completions <https://platform.openai.com/docs/api-reference/chat>`_ API,
143
+ a chat template is **required** to launch the API server.
144
+
145
+ Although Phi-3.5-Vision comes with a chat template, for other models you may have to provide one if the model's tokenizer does not come with it.
146
+ The chat template can be inferred based on the documentation on the model's HuggingFace repo.
147
+ For example, LLaVA-1.5 (``llava-hf/llava-1.5-7b-hf``) requires a chat template that can be found `here <https://github.com/vllm-project/vllm/blob/main/examples/template_llava.jinja>`_.
148
149
To consume the server, you can use the OpenAI client like in the example below:
150
151
.. code-block:: python
152
153
from openai import OpenAI
154
+
155
openai_api_key = "EMPTY"
156
openai_api_base = "http://localhost:8000/v1"
157
+
158
client = OpenAI(
159
api_key=openai_api_key,
160
base_url=openai_api_base,
161
)
162
+
163
+ # Single-image input inference
164
+ image_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
165
+
166
chat_response = client.chat.completions.create(
127
- model="llava-hf/llava-1.5-7b-hf",
167
+ model="microsoft/Phi-3.5-vision-instruct",
168
messages=[{
169
"role": "user",
170
"content": [
171
# NOTE: The prompt formatting with the image token `<image>` is not needed
172
# since the prompt will be processed automatically by the API server.
133
- {"type": "text", "text": "What's in this image?"},
134
- {
135
- "type": "image_url",
136
- "image_url": {
137
- "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg",
138
- },
139
- },
173
+ {"type": "text", "text": "What’s in this image?"},
174
+ {"type": "image_url", "image_url": {"url": image_url}},
175
],
176
}],
177
)
143
- print("Chat response:", chat_response)
178
+ print("Chat completion output:", chat_response.choices[0].message.content)
179
+
180
+ # Multi-image input inference
181
+ image_url_duck = "https://upload.wikimedia.org/wikipedia/commons/d/da/2015_Kaczka_krzy%C5%BCowka_w_wodzie_%28samiec%29.jpg"
182
+ image_url_lion = "https://upload.wikimedia.org/wikipedia/commons/7/77/002_The_lion_king_Snyggve_in_the_Serengeti_National_Park_Photo_by_Giles_Laurent.jpg"
183
+
184
+ chat_response = client.chat.completions.create(
185
+ model="microsoft/Phi-3.5-vision-instruct",
186
+ messages=[{
187
+ "role": "user",
188
+ "content": [
189
+ {"type": "text", "text": "What are the animals in these images?"},
190
+ {"type": "image_url", "image_url": {"url": image_url_duck}},
191
+ {"type": "image_url", "image_url": {"url": image_url_lion}},
192
+ ],
193
+ }],
194
+ )
195
+ print("Chat completion output:", chat_response.choices[0].message.content)
196
+
197
198
A full code example can be found in `examples/openai_vision_api_client.py <https://github.com/vllm-project/vllm/blob/main/examples/openai_vision_api_client.py>`_.
199
examples/offline_inference_vision_language_multi_image.py
new
+95
@@ -0,0 +1,95 @@
1
+"""
2
+This example shows how to use vLLM for running offline inference with
3
+multi-image input on vision language models, using the chat template defined
4
+by the model.
5
+"""
6
+from argparse import Namespace
7
+from typing import List
8
+
9
+from vllm import LLM
10
+from vllm.multimodal.utils import fetch_image
11
+from vllm.utils import FlexibleArgumentParser
12
+
13
+QUESTION = "What is the content of each image?"
14
+IMAGE_URLS = [
15
+ "https://upload.wikimedia.org/wikipedia/commons/d/da/2015_Kaczka_krzy%C5%BCowka_w_wodzie_%28samiec%29.jpg",
16
+ "https://upload.wikimedia.org/wikipedia/commons/7/77/002_The_lion_king_Snyggve_in_the_Serengeti_National_Park_Photo_by_Giles_Laurent.jpg",
17
+]
18
+
19
+
20
+def _load_phi3v(image_urls: List[str]):
21
+ return LLM(
22
+ model="microsoft/Phi-3.5-vision-instruct",
23
+ trust_remote_code=True,
24
+ max_model_len=4096,
25
+ limit_mm_per_prompt={"image": len(image_urls)},
26
+ )
27
+
28
+
29
+def run_phi3v_generate(question: str, image_urls: List[str]):
30
+ llm = _load_phi3v(image_urls)
31
+
32
+ placeholders = "\n".join(f"<|image_{i}|>"
33
+ for i, _ in enumerate(image_urls, start=1))
34
+ prompt = f"<|user|>\n{placeholders}\n{question}<|end|>\n<|assistant|>\n"
35
+
36
+ outputs = llm.generate({
37
+ "prompt": prompt,
38
+ "multi_modal_data": {
39
+ "image": [fetch_image(url) for url in image_urls]
40
+ },
41
+ })
42
+
43
+ for o in outputs:
44
+ generated_text = o.outputs[0].text
45
+ print(generated_text)
46
+
47
+
48
+def run_phi3v_chat(question: str, image_urls: List[str]):
49
+ llm = _load_phi3v(image_urls)
50
+
51
+ outputs = llm.chat([{
52
+ "role":
53
+ "user",
54
+ "content": [
55
+ {
56
+ "type": "text",
57
+ "text": question,
58
+ },
59
+ *({
60
+ "type": "image_url",
61
+ "image_url": {
62
+ "url": image_url
63
+ },
64
+ } for image_url in image_urls),
65
+ ],
66
+ }])
67
+
68
+ for o in outputs:
69
+ generated_text = o.outputs[0].text
70
+ print(generated_text)
71
+
72
+
73
+def main(args: Namespace):
74
+ method = args.method
75
+
76
+ if method == "generate":
77
+ run_phi3v_generate(QUESTION, IMAGE_URLS)
78
+ elif method == "chat":
79
+ run_phi3v_chat(QUESTION, IMAGE_URLS)
80
+ else:
81
+ raise ValueError(f"Invalid method: {method}")
82
+
83
+
84
+if __name__ == "__main__":
85
+ parser = FlexibleArgumentParser(
86
+ description='Demo on using vLLM for offline inference with '
87
+ 'vision language models that support multi-image input')
88
+ parser.add_argument("--method",
89
+ type=str,
90
+ default="generate",
91
+ choices=["generate", "chat"],
92
+ help="The method to run in `vllm.LLM`.")
93
+
94
+ args = parser.parse_args()
95
+ main(args)
examples/openai_vision_api_client.py
+5
-4
@@ -27,9 +27,10 @@ client = OpenAI(
27
models = client.models.list()
28
model = models.data[0].id
29
30
+# Single-image input inference
31
image_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
32
32
-# Use image url in the payload
33
+## Use image url in the payload
34
chat_completion_from_url = client.chat.completions.create(
35
messages=[{
36
"role":
@@ -52,10 +53,10 @@ chat_completion_from_url = client.chat.completions.create(
53
)
54
55
result = chat_completion_from_url.choices[0].message.content
55
-print(f"Chat completion output:{result}")
56
+print("Chat completion output:", result)
57
58
58
-# Use base64 encoded image in the payload
59
+## Use base64 encoded image in the payload
60
def encode_image_base64_from_url(image_url: str) -> str:
61
"""Encode an image retrieved from a remote url to base64 format."""
62
@@ -122,4 +123,4 @@ chat_completion_from_url = client.chat.completions.create(
123
)
124
125
result = chat_completion_from_url.choices[0].message.content
125
-print(f"Chat completion output:{result}")
126
+print("Chat completion output:", result)