A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 10, 16:30 UTC
[Doc] Fix format of multimodal_inputs.md (#31800)
Signed-off-by: BlankR <hjyblanche@gmail.com>
BlankR committed
Jan 6, 2026 at 03:30 UTC
6ebb66ccea611fa8bd8718074836917c6ad9f5ef
1 file changed
+45
-41
docs/features/multimodal_inputs.md
+45
-41
@@ -166,49 +166,51 @@ Full example: [examples/offline_inference/vision_language_multi_image.py](../../
166
167
If using the [LLM.chat](../models/generative_models.md#llmchat) method, you can pass images directly in the message content using various formats: image URLs, PIL Image objects, or pre-computed embeddings:
168
169
-```python
170
-from vllm import LLM
171
-from vllm.assets.image import ImageAsset
172
-
173
-llm = LLM(model="llava-hf/llava-1.5-7b-hf")
174
-image_url = "https://picsum.photos/id/32/512/512"
175
-image_pil = ImageAsset('cherry_blossom').pil_image
176
-image_embeds = torch.load(...)
177
-
178
-conversation = [
179
- {"role": "system", "content": "You are a helpful assistant"},
180
- {"role": "user", "content": "Hello"},
181
- {"role": "assistant", "content": "Hello! How can I assist you today?"},
182
- {
183
- "role": "user",
184
- "content": [
185
- {
186
- "type": "image_url",
187
- "image_url": {"url": image_url},
188
- },
189
- {
190
- "type": "image_pil",
191
- "image_pil": image_pil,
192
- },
193
- {
194
- "type": "image_embeds",
195
- "image_embeds": image_embeds,
196
- },
197
- {
198
- "type": "text",
199
- "text": "What's in these images?",
200
- },
201
- ],
202
- },
203
-]
169
+??? code
170
205
-# Perform inference and log output.
206
-outputs = llm.chat(conversation)
171
+ ```python
172
+ from vllm import LLM
173
+ from vllm.assets.image import ImageAsset
174
208
-for o in outputs:
209
- generated_text = o.outputs[0].text
210
- print(generated_text)
211
-```
175
+ llm = LLM(model="llava-hf/llava-1.5-7b-hf")
176
+ image_url = "https://picsum.photos/id/32/512/512"
177
+ image_pil = ImageAsset('cherry_blossom').pil_image
178
+ image_embeds = torch.load(...)
179
+
180
+ conversation = [
181
+ {"role": "system", "content": "You are a helpful assistant"},
182
+ {"role": "user", "content": "Hello"},
183
+ {"role": "assistant", "content": "Hello! How can I assist you today?"},
184
+ {
185
+ "role": "user",
186
+ "content": [
187
+ {
188
+ "type": "image_url",
189
+ "image_url": {"url": image_url},
190
+ },
191
+ {
192
+ "type": "image_pil",
193
+ "image_pil": image_pil,
194
+ },
195
+ {
196
+ "type": "image_embeds",
197
+ "image_embeds": image_embeds,
198
+ },
199
+ {
200
+ "type": "text",
201
+ "text": "What's in these images?",
202
+ },
203
+ ],
204
+ },
205
+ ]
206
+
207
+ # Perform inference and log output.
208
+ outputs = llm.chat(conversation)
209
+
210
+ for o in outputs:
211
+ generated_text = o.outputs[0].text
212
+ print(generated_text)
213
+ ```
214
215
Multi-image input can be extended to perform video captioning. We show this with [Qwen2-VL](https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct) as it supports videos:
216
@@ -893,6 +895,8 @@ The following example demonstrates how to pass image embeddings to the OpenAI se
895
896
For Online Serving, you can also skip sending media if you expect cache hits with provided UUIDs. You can do so by sending media like this:
897
898
+??? code
899
+
900
```python
901
# Image/video/audio URL:
902
{