@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 1,411 lines 47.9 KB
Rendered Raw
1 # Multimodal Inputs
2
3 This page teaches you how to pass multi-modal inputs to [multi-modal models](../models/supported_models.md#list-of-multimodal-language-models) in vLLM.
4
5 !!! note
6 We are actively iterating on multi-modal support. See [this RFC](https://github.com/vllm-project/vllm/issues/4194) for upcoming changes,
7 and [open an issue on GitHub](https://github.com/vllm-project/vllm/issues/new/choose) if you have any feedback or feature requests.
8
9 !!! tip
10 When serving multi-modal models, consider setting `--allowed-media-domains` to restrict domain that vLLM can access to prevent it from accessing arbitrary endpoints that can potentially be vulnerable to Server-Side Request Forgery (SSRF) attacks. You can provide a list of domains for this arg. For example: `--allowed-media-domains upload.wikimedia.org github.com www.bogotobogo.com`
11
12 Also, consider setting `VLLM_MEDIA_URL_ALLOW_REDIRECTS=0` to prevent HTTP redirects from being followed to bypass domain restrictions.
13
14 This restriction is especially important if you run vLLM in a containerized environment where the vLLM pods may have unrestricted access to internal networks.
15
16 ## Offline Inference
17
18 To input multi-modal data, follow this schema in [vllm.inputs.PromptType][]:
19
20 - `prompt`: The prompt should follow the format that is documented on HuggingFace.
21 - `multi_modal_data`: This is a dictionary that follows the schema defined in [vllm.inputs.MultiModalDataDict][].
22
23 ### Image Inputs
24
25 You can pass a single image to the `'image'` field of the multi-modal dictionary, as shown in the following examples:
26
27 ??? code
28
29 ```python
30 from vllm import LLM
31
32 llm = LLM(model="llava-hf/llava-1.5-7b-hf")
33
34 # Refer to the HuggingFace repo for the correct format to use
35 prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:"
36
37 # Load the image using PIL.Image
38 image = PIL.Image.open(...)
39
40 # Single prompt inference
41 outputs = llm.generate({
42 "prompt": prompt,
43 "multi_modal_data": {"image": image},
44 })
45
46 for o in outputs:
47 generated_text = o.outputs[0].text
48 print(generated_text)
49
50 # Batch inference
51 image_1 = PIL.Image.open(...)
52 image_2 = PIL.Image.open(...)
53 outputs = llm.generate(
54 [
55 {
56 "prompt": "USER: <image>\nWhat is the content of this image?\nASSISTANT:",
57 "multi_modal_data": {"image": image_1},
58 },
59 {
60 "prompt": "USER: <image>\nWhat's the color of this image?\nASSISTANT:",
61 "multi_modal_data": {"image": image_2},
62 }
63 ]
64 )
65
66 for o in outputs:
67 generated_text = o.outputs[0].text
68 print(generated_text)
69 ```
70
71 Full example: [examples/generate/multimodal/vision_language_offline.py](../../examples/generate/multimodal/vision_language_offline.py)
72
73 To substitute multiple images inside the same text prompt, you can pass in a list of images instead:
74
75 ??? code
76
77 ```python
78 from vllm import LLM
79
80 llm = LLM(
81 model="microsoft/Phi-3.5-vision-instruct",
82 trust_remote_code=True, # Required to load Phi-3.5-vision
83 max_model_len=4096, # Otherwise, it may not fit in smaller GPUs
84 limit_mm_per_prompt={"image": 2}, # The maximum number to accept
85 )
86
87 # Refer to the HuggingFace repo for the correct format to use
88 prompt = "<|user|>\n<|image_1|>\n<|image_2|>\nWhat is the content of each image?<|end|>\n<|assistant|>\n"
89
90 # Load the images using PIL.Image
91 image1 = PIL.Image.open(...)
92 image2 = PIL.Image.open(...)
93
94 outputs = llm.generate({
95 "prompt": prompt,
96 "multi_modal_data": {"image": [image1, image2]},
97 })
98
99 for o in outputs:
100 generated_text = o.outputs[0].text
101 print(generated_text)
102 ```
103
104 Full example: [examples/generate/multimodal/vision_language_multi_image_offline.py](../../examples/generate/multimodal/vision_language_multi_image_offline.py)
105
106 If using the [LLM.chat](../models/generative_models.md#llmchat) method, you can pass images directly in the message content using various formats: image URLs, PIL Image objects, or pre-computed embeddings:
107
108 ??? code
109
110 ```python
111 from vllm import LLM
112 from vllm.assets.image import ImageAsset
113
114 llm = LLM(model="llava-hf/llava-1.5-7b-hf")
115 image_url = "https://picsum.photos/id/32/512/512"
116 image_pil = ImageAsset('cherry_blossom').pil_image
117 image_embeds = torch.load(...)
118
119 conversation = [
120 {"role": "system", "content": "You are a helpful assistant"},
121 {"role": "user", "content": "Hello"},
122 {"role": "assistant", "content": "Hello! How can I assist you today?"},
123 {
124 "role": "user",
125 "content": [
126 {
127 "type": "image_url",
128 "image_url": {"url": image_url},
129 },
130 {
131 "type": "image_pil",
132 "image_pil": image_pil,
133 },
134 {
135 "type": "image_embeds",
136 "image_embeds": image_embeds,
137 },
138 {
139 "type": "text",
140 "text": "What's in these images?",
141 },
142 ],
143 },
144 ]
145
146 # Perform inference and log output.
147 outputs = llm.chat(conversation)
148
149 for o in outputs:
150 generated_text = o.outputs[0].text
151 print(generated_text)
152 ```
153
154 Multi-image input can be extended to perform video captioning. We show this with [Qwen2-VL](https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct) as it supports videos:
155
156 ??? code
157
158 ```python
159 from vllm import LLM
160
161 # Specify the maximum number of frames per video to be 4. This can be changed.
162 llm = LLM("Qwen/Qwen2-VL-2B-Instruct", limit_mm_per_prompt={"image": 4})
163
164 # Create the request payload.
165 video_frames = ... # load your video making sure it only has the number of frames specified earlier.
166 message = {
167 "role": "user",
168 "content": [
169 {
170 "type": "text",
171 "text": "Describe this set of frames. Consider the frames to be a part of the same video.",
172 },
173 ],
174 }
175 for i in range(len(video_frames)):
176 base64_image = encode_image(video_frames[i]) # base64 encoding.
177 new_image = {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}}
178 message["content"].append(new_image)
179
180 # Perform inference and log output.
181 outputs = llm.chat([message])
182
183 for o in outputs:
184 generated_text = o.outputs[0].text
185 print(generated_text)
186 ```
187
188 #### Custom RGBA Background Color
189
190 When loading RGBA images (images with transparency), vLLM converts them to RGB format. By default, transparent pixels are replaced with white background. You can customize this background color using the `rgba_background_color` parameter in `media_io_kwargs`.
191
192 ??? code
193
194 ```python
195 from vllm import LLM
196
197 # Default white background (no configuration needed)
198 llm = LLM(model="llava-hf/llava-1.5-7b-hf")
199
200 # Custom black background for dark theme
201 llm = LLM(
202 model="llava-hf/llava-1.5-7b-hf",
203 media_io_kwargs={"image": {"rgba_background_color": [0, 0, 0]}},
204 )
205
206 # Custom brand color background (e.g., blue)
207 llm = LLM(
208 model="llava-hf/llava-1.5-7b-hf",
209 media_io_kwargs={"image": {"rgba_background_color": [0, 0, 255]}},
210 )
211 ```
212
213 !!! note
214 - The `rgba_background_color` accepts RGB values as a list `[R, G, B]` or tuple `(R, G, B)` where each value is 0-255
215 - This setting only affects RGBA images with transparency; RGB images are unchanged
216 - If not specified, the default white background `(255, 255, 255)` is used for backward compatibility
217
218 #### Moondream3 Prompt Recipes { #moondream3-prompt-recipes }
219
220 `Moondream3ForCausalLM` supports two task-specific prompt formats:
221
222 - `query`: ask a question about the image.
223 - `caption`: generate a caption for the image.
224
225 ```python
226 from vllm import LLM, SamplingParams
227 from vllm.assets.image import ImageAsset
228
229 llm = LLM(
230 model="moondream/moondream3-preview",
231 tokenizer="moondream/starmie-v1",
232 trust_remote_code=True,
233 max_model_len=2048,
234 limit_mm_per_prompt={"image": 1},
235 )
236
237 image = ImageAsset("stop_sign").pil_image
238
239
240 def make_query_prompt(question: str) -> str:
241 return (
242 "<|endoftext|><image><|md_reserved_0|>query<|md_reserved_1|>"
243 f"{question}<|md_reserved_2|>"
244 )
245
246
247 def make_caption_prompt(length: str = "normal") -> str:
248 return (
249 "<|endoftext|><image><|md_reserved_0|>"
250 f"describe<|md_reserved_1|>{length}<|md_reserved_2|>"
251 )
252
253
254 query_out = llm.generate(
255 {
256 "prompt": make_query_prompt("What is shown in this image?"),
257 "multi_modal_data": {"image": image},
258 },
259 SamplingParams(max_tokens=64, temperature=0),
260 )[0].outputs[0].text
261
262 caption_out = llm.generate(
263 {
264 "prompt": make_caption_prompt(),
265 "multi_modal_data": {"image": image},
266 },
267 SamplingParams(max_tokens=100, temperature=0),
268 )[0].outputs[0].text
269
270 print("query:", query_out)
271 print("caption:", caption_out)
272 ```
273
274 !!! note
275 The native Moondream3 model also has `detect` and `point` skills. Those
276 require custom coordinate decoding and are not exposed by this vLLM
277 implementation.
278
279 ### Video Inputs
280
281 You can pass a list of NumPy arrays directly to the `'video'` field of the multi-modal dictionary
282 instead of using multi-image input.
283
284 Instead of NumPy arrays, you can also pass `'torch.Tensor'` instances, as shown in this example using Qwen2.5-VL:
285
286 ??? code
287
288 ```python
289 from transformers import AutoProcessor
290 from vllm import LLM, SamplingParams
291 from qwen_vl_utils import process_vision_info
292
293 model_path = "Qwen/Qwen2.5-VL-3B-Instruct"
294 video_path = "https://content.pexels.com/videos/free-videos.mp4"
295
296 llm = LLM(
297 model=model_path,
298 gpu_memory_utilization=0.8,
299 enforce_eager=True,
300 limit_mm_per_prompt={"video": 1},
301 )
302
303 sampling_params = SamplingParams(max_tokens=1024)
304
305 video_messages = [
306 {
307 "role": "system",
308 "content": "You are a helpful assistant.",
309 },
310 {
311 "role": "user",
312 "content": [
313 {"type": "text", "text": "describe this video."},
314 {
315 "type": "video",
316 "video": video_path,
317 "total_pixels": 20480 * 28 * 28,
318 "min_pixels": 16 * 28 * 28,
319 },
320 ]
321 },
322 ]
323
324 messages = video_messages
325 processor = AutoProcessor.from_pretrained(model_path)
326 prompt = processor.apply_chat_template(
327 messages,
328 tokenize=False,
329 add_generation_prompt=True,
330 )
331
332 image_inputs, video_inputs = process_vision_info(messages)
333 mm_data = {}
334 if video_inputs is not None:
335 mm_data["video"] = video_inputs
336
337 llm_inputs = {
338 "prompt": prompt,
339 "multi_modal_data": mm_data,
340 }
341
342 outputs = llm.generate([llm_inputs], sampling_params=sampling_params)
343 for o in outputs:
344 generated_text = o.outputs[0].text
345 print(generated_text)
346 ```
347
348 !!! note
349 'process_vision_info' is only applicable to Qwen2.5-VL and similar models.
350
351 Full example: [examples/generate/multimodal/vision_language_offline.py](../../examples/generate/multimodal/vision_language_offline.py)
352
353 #### Video Token Pruning
354
355 For supported models, vLLM can prune video tokens after the vision encoder to
356 reduce prefill time and KV cache usage, at some cost in accuracy. Set
357 `--video-pruning-rate <q>` to prune the fraction `q` of video tokens from each
358 video, and `--video-pruning-method` to choose the training-free algorithm:
359
360 - **`evs`** (Efficient Video Sampling, default): drops the tokens with the
361 lowest temporal dissimilarity to the previous frame. The first frame is
362 always fully retained.
363 - **`vidcom2`** (Video Compression Commander): scores tokens by similarity to
364 video-level and frame-level feature centers and gives distinctive frames a
365 larger share of the budget. At least one token per frame is retained.
366
367 ```bash
368 vllm serve Qwen/Qwen3-VL-8B-Instruct \
369 --video-pruning-rate 0.75 --video-pruning-method vidcom2
370 ```
371
372 !!! note
373 `evs` is supported by all models implementing multimodal pruning;
374 `vidcom2` is currently supported by Qwen3-VL only. Unsupported combinations
375 are rejected at startup. Enabling video pruning also disables encoder CUDA
376 graphs, since the retained token count becomes data-dependent.
377
378 ### Audio Inputs
379
380 You can pass a tuple `(array, sampling_rate)` to the `'audio'` field of the multi-modal dictionary.
381
382 Full example: [examples/generate/multimodal/audio_language_offline.py](../../examples/generate/multimodal/audio_language_offline.py)
383
384 #### Chunking Long Audio for Transcription
385
386 Speech-to-text models like Whisper have a maximum audio length they can process (typically 30 seconds). For longer audio files, vLLM provides a utility to intelligently split audio into chunks at quiet points to minimize cutting through speech.
387
388 ```python
389 from vllm import LLM, SamplingParams
390 from vllm.multimodal.audio import split_audio
391 from vllm.multimodal.media.audio import load_audio
392
393 # Load long audio file
394 audio, sr = load_audio("long_audio.wav", sr=16000)
395
396 # Split into chunks at low-energy (quiet) regions
397 chunks = split_audio(
398 audio_data=audio,
399 sample_rate=sr,
400 max_clip_duration_s=30.0, # Maximum chunk length in seconds
401 overlap_duration_s=1.0, # Search window for finding quiet split points
402 min_energy_window_size=1600, # Window size for energy calculation (~100ms at 16kHz)
403 )
404
405 # Initialize Whisper model
406 llm = LLM(model="openai/whisper-large-v3-turbo")
407 sampling_params = SamplingParams(temperature=0, max_tokens=256)
408
409 # Transcribe each chunk
410 transcriptions = []
411 for chunk in chunks:
412 outputs = llm.generate({
413 "prompt": "<|startoftranscript|><|en|><|transcribe|><|notimestamps|>",
414 "multi_modal_data": {"audio": (chunk, sr)},
415 }, sampling_params)
416 transcriptions.append(outputs[0].outputs[0].text)
417
418 # Combine results
419 full_transcription = " ".join(transcriptions)
420 ```
421
422 The `split_audio` function:
423
424 - Expects 1D mono audio (`load_audio` downmixes by default)
425 - Splits audio at quiet points to avoid cutting through speech
426 - Uses RMS energy to find low-amplitude regions within the overlap window
427 - Preserves all audio samples (no data loss)
428 - Supports any sample rate
429
430 #### Automatic Audio Channel Normalization
431
432 vLLM automatically normalizes audio channels for models that require specific audio formats. When loading audio with libraries like `torchaudio`, stereo files return shape `[channels, time]`, but many audio models (particularly Whisper-based models) expect mono audio with shape `[time]`.
433
434 **Supported models with automatic mono conversion:**
435
436 - **Whisper** and all Whisper-based models
437 - **Qwen2-Audio**
438 - **Qwen2.5-Omni** / **Qwen3-Omni** (inherits from Qwen2.5-Omni)
439 - **Ultravox**
440
441 For these models, vLLM automatically:
442
443 1. Detects if the model requires mono audio via the feature extractor
444 2. Converts multi-channel audio to mono using channel averaging
445 3. Handles both `(channels, time)` format (torchaudio) and `(time, channels)` format (soundfile)
446
447 **Example with stereo audio:**
448
449 ```python
450 import torchaudio
451 from vllm import LLM
452
453 # Load stereo audio file - returns (channels, time) shape
454 audio, sr = torchaudio.load("stereo_audio.wav")
455 print(f"Original shape: {audio.shape}") # e.g., torch.Size([2, 16000])
456
457 # vLLM automatically converts to mono for Whisper-based models
458 llm = LLM(model="openai/whisper-large-v3")
459
460 outputs = llm.generate({
461 "prompt": "",
462 "multi_modal_data": {"audio": (audio.numpy(), sr)},
463 })
464 ```
465
466 No manual conversion is needed - vLLM handles the channel normalization automatically based on the model's requirements.
467
468 ### Embedding Inputs
469
470 To input pre-computed embeddings belonging to a data type (i.e. image, video, or audio) directly to the language model,
471 pass a tensor of shape `(..., hidden_size of LM)` to the corresponding field of the multi-modal dictionary.
472 The exact shape depends on the model being used.
473
474 You must enable this feature via `enable_mm_embeds=True`.
475
476 !!! warning
477 The vLLM engine may crash if incorrect shape of embeddings is passed.
478 Only enable this flag for trusted users!
479
480 #### Image Embeddings
481
482 ??? code
483
484 ```python
485 from vllm import LLM
486
487 # Inference with image embeddings as input
488 llm = LLM(model="llava-hf/llava-1.5-7b-hf", enable_mm_embeds=True)
489
490 # Refer to the HuggingFace repo for the correct format to use
491 prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:"
492
493 # For most models, `image_embeds` has shape: (num_images, image_feature_size, hidden_size)
494 image_embeds = torch.load(...)
495
496 outputs = llm.generate({
497 "prompt": prompt,
498 "multi_modal_data": {"image": image_embeds},
499 })
500
501 for o in outputs:
502 generated_text = o.outputs[0].text
503 print(generated_text)
504
505 # Additional examples for models that require extra fields
506 llm = LLM(
507 "Qwen/Qwen2-VL-2B-Instruct",
508 limit_mm_per_prompt={"image": 4},
509 enable_mm_embeds=True,
510 )
511 mm_data = {
512 "image": {
513 # Shape: (total_feature_size, hidden_size)
514 # total_feature_size = sum(image_feature_size for image in images)
515 "image_embeds": torch.load(...),
516 # Shape: (num_images, 3)
517 # image_grid_thw is needed to calculate positional encoding.
518 "image_grid_thw": torch.load(...),
519 }
520 }
521
522 llm = LLM(
523 "openbmb/MiniCPM-V-2_6",
524 trust_remote_code=True,
525 limit_mm_per_prompt={"image": 4},
526 enable_mm_embeds=True,
527 )
528 mm_data = {
529 "image": {
530 # Shape: (num_images, num_slices, hidden_size)
531 # num_slices can differ for each image
532 "image_embeds": [torch.load(...) for image in images],
533 # Shape: (num_images, 2)
534 # image_sizes is needed to calculate details of the sliced image.
535 "image_sizes": [image.size for image in images],
536 }
537 }
538 ```
539
540 For Qwen3-VL, the `image_embeds` should contain both the base image embedding and deepstack features.
541
542 #### Audio Embedding Inputs
543
544 You can pass pre-computed audio embeddings similar to image embeddings:
545
546 ??? code
547
548 ```python
549 from vllm import LLM
550 import torch
551
552 # Enable audio embeddings support
553 llm = LLM(model="fixie-ai/ultravox-v0_5-llama-3_2-1b", enable_mm_embeds=True)
554
555 # Refer to the HuggingFace repo for the correct format to use
556 prompt = "USER: <audio>\nWhat is in this audio?\nASSISTANT:"
557
558 # Load pre-computed audio embeddings, usually with shape:
559 # (num_audios, audio_feature_size, hidden_size of LM)
560 audio_embeds = torch.load(...)
561
562 outputs = llm.generate({
563 "prompt": prompt,
564 "multi_modal_data": {"audio": audio_embeds},
565 })
566
567 for o in outputs:
568 generated_text = o.outputs[0].text
569 print(generated_text)
570 ```
571
572 ### Cached Inputs
573
574 When using multi-modal inputs, vLLM normally hashes each media item by content to enable caching across requests. You can optionally pass `multi_modal_uuids` to provide your own stable IDs for each item so caching can reuse work across requests without rehashing the raw content.
575
576 ??? code
577
578 ```python
579 from vllm import LLM
580 from PIL import Image
581
582 # Qwen2.5-VL example with two images
583 llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct")
584
585 prompt = "USER: <image><image>\nDescribe the differences.\nASSISTANT:"
586 img_a = Image.open("/path/to/a.jpg")
587 img_b = Image.open("/path/to/b.jpg")
588
589 outputs = llm.generate({
590 "prompt": prompt,
591 "multi_modal_data": {"image": [img_a, img_b]},
592 # Provide stable IDs for caching.
593 # Requirements (matched by this example):
594 # - Include every modality present in multi_modal_data.
595 # - For lists, provide the same number of entries.
596 # - Use None to fall back to content hashing for that item.
597 "multi_modal_uuids": {"image": ["sku-1234-a", None]},
598 })
599
600 for o in outputs:
601 print(o.outputs[0].text)
602 ```
603
604 Using UUIDs, you can also skip sending media data entirely if you expect cache hits for respective items. Note that the request will fail if the skipped media doesn't have a corresponding UUID, or if the UUID fails to hit the cache.
605
606 ??? code
607
608 ```python
609 from vllm import LLM
610 from PIL import Image
611
612 # Qwen2.5-VL example with two images
613 llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct")
614
615 prompt = "USER: <image><image>\nDescribe the differences.\nASSISTANT:"
616 img_b = Image.open("/path/to/b.jpg")
617
618 outputs = llm.generate({
619 "prompt": prompt,
620 "multi_modal_data": {"image": [None, img_b]},
621 # Since img_a is expected to be cached, we can skip sending the actual
622 # image entirely.
623 "multi_modal_uuids": {"image": ["sku-1234-a", None]},
624 })
625
626 for o in outputs:
627 print(o.outputs[0].text)
628 ```
629
630 !!! warning
631 If both multimodal processor caching and prefix caching are disabled, user-provided `multi_modal_uuids` are ignored.
632
633 ## Online Serving
634
635 Our OpenAI-compatible server accepts multi-modal data via the [Chat Completions API](https://platform.openai.com/docs/api-reference/chat). Media inputs also support optional UUIDs users can provide to uniquely identify each media, which is used to cache the media results across requests.
636
637 !!! important
638 A chat template is **required** to use Chat Completions API.
639 For HF format models, the default chat template is defined inside `chat_template.json` or `tokenizer_config.json`.
640
641 If no default chat template is available, we will first look for a built-in fallback in [vllm/transformers_utils/chat_templates/registry.py](../../vllm/transformers_utils/chat_templates/registry.py).
642 If no fallback is available, an error is raised and you have to provide the chat template manually via the `--chat-template` argument.
643
644 For certain models, we provide alternative chat templates inside [examples](../../examples).
645 For example, VLM2Vec uses [examples/pooling/embed/template/vlm2vec_phi3v.jinja](../../examples/pooling/embed/template/vlm2vec_phi3v.jinja) which is different from the default one for Phi-3-Vision.
646
647 ### Image Inputs
648
649 Image input is supported according to [OpenAI Vision API](https://platform.openai.com/docs/guides/vision).
650 Here is a simple example using Phi-3.5-Vision.
651
652 First, launch the OpenAI-compatible server:
653
654 ```bash
655 vllm serve microsoft/Phi-3.5-vision-instruct --runner generate \
656 --trust-remote-code --max-model-len 4096 --limit-mm-per-prompt.image 2
657 ```
658
659 Then, you can use the OpenAI client as follows:
660
661 ??? code
662
663 ```python
664 import os
665 from openai import OpenAI
666
667 openai_api_key = "EMPTY"
668 openai_api_base = "http://localhost:8000/v1"
669
670 client = OpenAI(
671 api_key=openai_api_key,
672 base_url=openai_api_base,
673 )
674
675 # Single-image input inference
676
677 # Public image URL for testing remote image processing
678 image_url = "https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
679
680 # Create chat completion with remote image
681 chat_response = client.chat.completions.create(
682 model="microsoft/Phi-3.5-vision-instruct",
683 messages=[
684 {
685 "role": "user",
686 "content": [
687 # NOTE: The prompt formatting with the image token `<image>` is not needed
688 # since the prompt will be processed automatically by the API server.
689 {
690 "type": "text",
691 "text": "What’s in this image?",
692 },
693 {
694 "type": "image_url",
695 "image_url": {"url": image_url},
696 "uuid": image_url, # Optional
697 },
698 ],
699 }
700 ],
701 )
702 print("Chat completion output:", chat_response.choices[0].message.content)
703
704 # Local image file path (update this to point to your actual image file)
705 image_file = "/path/to/image.jpg"
706
707 # Create chat completion with local image file
708 # Launch the API server/engine with the --allowed-local-media-path argument.
709 if os.path.exists(image_file):
710 chat_completion_from_local_image_url = client.chat.completions.create(
711 model="microsoft/Phi-3.5-vision-instruct",
712 messages=[
713 {
714 "role": "user",
715 "content": [
716 {
717 "type": "text",
718 "text": "What’s in this image?",
719 },
720 {
721 "type": "image_url",
722 "image_url": {"url": f"file://{image_file}"},
723 },
724 ],
725 }
726 ],
727 )
728 result = chat_completion_from_local_image_url.choices[0].message.content
729 print("Chat completion output from local image file:\n", result)
730 else:
731 print(f"Local image file not found at {image_file}, skipping local file test.")
732
733 # Multi-image input inference
734 image_url_duck = "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/duck.jpg"
735 image_url_lion = "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/lion.jpg"
736
737 chat_response = client.chat.completions.create(
738 model="microsoft/Phi-3.5-vision-instruct",
739 messages=[
740 {
741 "role": "user",
742 "content": [
743 {
744 "type": "text",
745 "text": "What are the animals in these images?",
746 },
747 {
748 "type": "image_url",
749 "image_url": {"url": image_url_duck},
750 "uuid": image_url_duck, # Optional
751 },
752 {
753 "type": "image_url",
754 "image_url": {"url": image_url_lion},
755 "uuid": image_url_lion, # Optional
756 },
757 ],
758 }
759 ],
760 )
761 print("Chat completion output:", chat_response.choices[0].message.content)
762 ```
763
764 Full example: [examples/generate/multimodal/openai_chat_completion_client_for_multimodal.py](../../examples/generate/multimodal/openai_chat_completion_client_for_multimodal.py)
765
766 !!! tip
767 Loading from local file paths is also supported on vLLM: You can specify the allowed local media path via `--allowed-local-media-path` when launching the API server/engine,
768 and pass the file path as `url` in the API request.
769
770 !!! tip
771 There is no need to place image placeholders in the text content of the API request - they are already represented by the image content.
772 In fact, you can place image placeholders in the middle of the text by interleaving text and image content.
773
774 !!! note
775 By default, the timeout for fetching images through HTTP URL is `5` seconds.
776 You can override this by setting the environment variable:
777
778 ```bash
779 export VLLM_IMAGE_FETCH_TIMEOUT=<timeout>
780 ```
781
782 ### Video Inputs
783
784 Instead of `image_url`, you can pass a video file via `video_url`. Here is a simple example using [LLaVA-OneVision](https://huggingface.co/llava-hf/llava-onevision-qwen2-0.5b-ov-hf).
785
786 First, launch the OpenAI-compatible server:
787
788 ```bash
789 vllm serve llava-hf/llava-onevision-qwen2-0.5b-ov-hf --runner generate --max-model-len 8192
790 ```
791
792 Then, you can use the OpenAI client as follows:
793
794 ??? code
795
796 ```python
797 from openai import OpenAI
798
799 openai_api_key = "EMPTY"
800 openai_api_base = "http://localhost:8000/v1"
801
802 client = OpenAI(
803 api_key=openai_api_key,
804 base_url=openai_api_base,
805 )
806
807 video_url = "https://huggingface.co/datasets/raushan-testing-hf/videos-test/resolve/main/sample_demo_1.mp4"
808
809 ## Use video url in the payload
810 chat_completion_from_url = client.chat.completions.create(
811 messages=[
812 {
813 "role": "user",
814 "content": [
815 {
816 "type": "text",
817 "text": "What's in this video?",
818 },
819 {
820 "type": "video_url",
821 "video_url": {"url": video_url},
822 "uuid": video_url, # Optional
823 },
824 ],
825 }
826 ],
827 model=model,
828 max_completion_tokens=64,
829 )
830
831 result = chat_completion_from_url.choices[0].message.content
832 print("Chat completion output from image url:", result)
833 ```
834
835 Full example: [examples/generate/multimodal/openai_chat_completion_client_for_multimodal.py](../../examples/generate/multimodal/openai_chat_completion_client_for_multimodal.py)
836
837 !!! note
838 By default, the timeout for fetching videos through HTTP URL is `30` seconds.
839 You can override this by setting the environment variable:
840
841 ```bash
842 export VLLM_VIDEO_FETCH_TIMEOUT=<timeout>
843 ```
844
845 #### Video Decoding Backend
846
847 vLLM decodes video bytes into frames using a selectable decoding backend. Four
848 backends are supported:
849
850 | Backend | Device | Description |
851 | --- | --- | --- |
852 | `opencv` (default) | CPU | OpenCV-based decoder |
853 | `torchcodec` | CPU | TorchCodec (PyTorch-native) decoder |
854 | `pynvvideocodec` | GPU | NVIDIA PyNvVideoCodec decoder |
855 | `deepstream` | GPU | NVIDIA DeepStream decoder |
856
857 The two CPU backends are ultimately backed by FFmpeg. `torchcodec` lets you
858 choose which FFmpeg version is used while `opencv` relies on whichever
859 FFmpeg build it was linked against.
860
861 Select the backend by passing the `backend` parameter via `--media-io-kwargs`:
862
863 ```bash
864 vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
865 --media-io-kwargs '{"video": {"backend": "torchcodec"}}'
866 ```
867
868 **TorchCodec-specific parameters:**
869
870 The following parameters only apply to the `torchcodec` backend:
871
872 - `num_ffmpeg_threads`: Number of FFmpeg decoding threads. `0` (default) relies
873 on the FFmpeg default, which is `min(cpu_count + 1, 16)`. This allows you to
874 control thread over-subscription.
875 - `seek_mode`: Seek mode for the decoder. `"exact"` (default) guarantees
876 frame-accurate sampling by scanning the file when the decoder is created.
877 `"approximate"` skips that scan for faster decoder creation, at the cost of
878 relying on the file's metadata (which may yield less accurate seeking).
879
880 ```bash
881 # Example: TorchCodec with approximate seek mode and 4 FFmpeg threads
882 vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
883 --media-io-kwargs '{"video": {"backend": "torchcodec", "seek_mode": "approximate", "num_ffmpeg_threads": 4}}'
884 ```
885
886 **PyNvVideoCodec-specific parameters:**
887
888 - `hw_decoders`: Maximum number of concurrent hardware decoder slots retained
889 by each API server process. It must be a positive integer and defaults to `2`,
890 which is the recommended starting point for concurrent video workloads.
891 Because vLLM reserves GPU memory for these slots at startup, this value cannot
892 be overridden per request. Benchmark before increasing it because each
893 additional slot increases the GPU memory reservation.
894
895 ```bash
896 # Example: explicitly use the recommended 2 hardware decoders
897 vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
898 --media-io-kwargs '{"video": {"backend": "pynvvideocodec", "hw_decoders": 2}}'
899 ```
900
901 #### Video Frame Recovery
902
903 For improved robustness when processing potentially corrupted or truncated video files, vLLM supports optional frame recovery using a dynamic window forward-scan approach. When enabled, if a target frame fails to load during sequential reading, the next successfully grabbed frame (before the next target frame) will be used in its place.
904
905 To enable video frame recovery, pass the `frame_recovery` parameter via `--media-io-kwargs`:
906
907 ```bash
908 # Example: Enable frame recovery
909 vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
910 --media-io-kwargs '{"video": {"frame_recovery": true}}'
911 ```
912
913 **Parameters:**
914
915 - `frame_recovery`: Boolean flag to enable forward-scan recovery. When `true`, failed frames are recovered using the next available frame within the dynamic window (up to the next target frame). Default is `false`.
916
917 **How it works:**
918
919 1. The system reads frames sequentially
920 2. If a target frame fails to grab, it's marked as "failed"
921 3. The next successfully grabbed frame (before reaching the next target) is used to recover the failed frame
922 4. This approach handles both mid-video corruption and end-of-video truncation
923
924 Works with common video formats like MP4 when using OpenCV backends.
925
926 #### GPU Video Decoding with PyNvVideoCodec (NVDEC)
927
928 The `pynvvideocodec` backend uses NVIDIA NVDEC to decode the sampled video
929 frames on the GPU before copying them into host memory for multimodal
930 preprocessing. For workloads with large videos and relatively light inference,
931 such as video tagging, this can alleviate bottlenecks in CPU-based video
932 decoders.
933
934 !!! warning
935 [CUDA Multi-Process Service (MPS)](https://docs.nvidia.com/deploy/mps/quick-start.html)
936 is required when using this backend. Video decoding runs in the API server
937 process while model serving runs in the engine process, so multiple CUDA
938 processes share the same GPU. Configure and start MPS before starting vLLM.
939
940 You must also set a positive `--mm-ipc-gpu-memory-gb` value to reserve VRAM for
941 video decoding. vLLM carves this budget out of the memory available to the KV
942 cache and uses it to bound concurrent frontend decode allocations. If the
943 budget is exhausted, decode work waits instead of consuming the engine's VRAM
944 headroom and potentially causing an out-of-memory error while serving requests.
945
946 Select the backend with an environment variable and specify a workload-appropriate
947 VRAM budget. For example, to reserve 1 GiB:
948
949 ```bash
950 export VLLM_VIDEO_LOADER_BACKEND=pynvvideocodec
951 vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
952 --mm-ipc-gpu-memory-gb 1
953 ```
954
955 Alternatively, select it with `--media-io-kwargs`:
956
957 ```bash
958 vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
959 --media-io-kwargs '{"video": {"backend": "pynvvideocodec"}}' \
960 --mm-ipc-gpu-memory-gb 1
961 ```
962
963 Choose a budget large enough for the largest sampled video that a single API
964 server process must decode. When using multiple API server processes, vLLM
965 divides the configured budget evenly among them.
966
967 For streaming video sources, use the DeepStream backend instead.
968
969 #### GPU Video Decoding with DeepStream (NVDEC)
970
971 By default vLLM decodes video on the CPU. On NVIDIA GPUs you can instead decode
972 directly on the hardware video engine (NVDEC) with the DeepStream backend, which
973 keeps decoding off the CPU and can significantly increase video throughput. It
974 is the recommended GPU backend for streaming video sources.
975
976 Install the backend (Linux x86-64 only):
977
978 ```bash
979 pip install vllm[deepstream]
980 ```
981
982 The pip wheel bundles the DeepStream libraries but still relies on a few system
983 packages that pip cannot install. On Ubuntu:
984
985 ```bash
986 apt-get install -y \
987 gstreamer1.0-tools gstreamer1.0-plugins-base gstreamer1.0-plugins-good \
988 gstreamer1.0-plugins-bad gstreamer1.0-libav \
989 python3-gi python3-gst-1.0 libv4l-0 cuda-libraries-13-0
990 ```
991
992 Select the backend either with an environment variable:
993
994 ```bash
995 export VLLM_VIDEO_LOADER_BACKEND=deepstream
996 vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct
997 ```
998
999 or per request via `--media-io-kwargs`:
1000
1001 ```bash
1002 vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
1003 --media-io-kwargs '{"video": {"backend": "deepstream"}}'
1004 ```
1005
1006 **Parameters:**
1007
1008 - `pool_size`: Number of GPU decode workers in the process-wide decode pool
1009 (clamped to `[1, 16]`). When unset it defaults to
1010 `VLLM_MEDIA_LOADING_THREAD_COUNT` (default `8`). The pool is a singleton, so
1011 the first request's value wins.
1012
1013 ```bash
1014 # Example: 12 decode workers
1015 vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
1016 --media-io-kwargs '{"video": {"backend": "deepstream", "pool_size": 12}}'
1017 ```
1018
1019 #### Pre-extracted Frame Sequences with `media_io_kwargs`
1020
1021 When you extract video frames on the client side and send them as `video/jpeg` (base64-concatenated JPEG frames), you can preserve the original video metadata by using `media_io_kwargs` in your request. This enables more accurate video understanding by preserving temporal information that would otherwise be lost during client-side frame extraction.
1022
1023 **Supported Parameters:**
1024
1025 | Parameter | Type | Description |
1026 | --------- | ---- | ----------- |
1027 | `fps` | float | Frame rate of the original video |
1028 | `frames_indices` | list[int] | Indices of the actually sampled frames |
1029 | `total_num_frames` | int | Total frame count of the original video |
1030 | `duration` | float | Duration of the original video in seconds |
1031 | `do_sample_frames` | bool | Whether to perform frame sampling |
1032
1033 ??? code
1034
1035 ```python
1036 from openai import OpenAI
1037
1038 client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
1039
1040 # Client-side frame extraction
1041 frames = extract_frames(video_path, num_frames=32)
1042 frames_b64 = ",".join([encode_image(f) for f in frames])
1043 video_url = f"data:video/jpeg;base64,{frames_b64}"
1044
1045 # Pass video metadata via media_io_kwargs
1046 response = client.chat.completions.create(
1047 model="your-multimodal-model",
1048 messages=[{
1049 "role": "user",
1050 "content": [
1051 {"type": "video_url", "video_url": {"url": video_url}},
1052 {"type": "text", "text": "Describe what happens in this video."}
1053 ]
1054 }],
1055 extra_body={
1056 "media_io_kwargs": {
1057 "video": {
1058 "fps": 30.0,
1059 "frames_indices": [0, 10, 20, 30, 40, 50, 60, 70, 80, 90,
1060 100, 110, 120, 130, 140, 150, 160, 170,
1061 180, 190, 200, 210, 220, 230, 240, 250,
1062 260, 270, 280, 290, 300, 310],
1063 "total_num_frames": 900,
1064 "duration": 30.0,
1065 }
1066 }
1067 },
1068 )
1069
1070 print(response.choices[0].message.content)
1071 ```
1072
1073 **Why use `media_io_kwargs`?**
1074
1075 When extracting frames client-side, the server loses important context about the original video:
1076
1077 - **Temporal information**: Which frames were sampled and their positions in the original timeline
1078 - **Video duration**: How long the original video was
1079 - **Frame rate**: The original playback speed
1080
1081 By passing this metadata, the model can better understand the temporal distribution of the sampled frames and whether important moments might have been skipped.
1082
1083 #### Custom RGBA Background Color
1084
1085 To use a custom background color for RGBA images, pass the `rgba_background_color` parameter via `--media-io-kwargs`:
1086
1087 ```bash
1088 # Example: Black background for dark theme
1089 vllm serve llava-hf/llava-1.5-7b-hf \
1090 --media-io-kwargs '{"image": {"rgba_background_color": [0, 0, 0]}}'
1091
1092 # Example: Custom gray background
1093 vllm serve llava-hf/llava-1.5-7b-hf \
1094 --media-io-kwargs '{"image": {"rgba_background_color": [128, 128, 128]}}'
1095 ```
1096
1097 ### Audio Inputs
1098
1099 Audio input is supported according to [OpenAI Audio API](https://platform.openai.com/docs/guides/audio?audio-generation-quickstart-example=audio-in).
1100 Here is a simple example using Ultravox-v0.5-1B.
1101
1102 First, launch the OpenAI-compatible server:
1103
1104 ```bash
1105 vllm serve fixie-ai/ultravox-v0_5-llama-3_2-1b
1106 ```
1107
1108 Then, you can use the OpenAI client as follows:
1109
1110 ??? code
1111
1112 ```python
1113 import base64
1114 import requests
1115 from openai import OpenAI
1116 from vllm.assets.audio import AudioAsset
1117
1118 def encode_base64_content_from_url(content_url: str) -> str:
1119 """Encode a content retrieved from a remote url to base64 format."""
1120
1121 with requests.get(content_url) as response:
1122 response.raise_for_status()
1123 result = base64.b64encode(response.content).decode('utf-8')
1124
1125 return result
1126
1127 openai_api_key = "EMPTY"
1128 openai_api_base = "http://localhost:8000/v1"
1129
1130 client = OpenAI(
1131 api_key=openai_api_key,
1132 base_url=openai_api_base,
1133 )
1134
1135 # Any format supported by soundfile/PyAV is supported
1136 audio_url = AudioAsset("winning_call").url
1137 audio_base64 = encode_base64_content_from_url(audio_url)
1138
1139 chat_completion_from_base64 = client.chat.completions.create(
1140 messages=[
1141 {
1142 "role": "user",
1143 "content": [
1144 {
1145 "type": "text",
1146 "text": "What's in this audio?",
1147 },
1148 {
1149 "type": "input_audio",
1150 "input_audio": {
1151 "data": audio_base64,
1152 "format": "wav",
1153 },
1154 "uuid": audio_url, # Optional
1155 },
1156 ],
1157 },
1158 ],
1159 model=model,
1160 max_completion_tokens=64,
1161 )
1162
1163 result = chat_completion_from_base64.choices[0].message.content
1164 print("Chat completion output from input audio:", result)
1165 ```
1166
1167 Alternatively, you can pass `audio_url`, which is the audio counterpart of `image_url` for image input:
1168
1169 ??? code
1170
1171 ```python
1172 chat_completion_from_url = client.chat.completions.create(
1173 messages=[
1174 {
1175 "role": "user",
1176 "content": [
1177 {
1178 "type": "text",
1179 "text": "What's in this audio?",
1180 },
1181 {
1182 "type": "audio_url",
1183 "audio_url": {"url": audio_url},
1184 "uuid": audio_url, # Optional
1185 },
1186 ],
1187 }
1188 ],
1189 model=model,
1190 max_completion_tokens=64,
1191 )
1192
1193 result = chat_completion_from_url.choices[0].message.content
1194 print("Chat completion output from audio url:", result)
1195 ```
1196
1197 Full example: [examples/generate/multimodal/openai_chat_completion_client_for_multimodal.py](../../examples/generate/multimodal/openai_chat_completion_client_for_multimodal.py)
1198
1199 !!! note
1200 By default, the timeout for fetching audios through HTTP URL is `10` seconds.
1201 You can override this by setting the environment variable:
1202
1203 ```bash
1204 export VLLM_AUDIO_FETCH_TIMEOUT=<timeout>
1205 ```
1206
1207 #### Audio Decoding Backend
1208
1209 vLLM decodes audio bytes into waveforms using a selectable decoding backend:
1210
1211 | Backend | Description |
1212 | --- | --- |
1213 | `auto` (default) | torchcodec, falling back to soundfile, then PyAV |
1214 | `soundfile` | libsndfile only, no fallback |
1215 | `pyav` | PyAV (FFmpeg) only, no fallback |
1216 | `torchcodec` | TorchCodec (PyTorch-native) only, no fallback |
1217
1218 Select the backend per server via `--media-io-kwargs`:
1219
1220 ```bash
1221 vllm serve mistralai/Voxtral-Mini-3B-2507 \
1222 --media-io-kwargs '{"audio": {"audio_backend": "soundfile"}}'
1223 ```
1224
1225 !!! tip
1226 `pyav` drives FFmpeg through a per-frame Python generator, so under
1227 concurrency the Python/C crossings contend on the GIL. `torchcodec`
1228 decodes each stream in a single call that releases the GIL for its whole
1229 duration, which is why `auto` prefers it when many requests decode audio
1230 concurrently, such as when audio tracks are extracted from video.
1231 vLLM normalizes codec padding on this path (e.g. it trims trailing Vorbis
1232 padding that older system FFmpeg versions (< 5.0) fail to trim), so the
1233 decoded waveform matches the soundfile reference length.
1234
1235 !!! note
1236 `torchcodec` ships as a requirement on CUDA, CPU and XPU builds. On other
1237 platforms (e.g. ROCm, TPU) `auto` falls back to the soundfile → PyAV
1238 chain unless you install it manually. torchcodec also links against a
1239 system FFmpeg installation; if FFmpeg is missing, `auto` falls back the
1240 same way.
1241
1242 ### Embedding Inputs
1243
1244 To input pre-computed embeddings belonging to a data type (i.e. image, video, or audio) directly to the language model,
1245 pass a tensor of shape `(..., hidden_size of LM)` for each item to the corresponding field of the multi-modal dictionary.
1246
1247 !!! important
1248 Unlike offline inference, the embeddings for each item must be passed separately
1249 in order for placeholder tokens to be applied correctly by the chat template.
1250
1251 You must enable this feature via the `--enable-mm-embeds` flag in `vllm serve`.
1252
1253 !!! warning
1254 The vLLM engine may crash if incorrect shape of embeddings is passed.
1255 Only enable this flag for trusted users!
1256
1257 #### Image Embedding Inputs
1258
1259 For image embeddings, you can pass the base64-encoded tensor to the `image_embeds` field.
1260 The following example demonstrates how to pass image embeddings to the OpenAI server:
1261
1262 ??? code
1263
1264 ```python
1265 from vllm.utils.serial_utils import tensor2base64
1266
1267 client = OpenAI(
1268 # defaults to os.environ.get("OPENAI_API_KEY")
1269 api_key=openai_api_key,
1270 base_url=openai_api_base,
1271 )
1272
1273 # Basic usage - this is equivalent to the LLaVA example for offline inference
1274 model = "llava-hf/llava-1.5-7b-hf"
1275 embeds = {
1276 "type": "image_embeds",
1277 "image_embeds": tensor2base64(torch.load(...)), # Shape: (image_feature_size, hidden_size)
1278 "uuid": image_url, # Optional
1279 }
1280
1281
1282 # Additional examples for models that require extra fields
1283 model = "Qwen/Qwen2-VL-2B-Instruct"
1284 embeds = {
1285 "type": "image_embeds",
1286 "image_embeds": {
1287 "image_embeds": tensor2base64(torch.load(...)), # Shape: (image_feature_size, hidden_size)
1288 "image_grid_thw": tensor2base64(torch.load(...)), # Shape: (3,)
1289 },
1290 "uuid": image_url, # Optional
1291 }
1292
1293 model = "openbmb/MiniCPM-V-2_6"
1294 embeds = {
1295 "type": "image_embeds",
1296 "image_embeds": {
1297 "image_embeds": tensor2base64(torch.load(...)), # Shape: (num_slices, hidden_size)
1298 "image_sizes": tensor2base64(torch.load(...)), # Shape: (2,)
1299 },
1300 "uuid": image_url, # Optional
1301 }
1302
1303 # Single image input
1304 chat_completion = client.chat.completions.create(
1305 messages=[
1306 {
1307 "role": "system",
1308 "content": "You are a helpful assistant.",
1309 },
1310 {
1311 "role": "user",
1312 "content": [
1313 {
1314 "type": "text",
1315 "text": "What's in this image?",
1316 },
1317 embeds,
1318 ],
1319 },
1320 ],
1321 model=model,
1322 )
1323
1324 # Multi image input
1325 chat_completion = client.chat.completions.create(
1326 messages=[
1327 {
1328 "role": "system",
1329 "content": "You are a helpful assistant.",
1330 },
1331 {
1332 "role": "user",
1333 "content": [
1334 {
1335 "type": "text",
1336 "text": "What's in this image?",
1337 },
1338 embeds,
1339 embeds,
1340 ],
1341 },
1342 ],
1343 model=model,
1344 )
1345
1346 # Multi image input (interleaved)
1347 chat_completion = client.chat.completions.create(
1348 messages=[
1349 {
1350 "role": "system",
1351 "content": "You are a helpful assistant.",
1352 },
1353 {
1354 "role": "user",
1355 "content": [
1356 embeds,
1357 {
1358 "type": "text",
1359 "text": "What's in this image?",
1360 },
1361 embeds,
1362 ],
1363 },
1364 ],
1365 model=model,
1366 )
1367 ```
1368
1369 ### Cached Inputs
1370
1371 Just like with offline inference, you can skip sending media if you expect cache hits with provided UUIDs. You can do so by sending media like this:
1372
1373 ??? code
1374
1375 ```python
1376 # Image/video/audio URL:
1377 {
1378 "type": "image_url",
1379 "image_url": None,
1380 "uuid": image_uuid,
1381 },
1382
1383 # image_embeds
1384 {
1385 "type": "image_embeds",
1386 "image_embeds": None,
1387 "uuid": image_uuid,
1388 },
1389
1390 # input_audio:
1391 {
1392 "type": "input_audio",
1393 "input_audio": None,
1394 "uuid": audio_uuid,
1395 },
1396
1397 # PIL Image:
1398 {
1399 "type": "image_pil",
1400 "image_pil": None,
1401 "uuid": image_uuid,
1402 },
1403
1404 # video_url:
1405 {
1406 "type": "video_url",
1407 "video_url": {},
1408 "uuid": video_uuid,
1409 },
1410
1411 ```