A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 10:45 UTC
[Doc] Update vlm.rst to include an example on videos (#9155)
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Sayak Paul committed
Oct 8, 2024 at 23:42 UTC
1874c6a1b0ae0f9eb2b485653b4e17ed1d861a32
1 file changed
+27
docs/source/models/vlm.rst
+27
@@ -135,6 +135,33 @@ Instead of passing in a single image, you can pass in a list of images.
135
136
A code example can be found in `examples/offline_inference_vision_language_multi_image.py <https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_vision_language_multi_image.py>`_.
137
138
+Multi-image input can be extended to perform video captioning. We show this with `Qwen2-VL <https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct>`_ as it supports videos:
139
+
140
+.. code-block:: python
141
+
142
+ # Specify the maximum number of frames per video to be 4. This can be changed.
143
+ llm = LLM("Qwen/Qwen2-VL-2B-Instruct", limit_mm_per_prompt={"image": 4})
144
+
145
+ # Create the request payload.
146
+ video_frames = ... # load your video making sure it only has the number of frames specified earlier.
147
+ message = {
148
+ "role": "user",
149
+ "content": [
150
+ {"type": "text", "text": "Describe this set of frames. Consider the frames to be a part of the same video."},
151
+ ],
152
+ }
153
+ for i in range(len(video_frames)):
154
+ base64_image = encode_image(video_frames[i]) # base64 encoding.
155
+ new_image = {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}}
156
+ message["content"].append(new_image)
157
+
158
+ # Perform inference and log output.
159
+ outputs = llm.chat([message])
160
+
161
+ for o in outputs:
162
+ generated_text = o.outputs[0].text
163
+ print(generated_text)
164
+
165
Online Inference
166
----------------
167