@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 1874c6a1b0
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 10:45 UTC

[Doc] Update vlm.rst to include an example on videos (#9155)

Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

Sayak Paul committed Oct 8, 2024 at 23:42 UTC 1874c6a1b0ae0f9eb2b485653b4e17ed1d861a32
1 file changed +27
docs/source/models/vlm.rst
+27
@@ -135,6 +135,33 @@ Instead of passing in a single image, you can pass in a list of images.
135
136 A code example can be found in `examples/offline_inference_vision_language_multi_image.py <https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_vision_language_multi_image.py>`_.
137
138 +Multi-image input can be extended to perform video captioning. We show this with `Qwen2-VL <https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct>`_ as it supports videos:
139 +
140 +.. code-block:: python
141 +
142 + # Specify the maximum number of frames per video to be 4. This can be changed.
143 + llm = LLM("Qwen/Qwen2-VL-2B-Instruct", limit_mm_per_prompt={"image": 4})
144 +
145 + # Create the request payload.
146 + video_frames = ... # load your video making sure it only has the number of frames specified earlier.
147 + message = {
148 + "role": "user",
149 + "content": [
150 + {"type": "text", "text": "Describe this set of frames. Consider the frames to be a part of the same video."},
151 + ],
152 + }
153 + for i in range(len(video_frames)):
154 + base64_image = encode_image(video_frames[i]) # base64 encoding.
155 + new_image = {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}}
156 + message["content"].append(new_image)
157 +
158 + # Perform inference and log output.
159 + outputs = llm.chat([message])
160 +
161 + for o in outputs:
162 + generated_text = o.outputs[0].text
163 + print(generated_text)
164 +
165 Online Inference
166 ----------------
167