@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / dc4aea677a
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 10:45 UTC

[Doc] Fix VLM prompt placeholder sample bug (#9170)

Jiangtao Hu committed Oct 9, 2024 at 16:59 UTC dc4aea677ab0520d91ff4979e80340cb5a090095
1 file changed +7 -7
docs/source/models/vlm.rst
+7 -7
@@ -25,7 +25,7 @@ The :class:`~vllm.LLM` class can be instantiated in much the same way as languag
25 To pass an image to the model, note the following in :class:`vllm.inputs.PromptType`:
26
27 * ``prompt``: The prompt should follow the format that is documented on HuggingFace.
28 -* ``multi_modal_data``: This is a dictionary that follows the schema defined in :class:`vllm.multimodal.MultiModalDataDict`.
28 +* ``multi_modal_data``: This is a dictionary that follows the schema defined in :class:`vllm.multimodal.MultiModalDataDict`.
29
30 .. code-block:: python
31
@@ -34,7 +34,7 @@ To pass an image to the model, note the following in :class:`vllm.inputs.PromptT
34
35 # Load the image using PIL.Image
36 image = PIL.Image.open(...)
37 -
37 +
38 # Single prompt inference
39 outputs = llm.generate({
40 "prompt": prompt,
@@ -68,7 +68,7 @@ To pass an image to the model, note the following in :class:`vllm.inputs.PromptT
68 "prompt": prompt,
69 "multi_modal_data": mm_data,
70 })
71 -
71 +
72 for o in outputs:
73 generated_text = o.outputs[0].text
74 print(generated_text)
@@ -116,7 +116,7 @@ Instead of passing in a single image, you can pass in a list of images.
116 .. code-block:: python
117
118 # Refer to the HuggingFace repo for the correct format to use
119 - prompt = "<|user|>\n<image_1>\n<image_2>\nWhat is the content of each image?<|end|>\n<|assistant|>\n"
119 + prompt = "<|user|>\n<|image_1|>\n<|image_2|>\nWhat is the content of each image?<|end|>\n<|assistant|>\n"
120
121 # Load the images using PIL.Image
122 image1 = PIL.Image.open(...)
@@ -135,11 +135,11 @@ Instead of passing in a single image, you can pass in a list of images.
135
136 A code example can be found in `examples/offline_inference_vision_language_multi_image.py <https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_vision_language_multi_image.py>`_.
137
138 -Multi-image input can be extended to perform video captioning. We show this with `Qwen2-VL <https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct>`_ as it supports videos:
138 +Multi-image input can be extended to perform video captioning. We show this with `Qwen2-VL <https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct>`_ as it supports videos:
139
140 .. code-block:: python
141
142 - # Specify the maximum number of frames per video to be 4. This can be changed.
142 + # Specify the maximum number of frames per video to be 4. This can be changed.
143 llm = LLM("Qwen/Qwen2-VL-2B-Instruct", limit_mm_per_prompt={"image": 4})
144
145 # Create the request payload.
@@ -157,7 +157,7 @@ Multi-image input can be extended to perform video captioning. We show this with
157
158 # Perform inference and log output.
159 outputs = llm.chat([message])
160 -
160 +
161 for o in outputs:
162 generated_text = o.outputs[0].text
163 print(generated_text)