A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 23:15 UTC
[Docs] Add Docs on Limitations of VLM Support (#5383)
Roger Wang committed
Jun 10, 2024 at 09:53 UTC
856c990041bf6cf4b2397401d4b18531382ecb50
2 files changed
+9
-1
docs/source/conf.py
+1
@@ -92,6 +92,7 @@ autodoc_mock_imports = [
92
"vllm._C",
93
"PIL",
94
"numpy",
95
+ 'triton'
96
"tqdm",
97
"tensorizer",
98
]
docs/source/models/vlm.rst
+8
-1
@@ -16,6 +16,13 @@ The following :ref:`engine arguments <engine_args>` are specific to VLMs:
16
:prog: -m vllm.entrypoints.openai.api_server
17
:nodefaultconst:
18
19
+.. important::
20
+ Currently, the support for vision language models on vLLM has the following limitations:
21
+
22
+ * Only single image input is supported per text prompt.
23
+ * Dynamic ``image_input_shape`` is not supported: the input image will be resized to the static ``image_input_shape``. This means model output might not exactly match the huggingface implementation.
24
+ We are continuously improving user & developer experience for VLMs. Please raise an issue on GitHub if you have any feedback or feature requests.
25
+
26
Offline Batched Inference
27
-------------------------
28
@@ -31,7 +38,7 @@ To initialize a VLM, the aforementioned arguments must be passed to the ``LLM``
38
image_feature_size=576,
39
)
40
34
-For now, we only support a single image per text prompt. To pass an image to the model, note the following in :class:`vllm.inputs.PromptStrictInputs`:
41
+To pass an image to the model, note the following in :class:`vllm.inputs.PromptStrictInputs`:
42
43
* ``prompt``: The prompt should have a number of ``<image>`` tokens equal to ``image_feature_size``.
44
* ``multi_modal_data``: This should be an instance of :class:`~vllm.multimodal.image.ImagePixelData` or :class:`~vllm.multimodal.image.ImageFeatureData`.