@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 856c990041
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 23:15 UTC

[Docs] Add Docs on Limitations of VLM Support (#5383)

Roger Wang committed Jun 10, 2024 at 09:53 UTC 856c990041bf6cf4b2397401d4b18531382ecb50
2 files changed +9 -1
docs/source/conf.py
+1
@@ -92,6 +92,7 @@ autodoc_mock_imports = [
92 "vllm._C",
93 "PIL",
94 "numpy",
95 + 'triton'
96 "tqdm",
97 "tensorizer",
98 ]
docs/source/models/vlm.rst
+8 -1
@@ -16,6 +16,13 @@ The following :ref:`engine arguments <engine_args>` are specific to VLMs:
16 :prog: -m vllm.entrypoints.openai.api_server
17 :nodefaultconst:
18
19 +.. important::
20 + Currently, the support for vision language models on vLLM has the following limitations:
21 +
22 + * Only single image input is supported per text prompt.
23 + * Dynamic ``image_input_shape`` is not supported: the input image will be resized to the static ``image_input_shape``. This means model output might not exactly match the huggingface implementation.
24 + We are continuously improving user & developer experience for VLMs. Please raise an issue on GitHub if you have any feedback or feature requests.
25 +
26 Offline Batched Inference
27 -------------------------
28
@@ -31,7 +38,7 @@ To initialize a VLM, the aforementioned arguments must be passed to the ``LLM``
38 image_feature_size=576,
39 )
40
34 -For now, we only support a single image per text prompt. To pass an image to the model, note the following in :class:`vllm.inputs.PromptStrictInputs`:
41 +To pass an image to the model, note the following in :class:`vllm.inputs.PromptStrictInputs`:
42
43 * ``prompt``: The prompt should have a number of ``<image>`` tokens equal to ``image_feature_size``.
44 * ``multi_modal_data``: This should be an instance of :class:`~vllm.multimodal.image.ImagePixelData` or :class:`~vllm.multimodal.image.ImageFeatureData`.