A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 23:15 UTC
[Doc] Add notice about breaking changes to VLMs (#5818)
Cyrus Leung committed
Jun 25, 2024 at 16:25 UTC
f23871e9eead900d6146961ca894f5bc91f30f5e
1 file changed
+13
docs/source/models/vlm.rst
+13
@@ -5,6 +5,9 @@ Using VLMs
5
6
vLLM provides experimental support for Vision Language Models (VLMs). This document shows you how to run and serve these models using vLLM.
7
8
+.. important::
9
+ We are actively iterating on VLM support. Expect breaking changes to VLM usage and development in upcoming releases without prior deprecation.
10
+
11
Engine Arguments
12
----------------
13
@@ -39,6 +42,10 @@ To initialize a VLM, the aforementioned arguments must be passed to the ``LLM``
42
image_feature_size=576,
43
)
44
45
+.. important::
46
+ We will remove most of the vision-specific arguments in a future release as they can be inferred from the HuggingFace configuration.
47
+
48
+
49
To pass an image to the model, note the following in :class:`vllm.inputs.PromptStrictInputs`:
50
51
* ``prompt``: The prompt should have a number of ``<image>`` tokens equal to ``image_feature_size``.
@@ -63,6 +70,9 @@ To pass an image to the model, note the following in :class:`vllm.inputs.PromptS
70
71
A code example can be found in `examples/llava_example.py <https://github.com/vllm-project/vllm/blob/main/examples/llava_example.py>`_.
72
73
+.. important::
74
+ We will remove the need to format image tokens in a future release. Afterwards, the input text will follow the same format as that for the original HuggingFace model.
75
+
76
Online OpenAI Vision API Compatible Inference
77
----------------------------------------------
78
@@ -89,6 +99,9 @@ Below is an example on how to launch the same ``llava-hf/llava-1.5-7b-hf`` with
99
--image-feature-size 576 \
100
--chat-template template_llava.jinja
101
102
+.. important::
103
+ We will remove most of the vision-specific arguments in a future release as they can be inferred from the HuggingFace configuration.
104
+
105
To consume the server, you can use the OpenAI client like in the example below:
106
107
.. code-block:: python