@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / f23871e9ee
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 23:15 UTC

[Doc] Add notice about breaking changes to VLMs (#5818)

Cyrus Leung committed Jun 25, 2024 at 16:25 UTC f23871e9eead900d6146961ca894f5bc91f30f5e
1 file changed +13
docs/source/models/vlm.rst
+13
@@ -5,6 +5,9 @@ Using VLMs
5
6 vLLM provides experimental support for Vision Language Models (VLMs). This document shows you how to run and serve these models using vLLM.
7
8 +.. important::
9 + We are actively iterating on VLM support. Expect breaking changes to VLM usage and development in upcoming releases without prior deprecation.
10 +
11 Engine Arguments
12 ----------------
13
@@ -39,6 +42,10 @@ To initialize a VLM, the aforementioned arguments must be passed to the ``LLM``
42 image_feature_size=576,
43 )
44
45 +.. important::
46 + We will remove most of the vision-specific arguments in a future release as they can be inferred from the HuggingFace configuration.
47 +
48 +
49 To pass an image to the model, note the following in :class:`vllm.inputs.PromptStrictInputs`:
50
51 * ``prompt``: The prompt should have a number of ``<image>`` tokens equal to ``image_feature_size``.
@@ -63,6 +70,9 @@ To pass an image to the model, note the following in :class:`vllm.inputs.PromptS
70
71 A code example can be found in `examples/llava_example.py <https://github.com/vllm-project/vllm/blob/main/examples/llava_example.py>`_.
72
73 +.. important::
74 + We will remove the need to format image tokens in a future release. Afterwards, the input text will follow the same format as that for the original HuggingFace model.
75 +
76 Online OpenAI Vision API Compatible Inference
77 ----------------------------------------------
78
@@ -89,6 +99,9 @@ Below is an example on how to launch the same ``llava-hf/llava-1.5-7b-hf`` with
99 --image-feature-size 576 \
100 --chat-template template_llava.jinja
101
102 +.. important::
103 + We will remove most of the vision-specific arguments in a future release as they can be inferred from the HuggingFace configuration.
104 +
105 To consume the server, you can use the OpenAI client like in the example below:
106
107 .. code-block:: python