A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 05:45 UTC
[Doc] Create a new "Usage" section (#10827)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Cyrus Leung committed
Dec 5, 2024 at 11:19 UTC
aa39a8e17537f9127b3da65dba6b33067bfd2f78
25 files changed
+218
-125
docs/source/design/multimodal/multimodal_index.rst
+1
-4
@@ -7,7 +7,7 @@ Multi-Modality
7
8
vLLM provides experimental support for multi-modal models through the :mod:`vllm.multimodal` package.
9
10
-Multi-modal inputs can be passed alongside text and token prompts to :ref:`supported models <supported_vlms>`
10
+Multi-modal inputs can be passed alongside text and token prompts to :ref:`supported models <supported_mm_models>`
11
via the ``multi_modal_data`` field in :class:`vllm.inputs.PromptType`.
12
13
Currently, vLLM only has built-in support for image data. You can extend vLLM to process additional modalities
@@ -15,9 +15,6 @@ by following :ref:`this guide <adding_multimodal_plugin>`.
15
16
Looking to add your own multi-modal model? Please follow the instructions listed :ref:`here <enabling_multimodal_inputs>`.
17
18
-..
19
- TODO: Add usage of --limit-mm-per-prompt when multi-image input is officially supported
20
-
18
Guides
19
++++++
20
docs/source/index.rst
+15
-10
@@ -85,12 +85,8 @@ Documentation
85
serving/deploying_with_nginx
86
serving/distributed_serving
87
serving/metrics
88
- serving/env_vars
89
- serving/usage_stats
88
serving/integrations
89
serving/tensorizer
92
- serving/compatibility_matrix
93
- serving/faq
90
91
.. toctree::
92
:maxdepth: 1
@@ -99,12 +95,21 @@ Documentation
95
models/supported_models
96
models/adding_model
97
models/enabling_multimodal_inputs
102
- models/engine_args
103
- models/lora
104
- models/vlm
105
- models/structured_outputs
106
- models/spec_decode
107
- models/performance
98
+
99
+.. toctree::
100
+ :maxdepth: 1
101
+ :caption: Usage
102
+
103
+ usage/lora
104
+ usage/multimodal_inputs
105
+ usage/structured_outputs
106
+ usage/spec_decode
107
+ usage/compatibility_matrix
108
+ usage/performance
109
+ usage/faq
110
+ usage/engine_args
111
+ usage/env_vars
112
+ usage/usage_stats
113
114
.. toctree::
115
:maxdepth: 1
docs/source/models/enabling_multimodal_inputs.rst
+1
-1
@@ -3,7 +3,7 @@
3
Enabling Multimodal Inputs
4
==========================
5
6
-This document walks you through the steps to extend a vLLM model so that it accepts :ref:`multi-modal <multi_modality>` inputs.
6
+This document walks you through the steps to extend a vLLM model so that it accepts :ref:`multi-modal inputs <multimodal_inputs>`.
7
8
.. seealso::
9
:ref:`adding_a_new_model`
docs/source/models/supported_models.rst
+17
-2
@@ -471,6 +471,8 @@ Sentence Pair Scoring
471
.. note::
472
These models are supported in both offline and online inference via Score API.
473
474
+.. _supported_mm_models:
475
+
476
Multimodal Language Models
477
^^^^^^^^^^^^^^^^^^^^^^^^^^
478
@@ -489,8 +491,6 @@ On the other hand, modalities separated by :code:`/` are mutually exclusive.
491
492
- e.g.: :code:`T / I` means that the model supports text-only and image-only inputs, but not text-with-image inputs.
493
492
-.. _supported_vlms:
493
-
494
Text Generation
495
---------------
496
@@ -646,6 +646,21 @@ Text Generation
646
| :sup:`E` Pre-computed embeddings can be inputted for this modality.
647
| :sup:`+` Multiple items can be inputted per text prompt for this modality.
648
649
+.. important::
650
+ To enable multiple multi-modal items per text prompt, you have to set :code:`limit_mm_per_prompt` (offline inference)
651
+ or :code:`--limit-mm-per-prompt` (online inference). For example, to enable passing up to 4 images per text prompt:
652
+
653
+ .. code-block:: python
654
+
655
+ llm = LLM(
656
+ model="Qwen/Qwen2-VL-7B-Instruct",
657
+ limit_mm_per_prompt={"image": 4},
658
+ )
659
+
660
+ .. code-block:: bash
661
+
662
+ vllm serve Qwen/Qwen2-VL-7B-Instruct --limit-mm-per-prompt image=4
663
+
664
.. note::
665
vLLM currently only supports adding LoRA to the language backbone of multimodal models.
666
docs/source/serving/openai_compatible_server.md
+2
-2
@@ -32,7 +32,7 @@ We currently support the following OpenAI APIs:
32
- [Completions API](https://platform.openai.com/docs/api-reference/completions)
33
- *Note: `suffix` parameter is not supported.*
34
- [Chat Completions API](https://platform.openai.com/docs/api-reference/chat)
35
- - [Vision](https://platform.openai.com/docs/guides/vision)-related parameters are supported; see [Using VLMs](../models/vlm.rst).
35
+ - [Vision](https://platform.openai.com/docs/guides/vision)-related parameters are supported; see [Multimodal Inputs](../usage/multimodal_inputs.rst).
36
- *Note: `image_url.detail` parameter is not supported.*
37
- We also support `audio_url` content type for audio files.
38
- Refer to [vllm.entrypoints.chat_utils](https://github.com/vllm-project/vllm/tree/main/vllm/entrypoints/chat_utils.py) for the exact schema.
@@ -41,7 +41,7 @@ We currently support the following OpenAI APIs:
41
- [Embeddings API](https://platform.openai.com/docs/api-reference/embeddings)
42
- Instead of `inputs`, you can pass in a list of `messages` (same schema as Chat Completions API),
43
which will be treated as a single prompt to the model according to its chat template.
44
- - This enables multi-modal inputs to be passed to embedding models, see [Using VLMs](../models/vlm.rst).
44
+ - This enables multi-modal inputs to be passed to embedding models, see [this page](../usage/multimodal_inputs.rst) for details.
45
- *Note: You should run `vllm serve` with `--task embedding` to ensure that the model is being run in embedding mode.*
46
47
## Score API for Cross Encoder Models
docs/source/usage/compatibility_matrix.rst
renamed
docs/source/usage/engine_args.rst
renamed
docs/source/usage/env_vars.rst
renamed
docs/source/usage/faq.rst
renamed
+2
@@ -1,3 +1,5 @@
1
+.. _faq:
2
+
3
Frequently Asked Questions
4
===========================
5
docs/source/usage/lora.rst
renamed
+2
-2
@@ -1,7 +1,7 @@
1
.. _lora:
2
3
-Using LoRA adapters
4
-===================
3
+LoRA Adapters
4
+=============
5
6
This document shows you how to use `LoRA adapters <https://arxiv.org/abs/2106.09685>`_ with vLLM on top of a base model.
7
docs/source/usage/multimodal_inputs.rst
renamed
+161
-87
@@ -1,34 +1,31 @@
1
-.. _vlm:
1
+.. _multimodal_inputs:
2
3
-Using VLMs
4
-==========
3
+Multimodal Inputs
4
+=================
5
6
-vLLM provides experimental support for Vision Language Models (VLMs). See the :ref:`list of supported VLMs here <supported_vlms>`.
7
-This document shows you how to run and serve these models using vLLM.
6
+This page teaches you how to pass multi-modal inputs to :ref:`multi-modal models <supported_mm_models>` in vLLM.
7
8
.. note::
10
- We are actively iterating on VLM support. See `this RFC <https://github.com/vllm-project/vllm/issues/4194>`_ for upcoming changes,
9
+ We are actively iterating on multi-modal support. See `this RFC <https://github.com/vllm-project/vllm/issues/4194>`_ for upcoming changes,
10
and `open an issue on GitHub <https://github.com/vllm-project/vllm/issues/new/choose>`_ if you have any feedback or feature requests.
11
12
Offline Inference
13
-----------------
14
16
-Single-image input
17
-^^^^^^^^^^^^^^^^^^
18
-
19
-The :class:`~vllm.LLM` class can be instantiated in much the same way as language-only models.
20
-
21
-.. code-block:: python
22
-
23
- llm = LLM(model="llava-hf/llava-1.5-7b-hf")
24
-
25
-To pass an image to the model, note the following in :class:`vllm.inputs.PromptType`:
15
+To input multi-modal data, follow this schema in :class:`vllm.inputs.PromptType`:
16
17
* ``prompt``: The prompt should follow the format that is documented on HuggingFace.
18
* ``multi_modal_data``: This is a dictionary that follows the schema defined in :class:`vllm.multimodal.MultiModalDataDict`.
19
20
+Image
21
+^^^^^
22
+
23
+You can pass a single image to the :code:`'image'` field of the multi-modal dictionary, as shown in the following examples:
24
+
25
.. code-block:: python
26
27
+ llm = LLM(model="llava-hf/llava-1.5-7b-hf")
28
+
29
# Refer to the HuggingFace repo for the correct format to use
30
prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:"
31
@@ -41,41 +38,6 @@ To pass an image to the model, note the following in :class:`vllm.inputs.PromptT
38
"multi_modal_data": {"image": image},
39
})
40
44
- for o in outputs:
45
- generated_text = o.outputs[0].text
46
- print(generated_text)
47
-
48
- # Inference with image embeddings as input
49
- image_embeds = torch.load(...) # torch.Tensor of shape (1, image_feature_size, hidden_size of LM)
50
- outputs = llm.generate({
51
- "prompt": prompt,
52
- "multi_modal_data": {"image": image_embeds},
53
- })
54
-
55
- for o in outputs:
56
- generated_text = o.outputs[0].text
57
- print(generated_text)
58
-
59
- # Inference with image embeddings as input with additional parameters
60
- # Specifically, we are conducting a trial run of Qwen2VL and MiniCPM-V with the new input format, which utilizes additional parameters.
61
- mm_data = {}
62
-
63
- image_embeds = torch.load(...) # torch.Tensor of shape (num_images, image_feature_size, hidden_size of LM)
64
- # For Qwen2VL, image_grid_thw is needed to calculate positional encoding.
65
- mm_data['image'] = {
66
- "image_embeds": image_embeds,
67
- "image_grid_thw": torch.load(...) # torch.Tensor of shape (1, 3),
68
- }
69
- # For MiniCPM-V, image_size_list is needed to calculate details of the sliced image.
70
- mm_data['image'] = {
71
- "image_embeds": image_embeds,
72
- "image_size_list": [image.size] # list of image sizes
73
- }
74
- outputs = llm.generate({
75
- "prompt": prompt,
76
- "multi_modal_data": mm_data,
77
- })
78
-
41
for o in outputs:
42
generated_text = o.outputs[0].text
43
print(generated_text)
@@ -102,12 +64,7 @@ To pass an image to the model, note the following in :class:`vllm.inputs.PromptT
64
65
A code example can be found in `examples/offline_inference_vision_language.py <https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_vision_language.py>`_.
66
105
-Multi-image input
106
-^^^^^^^^^^^^^^^^^
107
-
108
-Multi-image input is only supported for a subset of VLMs, as shown :ref:`here <supported_vlms>`.
109
-
110
-To enable multiple multi-modal items per text prompt, you have to set ``limit_mm_per_prompt`` for the :class:`~vllm.LLM` class.
67
+To substitute multiple images inside the same text prompt, you can pass in a list of images instead:
68
69
.. code-block:: python
70
@@ -118,10 +75,6 @@ To enable multiple multi-modal items per text prompt, you have to set ``limit_mm
75
limit_mm_per_prompt={"image": 2}, # The maximum number to accept
76
)
77
121
-Instead of passing in a single image, you can pass in a list of images.
122
-
123
-.. code-block:: python
124
-
78
# Refer to the HuggingFace repo for the correct format to use
79
prompt = "<|user|>\n<|image_1|>\n<|image_2|>\nWhat is the content of each image?<|end|>\n<|assistant|>\n"
80
@@ -169,30 +122,114 @@ Multi-image input can be extended to perform video captioning. We show this with
122
generated_text = o.outputs[0].text
123
print(generated_text)
124
125
+Video
126
+^^^^^
127
+
128
+You can pass a list of NumPy arrays directly to the :code:`'video'` field of the multi-modal dictionary
129
+instead of using multi-image input.
130
+
131
+Please refer to `examples/offline_inference_vision_language.py <https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_vision_language.py>`_ for more details.
132
+
133
+Audio
134
+^^^^^
135
+
136
+You can pass a tuple :code:`(array, sampling_rate)` to the :code:`'audio'` field of the multi-modal dictionary.
137
+
138
+Please refer to `examples/offline_inference_audio_language.py <https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_audio_language.py>`_ for more details.
139
+
140
+Embedding
141
+^^^^^^^^^
142
+
143
+To input pre-computed embeddings belonging to a data type (i.e. image, video, or audio) directly to the language model,
144
+pass a tensor of shape :code:`(num_items, feature_size, hidden_size of LM)` to the corresponding field of the multi-modal dictionary.
145
+
146
+.. code-block:: python
147
+
148
+ # Inference with image embeddings as input
149
+ llm = LLM(model="llava-hf/llava-1.5-7b-hf")
150
+
151
+ # Refer to the HuggingFace repo for the correct format to use
152
+ prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:"
153
+
154
+ # Embeddings for single image
155
+ # torch.Tensor of shape (1, image_feature_size, hidden_size of LM)
156
+ image_embeds = torch.load(...)
157
+
158
+ outputs = llm.generate({
159
+ "prompt": prompt,
160
+ "multi_modal_data": {"image": image_embeds},
161
+ })
162
+
163
+ for o in outputs:
164
+ generated_text = o.outputs[0].text
165
+ print(generated_text)
166
+
167
+For Qwen2-VL and MiniCPM-V, we accept additional parameters alongside the embeddings:
168
+
169
+.. code-block:: python
170
+
171
+ # Construct the prompt based on your model
172
+ prompt = ...
173
+
174
+ # Embeddings for multiple images
175
+ # torch.Tensor of shape (num_images, image_feature_size, hidden_size of LM)
176
+ image_embeds = torch.load(...)
177
+
178
+ # Qwen2-VL
179
+ llm = LLM("Qwen/Qwen2-VL-2B-Instruct", limit_mm_per_prompt={"image": 4})
180
+ mm_data = {
181
+ "image": {
182
+ "image_embeds": image_embeds,
183
+ # image_grid_thw is needed to calculate positional encoding.
184
+ "image_grid_thw": torch.load(...), # torch.Tensor of shape (1, 3),
185
+ }
186
+ }
187
+
188
+ # MiniCPM-V
189
+ llm = LLM("openbmb/MiniCPM-V-2_6", trust_remote_code=True, limit_mm_per_prompt={"image": 4})
190
+ mm_data = {
191
+ "image": {
192
+ "image_embeds": image_embeds,
193
+ # image_size_list is needed to calculate details of the sliced image.
194
+ "image_size_list": [image.size for image in images], # list of image sizes
195
+ }
196
+ }
197
+
198
+ outputs = llm.generate({
199
+ "prompt": prompt,
200
+ "multi_modal_data": mm_data,
201
+ })
202
+
203
+ for o in outputs:
204
+ generated_text = o.outputs[0].text
205
+ print(generated_text)
206
+
207
Online Inference
208
----------------
209
175
-OpenAI Vision API
176
-^^^^^^^^^^^^^^^^^
210
+Our OpenAI-compatible server accepts multi-modal data via the `Chat Completions API <https://platform.openai.com/docs/api-reference/chat>`_.
211
+
212
+.. important::
213
+ A chat template is **required** to use Chat Completions API.
214
+
215
+ Although most models come with a chat template, for others you have to define one yourself.
216
+ The chat template can be inferred based on the documentation on the model's HuggingFace repo.
217
+ For example, LLaVA-1.5 (``llava-hf/llava-1.5-7b-hf``) requires a chat template that can be found `here <https://github.com/vllm-project/vllm/blob/main/examples/template_llava.jinja>`__.
218
+
219
+Image
220
+^^^^^
221
178
-You can serve vision language models with vLLM's HTTP server that is compatible with `OpenAI Vision API <https://platform.openai.com/docs/guides/vision>`_.
222
+Image input is supported according to `OpenAI Vision API <https://platform.openai.com/docs/guides/vision>`_.
223
+Here is a simple example using Phi-3.5-Vision.
224
180
-Below is an example on how to launch the same ``microsoft/Phi-3.5-vision-instruct`` with vLLM's OpenAI-compatible API server.
225
+First, launch the OpenAI-compatible server:
226
227
.. code-block:: bash
228
229
vllm serve microsoft/Phi-3.5-vision-instruct --task generate \
230
--trust-remote-code --max-model-len 4096 --limit-mm-per-prompt image=2
231
187
-.. important::
188
- Since OpenAI Vision API is based on `Chat Completions API <https://platform.openai.com/docs/api-reference/chat>`_,
189
- a chat template is **required** to launch the API server.
190
-
191
- Although Phi-3.5-Vision comes with a chat template, for other models you may have to provide one if the model's tokenizer does not come with it.
192
- The chat template can be inferred based on the documentation on the model's HuggingFace repo.
193
- For example, LLaVA-1.5 (``llava-hf/llava-1.5-7b-hf``) requires a chat template that can be found `here <https://github.com/vllm-project/vllm/blob/main/examples/template_llava.jinja>`_.
194
-
195
-To consume the server, you can use the OpenAI client like in the example below:
232
+Then, you can use the OpenAI client as follows:
233
234
.. code-block:: python
235
@@ -252,22 +289,59 @@ A full code example can be found in `examples/openai_chat_completion_client_for_
289
290
.. note::
291
255
- By default, the timeout for fetching images through http url is ``5`` seconds. You can override this by setting the environment variable:
292
+ By default, the timeout for fetching images through HTTP URL is ``5`` seconds.
293
+ You can override this by setting the environment variable:
294
295
.. code-block:: console
296
297
$ export VLLM_IMAGE_FETCH_TIMEOUT=<timeout>
298
261
-Chat Embeddings API
262
-^^^^^^^^^^^^^^^^^^^
299
+Video
300
+^^^^^
301
+
302
+Instead of :code:`image_url`, you can pass a video file via :code:`video_url`.
303
+
304
+You can use `these tests <https://github.com/vllm-project/vllm/blob/main/tests/entrypoints/openai/test_video.py>`_ as reference.
305
+
306
+.. note::
307
+
308
+ By default, the timeout for fetching videos through HTTP URL url is ``30`` seconds.
309
+ You can override this by setting the environment variable:
310
+
311
+ .. code-block:: console
312
+
313
+ $ export VLLM_VIDEO_FETCH_TIMEOUT=<timeout>
314
264
-vLLM's Chat Embeddings API is a superset of OpenAI's `Embeddings API <https://platform.openai.com/docs/api-reference/embeddings>`_,
265
-where a list of ``messages`` can be passed instead of batched ``inputs``. This enables multi-modal inputs to be passed to embedding models.
315
+Audio
316
+^^^^^
317
+
318
+Instead of :code:`image_url`, you can pass an audio file via :code:`audio_url`.
319
+
320
+A full code example can be found in `examples/openai_chat_completion_client_for_multimodal.py <https://github.com/vllm-project/vllm/blob/main/examples/openai_chat_completion_client_for_multimodal.py>`_.
321
+
322
+.. note::
323
+
324
+ By default, the timeout for fetching audios through HTTP URL is ``10`` seconds.
325
+ You can override this by setting the environment variable:
326
+
327
+ .. code-block:: console
328
+
329
+ $ export VLLM_AUDIO_FETCH_TIMEOUT=<timeout>
330
+
331
+Embedding
332
+^^^^^^^^^
333
+
334
+vLLM's Embeddings API is a superset of OpenAI's `Embeddings API <https://platform.openai.com/docs/api-reference/embeddings>`_,
335
+where a list of chat ``messages`` can be passed instead of batched ``inputs``. This enables multi-modal inputs to be passed to embedding models.
336
337
.. tip::
338
The schema of ``messages`` is exactly the same as in Chat Completions API.
339
+ You can refer to the above tutorials for more details on how to pass each type of multi-modal data.
340
270
-In this example, we will serve the ``TIGER-Lab/VLM2Vec-Full`` model.
341
+Usually, embedding models do not expect chat-based input, so we need to use a custom chat template to format the text and images.
342
+Refer to the examples below for illustration.
343
+
344
+Here is an end-to-end example using VLM2Vec. To serve the model:
345
346
.. code-block:: bash
347
@@ -279,10 +353,8 @@ In this example, we will serve the ``TIGER-Lab/VLM2Vec-Full`` model.
353
Since VLM2Vec has the same model architecture as Phi-3.5-Vision, we have to explicitly pass ``--task embedding``
354
to run this model in embedding mode instead of text generation mode.
355
282
-.. important::
283
-
284
- VLM2Vec does not expect chat-based input. We use a `custom chat template <https://github.com/vllm-project/vllm/blob/main/examples/template_vlm2vec.jinja>`_
285
- to combine the text and images together.
356
+ The custom chat template is completely different from the original one for this model,
357
+ and can be found `here <https://github.com/vllm-project/vllm/blob/main/examples/template_vlm2vec.jinja>`__.
358
359
Since the request schema is not defined by OpenAI client, we post a request to the server using the lower-level ``requests`` library:
360
@@ -310,7 +382,7 @@ Since the request schema is not defined by OpenAI client, we post a request to t
382
response_json = response.json()
383
print("Embedding output:", response_json["data"][0]["embedding"])
384
313
-Here is an example for serving the ``MrLight/dse-qwen2-2b-mrl-v1`` model.
385
+Below is another example, this time using the ``MrLight/dse-qwen2-2b-mrl-v1`` model.
386
387
.. code-block:: bash
388
@@ -319,8 +391,10 @@ Here is an example for serving the ``MrLight/dse-qwen2-2b-mrl-v1`` model.
391
392
.. important::
393
322
- Like with VLM2Vec, we have to explicitly pass ``--task embedding``. Additionally, ``MrLight/dse-qwen2-2b-mrl-v1`` requires an EOS token for embeddings,
323
- which is handled by the jinja template.
394
+ Like with VLM2Vec, we have to explicitly pass ``--task embedding``.
395
+
396
+ Additionally, ``MrLight/dse-qwen2-2b-mrl-v1`` requires an EOS token for embeddings, which is handled
397
+ by `this custom chat template <https://github.com/vllm-project/vllm/blob/main/examples/template_dse_qwen2_vl.jinja>`__.
398
399
.. important::
400
docs/source/usage/performance.rst
renamed
docs/source/usage/spec_decode.rst
renamed
+4
-4
@@ -1,7 +1,7 @@
1
.. _spec_decode:
2
3
-Speculative decoding in vLLM
4
-============================
3
+Speculative decoding
4
+====================
5
6
.. warning::
7
Please note that speculative decoding in vLLM is not yet optimized and does
@@ -182,7 +182,7 @@ speculative decoding, breaking down the guarantees into three key areas:
182
3. **vLLM Logprob Stability**
183
- vLLM does not currently guarantee stable token log probabilities (logprobs). This can result in different outputs for the
184
same request across runs. For more details, see the FAQ section
185
- titled *Can the output of a prompt vary across runs in vLLM?* in the `FAQs <../serving/faq>`_.
185
+ titled *Can the output of a prompt vary across runs in vLLM?* in the :ref:`FAQs <faq>`.
186
187
188
**Conclusion**
@@ -197,7 +197,7 @@ can occur due to following factors:
197
198
**Mitigation Strategies**
199
200
-For mitigation strategies, please refer to the FAQ entry *Can the output of a prompt vary across runs in vLLM?* in the `FAQs <../serving/faq>`_.
200
+For mitigation strategies, please refer to the FAQ entry *Can the output of a prompt vary across runs in vLLM?* in the :ref:`FAQs <faq>`.
201
202
Resources for vLLM contributors
203
-------------------------------
docs/source/usage/structured_outputs.rst
renamed
docs/source/usage/usage_stats.md
renamed
vllm/attention/backends/rocm_flash_attn.py
+1
-1
@@ -430,7 +430,7 @@ class ROCmFlashAttentionImpl(AttentionImpl):
430
Returns:
431
shape = [num_tokens, num_heads * head_size]
432
"""
433
- # Reminder: Please update docs/source/serving/compatibility_matrix.rst
433
+ # Reminder: Please update docs/source/usage/compatibility_matrix.rst
434
# If the feature combo become valid
435
if attn_type != AttentionType.DECODER:
436
raise NotImplementedError("Encoder self-attention and "
vllm/config.py
+4
-4
@@ -509,7 +509,7 @@ class ModelConfig:
509
self.use_async_output_proc = False
510
return
511
512
- # Reminder: Please update docs/source/serving/compatibility_matrix.rst
512
+ # Reminder: Please update docs/source/usage/compatibility_matrix.rst
513
# If the feature combo become valid
514
if device_config.device_type not in ("cuda", "tpu", "xpu", "hpu"):
515
logger.warning(
@@ -525,7 +525,7 @@ class ModelConfig:
525
self.use_async_output_proc = False
526
return
527
528
- # Reminder: Please update docs/source/serving/compatibility_matrix.rst
528
+ # Reminder: Please update docs/source/usage/compatibility_matrix.rst
529
# If the feature combo become valid
530
if device_config.device_type == "cuda" and self.enforce_eager:
531
logger.warning(
@@ -540,7 +540,7 @@ class ModelConfig:
540
if self.task == "embedding":
541
self.use_async_output_proc = False
542
543
- # Reminder: Please update docs/source/serving/compatibility_matrix.rst
543
+ # Reminder: Please update docs/source/usage/compatibility_matrix.rst
544
# If the feature combo become valid
545
if speculative_config:
546
logger.warning("Async output processing is not supported with"
@@ -1704,7 +1704,7 @@ class LoRAConfig:
1704
model_config.quantization)
1705
1706
def verify_with_scheduler_config(self, scheduler_config: SchedulerConfig):
1707
- # Reminder: Please update docs/source/serving/compatibility_matrix.rst
1707
+ # Reminder: Please update docs/source/usage/compatibility_matrix.rst
1708
# If the feature combo become valid
1709
if scheduler_config.chunked_prefill_enabled:
1710
raise ValueError("LoRA is not supported with chunked prefill yet.")
vllm/engine/arg_utils.py
+1
-1
@@ -1111,7 +1111,7 @@ class EngineArgs:
1111
disable_logprobs=self.disable_logprobs_during_spec_decoding,
1112
)
1113
1114
- # Reminder: Please update docs/source/serving/compatibility_matrix.rst
1114
+ # Reminder: Please update docs/source/usage/compatibility_matrix.rst
1115
# If the feature combo become valid
1116
if self.num_scheduler_steps > 1:
1117
if speculative_config is not None:
vllm/engine/output_processor/multi_step.py
+1
-1
@@ -65,7 +65,7 @@ class MultiStepOutputProcessor(SequenceGroupOutputProcessor):
65
@staticmethod
66
@functools.lru_cache
67
def _log_prompt_logprob_unsupported_warning_once():
68
- # Reminder: Please update docs/source/serving/compatibility_matrix.rst
68
+ # Reminder: Please update docs/source/usage/compatibility_matrix.rst
69
# If the feature combo become valid
70
logger.warning(
71
"Prompt logprob is not supported by multi step workers. "
vllm/executor/cpu_executor.py
+1
-1
@@ -23,7 +23,7 @@ class CPUExecutor(ExecutorBase):
23
24
def _init_executor(self) -> None:
25
assert self.device_config.device_type == "cpu"
26
- # Reminder: Please update docs/source/serving/compatibility_matrix.rst
26
+ # Reminder: Please update docs/source/usage/compatibility_matrix.rst
27
# If the feature combo become valid
28
assert self.lora_config is None, "cpu backend doesn't support LoRA"
29
vllm/platforms/cpu.py
+1
-1
@@ -46,7 +46,7 @@ class CpuPlatform(Platform):
46
import vllm.envs as envs
47
from vllm.utils import GiB_bytes
48
model_config = vllm_config.model_config
49
- # Reminder: Please update docs/source/serving/compatibility_matrix.rst
49
+ # Reminder: Please update docs/source/usage/compatibility_matrix.rst
50
# If the feature combo become valid
51
if not model_config.enforce_eager:
52
logger.warning(
vllm/spec_decode/spec_decode_worker.py
+1
-1
@@ -104,7 +104,7 @@ def create_spec_worker(*args, **kwargs) -> "SpecDecodeWorker":
104
return spec_decode_worker
105
106
107
-# Reminder: Please update docs/source/serving/compatibility_matrix.rst
107
+# Reminder: Please update docs/source/usage/compatibility_matrix.rst
108
# If the feature combo become valid
109
class SpecDecodeWorker(LoraNotSupportedWorkerBase):
110
"""Worker which implements speculative decoding.
vllm/utils.py
+1
-1
@@ -47,7 +47,7 @@ logger = init_logger(__name__)
47
48
# Exception strings for non-implemented encoder/decoder scenarios
49
50
-# Reminder: Please update docs/source/serving/compatibility_matrix.rst
50
+# Reminder: Please update docs/source/usage/compatibility_matrix.rst
51
# If the feature combo become valid
52
53
STR_NOT_IMPL_ENC_DEC_SWA = \
vllm/worker/multi_step_model_runner.py
+1
-1
@@ -817,7 +817,7 @@ def _pythonize_sampler_output(
817
818
for sgdx, (seq_group,
819
sample_result) in enumerate(zip(seq_groups, samples_list)):
820
- # Reminder: Please update docs/source/serving/compatibility_matrix.rst
820
+ # Reminder: Please update docs/source/usage/compatibility_matrix.rst
821
# If the feature combo become valid
822
# (Check for Guided Decoding)
823
if seq_group.sampling_params.logits_processors:
vllm/worker/utils.py
+1
-1
@@ -13,7 +13,7 @@ def assert_enc_dec_mr_supported_scenario(
13
a supported scenario.
14
'''
15
16
- # Reminder: Please update docs/source/serving/compatibility_matrix.rst
16
+ # Reminder: Please update docs/source/usage/compatibility_matrix.rst
17
# If the feature combo become valid
18
19
if enc_dec_mr.cache_config.enable_prefix_caching: