A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 22:00 UTC
[Doc] Move guide for multimodal model and other improvements (#6168)
Cyrus Leung committed
Jul 6, 2024 at 17:18 UTC
9389380015b80c109b899a08840132780b9b3fc0
8 files changed
+61
-67
docs/source/dev/input_processing/model_inputs_index.rst
+3
-3
@@ -5,10 +5,10 @@ Input Processing
5
6
.. currentmodule:: vllm.inputs
7
8
-vLLM provides a mechanism for defining input processors for each model so that the inputs are processed
9
-in :class:`~vllm.LLMEngine` before they are passed to model executors.
8
+Each model can override parts of vLLM's :ref:`input processing pipeline <input_processing_pipeline>` via
9
+:data:`~vllm.inputs.INPUT_REGISTRY` and :data:`~vllm.multimodal.MULTIMODAL_REGISTRY`.
10
11
-Currently, this mechanism is only utilized in :ref:`multi-modal models <multi_modality>` for preprocessing multi-modal input
11
+Currently, this mechanism is only utilized in :ref:`multi-modal <multi_modality>` models for preprocessing multi-modal input
12
data in addition to input prompt, but it can be extended to text-only language models when needed.
13
14
Guides
docs/source/dev/multimodal/multimodal_index.rst
+6
-14
@@ -7,25 +7,17 @@ Multi-Modality
7
8
vLLM provides experimental support for multi-modal models through the :mod:`vllm.multimodal` package.
9
10
-:class:`vllm.inputs.PromptStrictInputs` accepts an additional attribute ``multi_modal_data``
11
-which allows you to pass in multi-modal input alongside text and token prompts.
10
+Multi-modal input can be passed alongside text and token prompts to :ref:`supported models <supported_vlms>`
11
+via the ``multi_modal_data`` field in :class:`vllm.inputs.PromptStrictInputs`.
12
13
.. note::
14
``multi_modal_data`` can accept keys and values beyond the builtin ones, as long as a customized plugin is registered through
15
- :class:`vllm.multimodal.MULTIMODAL_REGISTRY`.
15
+ the :class:`~vllm.multimodal.MULTIMODAL_REGISTRY`.
16
17
-By default, vLLM models do not support multi-modal inputs. To enable multi-modal support for a model, please follow :ref:`the guide for adding a new multimodal model. <adding_a_new_multimodal_model>`.
17
+To implement a new multi-modal model in vLLM, please follow :ref:`this guide <enabling_multimodal_inputs>`.
18
19
-
20
-# TODO: Add more instructions on how to do that once embeddings is in.
21
-
22
-Guides
23
-++++++
24
-
25
-.. toctree::
26
- :maxdepth: 1
27
-
28
- adding_multimodal_model
19
+..
20
+ TODO: Add more instructions on how to add new plugins once embeddings is in.
21
22
Module Contents
23
+++++++++++++++
docs/source/index.rst
+2
@@ -92,6 +92,7 @@ Documentation
92
93
models/supported_models
94
models/adding_model
95
+ models/enabling_multimodal_inputs
96
models/engine_args
97
models/lora
98
models/vlm
@@ -116,6 +117,7 @@ Documentation
117
automatic_prefix_caching/details
118
119
.. toctree::
120
+ :maxdepth: 2
121
:caption: Developer Documentation
122
123
dev/sampling_params
docs/source/models/adding_model.rst
+21
-17
@@ -10,6 +10,10 @@ This document provides a high-level guide on integrating a `HuggingFace Transfor
10
The process is considerably straightforward if the model shares a similar architecture with an existing model in vLLM.
11
However, for models that include new operators (e.g., a new attention mechanism), the process can be a bit more complex.
12
13
+.. note::
14
+ By default, vLLM models do not support multi-modal inputs. To enable multi-modal support,
15
+ please follow :ref:`this guide <enabling_multimodal_inputs>` after implementing the model here.
16
+
17
.. tip::
18
If you are encountering issues while integrating your model into vLLM, feel free to open an issue on our `GitHub <https://github.com/vllm-project/vllm/issues>`_ repository.
19
We will be happy to help you out!
@@ -44,23 +48,23 @@ Next, you need to rewrite the :meth:`~torch.nn.Module.forward` method of your mo
48
49
.. code-block:: diff
50
47
- def forward(
48
- self,
49
- input_ids: torch.Tensor,
50
- - attention_mask: Optional[torch.Tensor] = None,
51
- - position_ids: Optional[torch.LongTensor] = None,
52
- - past_key_values: Optional[List[torch.FloatTensor]] = None,
53
- - inputs_embeds: Optional[torch.FloatTensor] = None,
54
- - labels: Optional[torch.LongTensor] = None,
55
- - use_cache: Optional[bool] = None,
56
- - output_attentions: Optional[bool] = None,
57
- - output_hidden_states: Optional[bool] = None,
58
- - return_dict: Optional[bool] = None,
59
- -) -> Union[Tuple, CausalLMOutputWithPast]:
60
- + positions: torch.Tensor,
61
- + kv_caches: List[torch.Tensor],
62
- + attn_metadata: AttentionMetadata,
63
- +) -> Optional[SamplerOutput]:
51
+ def forward(
52
+ self,
53
+ input_ids: torch.Tensor,
54
+ - attention_mask: Optional[torch.Tensor] = None,
55
+ - position_ids: Optional[torch.LongTensor] = None,
56
+ - past_key_values: Optional[List[torch.FloatTensor]] = None,
57
+ - inputs_embeds: Optional[torch.FloatTensor] = None,
58
+ - labels: Optional[torch.LongTensor] = None,
59
+ - use_cache: Optional[bool] = None,
60
+ - output_attentions: Optional[bool] = None,
61
+ - output_hidden_states: Optional[bool] = None,
62
+ - return_dict: Optional[bool] = None,
63
+ - ) -> Union[Tuple, CausalLMOutputWithPast]:
64
+ + positions: torch.Tensor,
65
+ + kv_caches: List[torch.Tensor],
66
+ + attn_metadata: AttentionMetadata,
67
+ + ) -> Optional[SamplerOutput]:
68
69
1. Update the code by considering that :code:`input_ids` and :code:`positions` are now flattened tensors.
70
2. Replace the attention operation with either :code:`PagedAttention`, :code:`PagedAttentionWithRoPE`, or :code:`PagedAttentionWithALiBi` depending on the model's architecture.
docs/source/models/enabling_multimodal_inputs.rst
renamed
+21
-26
@@ -1,26 +1,21 @@
1
-.. _adding_a_new_multimodal_model:
1
+.. _enabling_multimodal_inputs:
2
3
-Adding a New Multimodal Model
4
-=============================
3
+Enabling Multimodal Inputs
4
+==========================
5
6
-This document provides a high-level guide on integrating a :ref:`multi-modal model <multi_modality>` into vLLM.
6
+This document walks you through the steps to extend a vLLM model so that it accepts :ref:`multi-modal <multi_modality>` inputs.
7
8
-.. note::
9
- The complexity of adding a new model depends heavily on the model's architecture.
10
- The process is considerably straightforward if the model shares a similar architecture with an existing model in vLLM.
11
- However, for models that include new operators (e.g., a new attention mechanism), the process can be a bit more complex.
12
-
13
-.. tip::
14
- If you are encountering issues while integrating your model into vLLM, feel free to open an issue on our `GitHub <https://github.com/vllm-project/vllm/issues>`_ repository.
15
- We will be happy to help you out!
8
+.. seealso::
9
+ :ref:`adding_a_new_model`
10
11
18
-1. Set up the base vLLM model
12
+1. Update the base vLLM model
13
-----------------------------
14
21
-As usual, follow :ref:`these steps <adding_a_new_model>` to implement the model in vLLM, but note the following:
15
+It is assumed that you have already implemented the model in vLLM according to :ref:`these steps <adding_a_new_model>`.
16
+Further update the model as follows:
17
23
-- You should additionally implement the :class:`~vllm.model_executor.models.interfaces.SupportsVision` interface.
18
+- Implement the :class:`~vllm.model_executor.models.interfaces.SupportsVision` interface.
19
20
.. code-block:: diff
21
@@ -33,19 +28,19 @@ As usual, follow :ref:`these steps <adding_a_new_model>` to implement the model
28
The model class does not have to be named :code:`*ForCausalLM`.
29
Check out `the HuggingFace Transformers documentation <https://huggingface.co/docs/transformers/model_doc/auto#multimodal>`__ for some examples.
30
36
-- While implementing the :meth:`~torch.nn.Module.forward` method, reserve a keyword parameter
31
+- If you haven't already done so, reserve a keyword parameter in :meth:`~torch.nn.Module.forward`
32
for each input tensor that corresponds to a multi-modal input, as shown in the following example:
33
34
.. code-block:: diff
35
41
- def forward(
42
- self,
43
- input_ids: torch.Tensor,
44
- positions: torch.Tensor,
45
- kv_caches: List[torch.Tensor],
46
- attn_metadata: AttentionMetadata,
47
- + pixel_values: torch.Tensor,
48
- ) -> SamplerOutput:
36
+ def forward(
37
+ self,
38
+ input_ids: torch.Tensor,
39
+ positions: torch.Tensor,
40
+ kv_caches: List[torch.Tensor],
41
+ attn_metadata: AttentionMetadata,
42
+ + pixel_values: torch.Tensor,
43
+ ) -> SamplerOutput:
44
45
46
2. Register input mappers
@@ -68,8 +63,8 @@ A default mapper is available for each modality in the core vLLM library. This i
63
:ref:`input_processing_pipeline`
64
65
71
-3. Register maximum number of multimodal tokens
72
-----------------------------------------------------------
66
+3. Register maximum number of multi-modal tokens
67
+------------------------------------------------
68
69
For each modality type that the model accepts as input, calculate the maximum possible number of tokens
70
and register it via :meth:`INPUT_REGISTRY.register_dummy_data <vllm.inputs.registry.InputRegistry.register_max_multimodal_tokens>`.
docs/source/models/supported_models.rst
+1
-1
@@ -192,7 +192,7 @@ Vision Language Models
192
-
193
194
If your model uses one of the above model architectures, you can seamlessly run your model with vLLM.
195
-Otherwise, please refer to :ref:`Adding a New Model <adding_a_new_model>` and :ref:`Adding a New Multimodal Model <adding_a_new_multimodal_model>`
195
+Otherwise, please refer to :ref:`Adding a New Model <adding_a_new_model>` and :ref:`Enabling Multimodal Inputs <enabling_multimodal_inputs>`
196
for instructions on how to implement support for your model.
197
Alternatively, you can raise an issue on our `GitHub <https://github.com/vllm-project/vllm/issues>`_ project.
198
vllm/inputs/registry.py
+1
-1
@@ -141,7 +141,7 @@ class InputRegistry:
141
The model is identified by ``model_config``.
142
143
See also:
144
- :ref:`adding_a_new_multimodal_model`
144
+ :ref:`enabling_multimodal_inputs`
145
"""
146
# Avoid circular import
147
from vllm.model_executor.model_loader import get_model_architecture
vllm/multimodal/base.py
+6
-5
@@ -162,8 +162,8 @@ class MultiModalPlugin(ABC):
162
If `None` is provided, then the default input mapper is used instead.
163
164
See also:
165
- :ref:`input_processing_pipeline`
166
- :ref:`adding_a_new_multimodal_model`
165
+ - :ref:`input_processing_pipeline`
166
+ - :ref:`enabling_multimodal_inputs`
167
"""
168
169
def wrapper(model_cls: N) -> N:
@@ -192,7 +192,8 @@ class MultiModalPlugin(ABC):
192
TypeError: If the data type is not supported.
193
194
See also:
195
- :ref:`adding_a_new_multimodal_model`
195
+ - :ref:`input_processing_pipeline`
196
+ - :ref:`enabling_multimodal_inputs`
197
"""
198
# Avoid circular import
199
from vllm.model_executor.model_loader import get_model_architecture
@@ -230,7 +231,7 @@ class MultiModalPlugin(ABC):
231
If `None` is provided, then the default calculation is used instead.
232
233
See also:
233
- :ref:`adding_a_new_multimodal_model`
234
+ :ref:`enabling_multimodal_inputs`
235
"""
236
237
def wrapper(model_cls: N) -> N:
@@ -260,7 +261,7 @@ class MultiModalPlugin(ABC):
261
The model is identified by ``model_config``.
262
263
See also:
263
- :ref:`adding_a_new_multimodal_model`
264
+ :ref:`enabling_multimodal_inputs`
265
"""
266
# Avoid circular import
267
from vllm.model_executor.model_loader import get_model_architecture