@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 9389380015
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 22:00 UTC

[Doc] Move guide for multimodal model and other improvements (#6168)

Cyrus Leung committed Jul 6, 2024 at 17:18 UTC 9389380015b80c109b899a08840132780b9b3fc0
8 files changed +61 -67
docs/source/dev/input_processing/model_inputs_index.rst
+3 -3
@@ -5,10 +5,10 @@ Input Processing
5
6 .. currentmodule:: vllm.inputs
7
8 -vLLM provides a mechanism for defining input processors for each model so that the inputs are processed
9 -in :class:`~vllm.LLMEngine` before they are passed to model executors.
8 +Each model can override parts of vLLM's :ref:`input processing pipeline <input_processing_pipeline>` via
9 +:data:`~vllm.inputs.INPUT_REGISTRY` and :data:`~vllm.multimodal.MULTIMODAL_REGISTRY`.
10
11 -Currently, this mechanism is only utilized in :ref:`multi-modal models <multi_modality>` for preprocessing multi-modal input
11 +Currently, this mechanism is only utilized in :ref:`multi-modal <multi_modality>` models for preprocessing multi-modal input
12 data in addition to input prompt, but it can be extended to text-only language models when needed.
13
14 Guides
docs/source/dev/multimodal/multimodal_index.rst
+6 -14
@@ -7,25 +7,17 @@ Multi-Modality
7
8 vLLM provides experimental support for multi-modal models through the :mod:`vllm.multimodal` package.
9
10 -:class:`vllm.inputs.PromptStrictInputs` accepts an additional attribute ``multi_modal_data``
11 -which allows you to pass in multi-modal input alongside text and token prompts.
10 +Multi-modal input can be passed alongside text and token prompts to :ref:`supported models <supported_vlms>`
11 +via the ``multi_modal_data`` field in :class:`vllm.inputs.PromptStrictInputs`.
12
13 .. note::
14 ``multi_modal_data`` can accept keys and values beyond the builtin ones, as long as a customized plugin is registered through
15 - :class:`vllm.multimodal.MULTIMODAL_REGISTRY`.
15 + the :class:`~vllm.multimodal.MULTIMODAL_REGISTRY`.
16
17 -By default, vLLM models do not support multi-modal inputs. To enable multi-modal support for a model, please follow :ref:`the guide for adding a new multimodal model. <adding_a_new_multimodal_model>`.
17 +To implement a new multi-modal model in vLLM, please follow :ref:`this guide <enabling_multimodal_inputs>`.
18
19 -
20 -# TODO: Add more instructions on how to do that once embeddings is in.
21 -
22 -Guides
23 -++++++
24 -
25 -.. toctree::
26 - :maxdepth: 1
27 -
28 - adding_multimodal_model
19 +..
20 + TODO: Add more instructions on how to add new plugins once embeddings is in.
21
22 Module Contents
23 +++++++++++++++
docs/source/index.rst
+2
@@ -92,6 +92,7 @@ Documentation
92
93 models/supported_models
94 models/adding_model
95 + models/enabling_multimodal_inputs
96 models/engine_args
97 models/lora
98 models/vlm
@@ -116,6 +117,7 @@ Documentation
117 automatic_prefix_caching/details
118
119 .. toctree::
120 + :maxdepth: 2
121 :caption: Developer Documentation
122
123 dev/sampling_params
docs/source/models/adding_model.rst
+21 -17
@@ -10,6 +10,10 @@ This document provides a high-level guide on integrating a `HuggingFace Transfor
10 The process is considerably straightforward if the model shares a similar architecture with an existing model in vLLM.
11 However, for models that include new operators (e.g., a new attention mechanism), the process can be a bit more complex.
12
13 +.. note::
14 + By default, vLLM models do not support multi-modal inputs. To enable multi-modal support,
15 + please follow :ref:`this guide <enabling_multimodal_inputs>` after implementing the model here.
16 +
17 .. tip::
18 If you are encountering issues while integrating your model into vLLM, feel free to open an issue on our `GitHub <https://github.com/vllm-project/vllm/issues>`_ repository.
19 We will be happy to help you out!
@@ -44,23 +48,23 @@ Next, you need to rewrite the :meth:`~torch.nn.Module.forward` method of your mo
48
49 .. code-block:: diff
50
47 - def forward(
48 - self,
49 - input_ids: torch.Tensor,
50 - - attention_mask: Optional[torch.Tensor] = None,
51 - - position_ids: Optional[torch.LongTensor] = None,
52 - - past_key_values: Optional[List[torch.FloatTensor]] = None,
53 - - inputs_embeds: Optional[torch.FloatTensor] = None,
54 - - labels: Optional[torch.LongTensor] = None,
55 - - use_cache: Optional[bool] = None,
56 - - output_attentions: Optional[bool] = None,
57 - - output_hidden_states: Optional[bool] = None,
58 - - return_dict: Optional[bool] = None,
59 - -) -> Union[Tuple, CausalLMOutputWithPast]:
60 - + positions: torch.Tensor,
61 - + kv_caches: List[torch.Tensor],
62 - + attn_metadata: AttentionMetadata,
63 - +) -> Optional[SamplerOutput]:
51 + def forward(
52 + self,
53 + input_ids: torch.Tensor,
54 + - attention_mask: Optional[torch.Tensor] = None,
55 + - position_ids: Optional[torch.LongTensor] = None,
56 + - past_key_values: Optional[List[torch.FloatTensor]] = None,
57 + - inputs_embeds: Optional[torch.FloatTensor] = None,
58 + - labels: Optional[torch.LongTensor] = None,
59 + - use_cache: Optional[bool] = None,
60 + - output_attentions: Optional[bool] = None,
61 + - output_hidden_states: Optional[bool] = None,
62 + - return_dict: Optional[bool] = None,
63 + - ) -> Union[Tuple, CausalLMOutputWithPast]:
64 + + positions: torch.Tensor,
65 + + kv_caches: List[torch.Tensor],
66 + + attn_metadata: AttentionMetadata,
67 + + ) -> Optional[SamplerOutput]:
68
69 1. Update the code by considering that :code:`input_ids` and :code:`positions` are now flattened tensors.
70 2. Replace the attention operation with either :code:`PagedAttention`, :code:`PagedAttentionWithRoPE`, or :code:`PagedAttentionWithALiBi` depending on the model's architecture.
docs/source/models/enabling_multimodal_inputs.rst renamed
+21 -26
@@ -1,26 +1,21 @@
1 -.. _adding_a_new_multimodal_model:
1 +.. _enabling_multimodal_inputs:
2
3 -Adding a New Multimodal Model
4 -=============================
3 +Enabling Multimodal Inputs
4 +==========================
5
6 -This document provides a high-level guide on integrating a :ref:`multi-modal model <multi_modality>` into vLLM.
6 +This document walks you through the steps to extend a vLLM model so that it accepts :ref:`multi-modal <multi_modality>` inputs.
7
8 -.. note::
9 - The complexity of adding a new model depends heavily on the model's architecture.
10 - The process is considerably straightforward if the model shares a similar architecture with an existing model in vLLM.
11 - However, for models that include new operators (e.g., a new attention mechanism), the process can be a bit more complex.
12 -
13 -.. tip::
14 - If you are encountering issues while integrating your model into vLLM, feel free to open an issue on our `GitHub <https://github.com/vllm-project/vllm/issues>`_ repository.
15 - We will be happy to help you out!
8 +.. seealso::
9 + :ref:`adding_a_new_model`
10
11
18 -1. Set up the base vLLM model
12 +1. Update the base vLLM model
13 -----------------------------
14
21 -As usual, follow :ref:`these steps <adding_a_new_model>` to implement the model in vLLM, but note the following:
15 +It is assumed that you have already implemented the model in vLLM according to :ref:`these steps <adding_a_new_model>`.
16 +Further update the model as follows:
17
23 -- You should additionally implement the :class:`~vllm.model_executor.models.interfaces.SupportsVision` interface.
18 +- Implement the :class:`~vllm.model_executor.models.interfaces.SupportsVision` interface.
19
20 .. code-block:: diff
21
@@ -33,19 +28,19 @@ As usual, follow :ref:`these steps <adding_a_new_model>` to implement the model
28 The model class does not have to be named :code:`*ForCausalLM`.
29 Check out `the HuggingFace Transformers documentation <https://huggingface.co/docs/transformers/model_doc/auto#multimodal>`__ for some examples.
30
36 -- While implementing the :meth:`~torch.nn.Module.forward` method, reserve a keyword parameter
31 +- If you haven't already done so, reserve a keyword parameter in :meth:`~torch.nn.Module.forward`
32 for each input tensor that corresponds to a multi-modal input, as shown in the following example:
33
34 .. code-block:: diff
35
41 - def forward(
42 - self,
43 - input_ids: torch.Tensor,
44 - positions: torch.Tensor,
45 - kv_caches: List[torch.Tensor],
46 - attn_metadata: AttentionMetadata,
47 - + pixel_values: torch.Tensor,
48 - ) -> SamplerOutput:
36 + def forward(
37 + self,
38 + input_ids: torch.Tensor,
39 + positions: torch.Tensor,
40 + kv_caches: List[torch.Tensor],
41 + attn_metadata: AttentionMetadata,
42 + + pixel_values: torch.Tensor,
43 + ) -> SamplerOutput:
44
45
46 2. Register input mappers
@@ -68,8 +63,8 @@ A default mapper is available for each modality in the core vLLM library. This i
63 :ref:`input_processing_pipeline`
64
65
71 -3. Register maximum number of multimodal tokens
72 -----------------------------------------------------------
66 +3. Register maximum number of multi-modal tokens
67 +------------------------------------------------
68
69 For each modality type that the model accepts as input, calculate the maximum possible number of tokens
70 and register it via :meth:`INPUT_REGISTRY.register_dummy_data <vllm.inputs.registry.InputRegistry.register_max_multimodal_tokens>`.
docs/source/models/supported_models.rst
+1 -1
@@ -192,7 +192,7 @@ Vision Language Models
192 -
193
194 If your model uses one of the above model architectures, you can seamlessly run your model with vLLM.
195 -Otherwise, please refer to :ref:`Adding a New Model <adding_a_new_model>` and :ref:`Adding a New Multimodal Model <adding_a_new_multimodal_model>`
195 +Otherwise, please refer to :ref:`Adding a New Model <adding_a_new_model>` and :ref:`Enabling Multimodal Inputs <enabling_multimodal_inputs>`
196 for instructions on how to implement support for your model.
197 Alternatively, you can raise an issue on our `GitHub <https://github.com/vllm-project/vllm/issues>`_ project.
198
vllm/inputs/registry.py
+1 -1
@@ -141,7 +141,7 @@ class InputRegistry:
141 The model is identified by ``model_config``.
142
143 See also:
144 - :ref:`adding_a_new_multimodal_model`
144 + :ref:`enabling_multimodal_inputs`
145 """
146 # Avoid circular import
147 from vllm.model_executor.model_loader import get_model_architecture
vllm/multimodal/base.py
+6 -5
@@ -162,8 +162,8 @@ class MultiModalPlugin(ABC):
162 If `None` is provided, then the default input mapper is used instead.
163
164 See also:
165 - :ref:`input_processing_pipeline`
166 - :ref:`adding_a_new_multimodal_model`
165 + - :ref:`input_processing_pipeline`
166 + - :ref:`enabling_multimodal_inputs`
167 """
168
169 def wrapper(model_cls: N) -> N:
@@ -192,7 +192,8 @@ class MultiModalPlugin(ABC):
192 TypeError: If the data type is not supported.
193
194 See also:
195 - :ref:`adding_a_new_multimodal_model`
195 + - :ref:`input_processing_pipeline`
196 + - :ref:`enabling_multimodal_inputs`
197 """
198 # Avoid circular import
199 from vllm.model_executor.model_loader import get_model_architecture
@@ -230,7 +231,7 @@ class MultiModalPlugin(ABC):
231 If `None` is provided, then the default calculation is used instead.
232
233 See also:
233 - :ref:`adding_a_new_multimodal_model`
234 + :ref:`enabling_multimodal_inputs`
235 """
236
237 def wrapper(model_cls: N) -> N:
@@ -260,7 +261,7 @@ class MultiModalPlugin(ABC):
261 The model is identified by ``model_config``.
262
263 See also:
263 - :ref:`adding_a_new_multimodal_model`
264 + :ref:`enabling_multimodal_inputs`
265 """
266 # Avoid circular import
267 from vllm.model_executor.model_loader import get_model_architecture