A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 02:00 UTC
[Doc][V1] Update model implementation guide for V1 support (#11998)
Signed-off-by: Roger Wang <ywang@roblox.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
Roger Wang committed
Jan 13, 2025 at 03:58 UTC
cd8249903f189c5f06424e67dbc6512ca494a046
2 files changed
+83
-16
docs/source/contributing/model/basic.md
+11
-1
@@ -57,7 +57,17 @@ class MyModelForCausalLM(nn.Module):
57
58
### Computation Code
59
60
-Rewrite the {meth}`~torch.nn.Module.forward` method of your model to remove any unnecessary code, such as training-specific code. Modify the input parameters to treat `input_ids` and `positions` as flattened tensors with a single batch size dimension, without a max-sequence length dimension.
60
+- Add a `get_input_embeddings` method inside `MyModel` module that returns the text embeddings given `input_ids`. This is equivalent to directly calling the text embedding layer, but provides a unified interface in case `MyModel` is used within a composite multimodal model.
61
+
62
+```python
63
+class MyModel(nn.Module):
64
+ ...
65
+
66
+ def get_input_embeddings(self, input_ids: torch.Tensor) -> torch.Tensor:
67
+ ...
68
+```
69
+
70
+- Rewrite the {meth}`~torch.nn.Module.forward` method of your model to remove any unnecessary code, such as training-specific code. Modify the input parameters to treat `input_ids` and `positions` as flattened tensors with a single batch size dimension, without a max-sequence length dimension.
71
72
```python
73
def forward(
docs/source/contributing/model/multimodal.md
+72
-15
@@ -9,7 +9,78 @@ This document walks you through the steps to extend a basic model so that it acc
9
It is assumed that you have already implemented the model in vLLM according to [these steps](#new-model-basic).
10
Further update the model as follows:
11
12
-- Implement the {class}`~vllm.model_executor.models.interfaces.SupportsMultiModal` interface.
12
+- Reserve a keyword parameter in {meth}`~torch.nn.Module.forward` for each input tensor that corresponds to a multi-modal input, as shown in the following example:
13
+
14
+ ```diff
15
+ def forward(
16
+ self,
17
+ input_ids: torch.Tensor,
18
+ positions: torch.Tensor,
19
+ kv_caches: List[torch.Tensor],
20
+ attn_metadata: AttentionMetadata,
21
+ + pixel_values: torch.Tensor,
22
+ ) -> SamplerOutput:
23
+ ```
24
+
25
+ More conveniently, you can simply pass `**kwargs` to the {meth}`~torch.nn.Module.forward` method and retrieve the keyword parameters for multimodal inputs from it.
26
+
27
+- Implement {meth}`~vllm.model_executor.models.interfaces.SupportsMultiModal.get_multimodal_embeddings` that returns the embeddings from running the multimodal inputs through the multimodal tokenizer of the model. Below we provide a boilerplate of a typical implementation pattern, but feel free to adjust it to your own needs.
28
+
29
+ ```python
30
+ class YourModelForImage2Seq(nn.Module):
31
+ ...
32
+
33
+ def _process_image_input(self, image_input: YourModelImageInputs) -> torch.Tensor:
34
+
35
+ assert self.vision_encoder is not None
36
+ image_features = self.vision_encoder(image_input)
37
+ return self.multi_modal_projector(image_features)
38
+
39
+ def get_multimodal_embeddings(self, **kwargs: object) -> Optional[NestedTensors]:
40
+
41
+ # Validate the multimodal input keyword arguments
42
+ image_input = self._parse_and_validate_image_input(**kwargs)
43
+ if image_input is None:
44
+ return None
45
+
46
+ # Run multimodal inputs through encoder and projector
47
+ vision_embeddings = self._process_image_input(image_input)
48
+ return vision_embeddings
49
+ ```
50
+
51
+ ```{important}
52
+ The returned `multimodal_embeddings` must be either a **3D {class}`torch.Tensor`** of shape `(num_items, feature_size, hidden_size)`, or a **list / tuple of 2D {class}`torch.Tensor`'s** of shape `(feature_size, hidden_size)`, so that `multimodal_embeddings[i]` retrieves the embeddings generated from the `i`-th multimodal data item (e.g, image) of the request.
53
+ ```
54
+
55
+- Implement {meth}`~vllm.model_executor.models.interfaces.SupportsMultiModal.get_input_embeddings` to merge `multimodal_embeddings` with text embeddings from the `input_ids`. If input processing for the model is implemented correctly (see sections below), then you can leverage the utility function we provide to easily merge the embeddings.
56
+
57
+ ```python
58
+ from .utils import merge_multimodal_embeddings
59
+
60
+ class YourModelForImage2Seq(nn.Module):
61
+ ...
62
+
63
+ def get_input_embeddings(
64
+ self,
65
+ input_ids: torch.Tensor,
66
+ multimodal_embeddings: Optional[NestedTensors] = None,
67
+ ) -> torch.Tensor:
68
+
69
+ # `get_input_embeddings` should already be implemented for the language
70
+ # model as one of the requirements of basic vLLM model implementation.
71
+ inputs_embeds = self.language_model.get_input_embeddings(input_ids)
72
+
73
+ if multimodal_embeddings is not None:
74
+ inputs_embeds = merge_multimodal_embeddings(
75
+ input_ids=input_ids,
76
+ inputs_embeds=inputs_embeds,
77
+ multimodal_embeddings=multimodal_embeddings,
78
+ placeholder_token_id=self.config.image_token_index)
79
+
80
+ return inputs_embeds
81
+ ```
82
+
83
+- Once the above steps are done, update the model class with the {class}`~vllm.model_executor.models.interfaces.SupportsMultiModal` interface.
84
85
```diff
86
+ from vllm.model_executor.models.interfaces import SupportsMultiModal
@@ -23,20 +94,6 @@ Further update the model as follows:
94
Check out [the HuggingFace Transformers documentation](https://huggingface.co/docs/transformers/model_doc/auto#multimodal) for some examples.
95
```
96
26
-- If you haven't already done so, reserve a keyword parameter in {meth}`~torch.nn.Module.forward`
27
- for each input tensor that corresponds to a multi-modal input, as shown in the following example:
28
-
29
- ```diff
30
- def forward(
31
- self,
32
- input_ids: torch.Tensor,
33
- positions: torch.Tensor,
34
- kv_caches: List[torch.Tensor],
35
- attn_metadata: AttentionMetadata,
36
- + pixel_values: torch.Tensor,
37
- ) -> SamplerOutput:
38
- ```
39
-
97
## 2. Specify processing information
98
99
Next, create a subclass of {class}`~vllm.multimodal.processing.BaseProcessingInfo`