A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 07:00 UTC
[doc] update the code to add models (#10603)
Signed-off-by: youkaichao <youkaichao@gmail.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
youkaichao committed
Nov 24, 2024 at 11:21 UTC
e4fbb1441454847fdd871c9959b5cb05b5037aa2
1 file changed
+57
-28
docs/source/models/adding_model.rst
+57
-28
@@ -38,41 +38,70 @@ For instance, vLLM's `OPT model <https://github.com/vllm-project/vllm/blob/main/
38
When copying the model code, make sure to review and adhere to the code's copyright and licensing terms.
39
40
41
-2. Rewrite the :code:`forward` methods
41
+2. Make your code compatible with vLLM
42
--------------------------------------
43
44
-Next, you need to rewrite the :meth:`~torch.nn.Module.forward` method of your model by following these steps:
45
-
46
-1. Remove any unnecessary code, such as the code only used for training.
47
-2. Change the input parameters:
48
-
49
-.. code-block:: diff
50
-
51
- def forward(
52
- self,
53
- input_ids: torch.Tensor,
54
- - attention_mask: Optional[torch.Tensor] = None,
55
- - position_ids: Optional[torch.LongTensor] = None,
56
- - past_key_values: Optional[List[torch.FloatTensor]] = None,
57
- - inputs_embeds: Optional[torch.FloatTensor] = None,
58
- - labels: Optional[torch.LongTensor] = None,
59
- - use_cache: Optional[bool] = None,
60
- - output_attentions: Optional[bool] = None,
61
- - output_hidden_states: Optional[bool] = None,
62
- - return_dict: Optional[bool] = None,
63
- - ) -> Union[Tuple, CausalLMOutputWithPast]:
64
- + positions: torch.Tensor,
65
- + kv_caches: List[torch.Tensor],
66
- + attn_metadata: AttentionMetadata,
67
- + ) -> Optional[SamplerOutput]:
68
-
69
-1. Update the code by considering that :code:`input_ids` and :code:`positions` are now flattened tensors.
70
-2. Replace the attention operation with either :code:`PagedAttention`, :code:`PagedAttentionWithRoPE`, or :code:`PagedAttentionWithALiBi` depending on the model's architecture.
44
+To ensure compatibility with vLLM, your model must meet the following requirements:
45
+
46
+Initialization Code
47
+^^^^^^^^^^^^^^^^^^^
48
+
49
+All vLLM modules within the model must include a ``prefix`` argument in their constructor. This ``prefix`` is typically the full name of the module in the model's state dictionary and is crucial for:
50
+
51
+* Runtime support: vLLM's attention operators are registered in a model's state by their full names. Each attention operator must have a unique prefix as its layer name to avoid conflicts.
52
+* Non-uniform quantization support: A quantized checkpoint can selectively quantize certain layers while keeping others in full precision. By providing the ``prefix`` during initialization, vLLM can match the current layer's ``prefix`` with the quantization configuration to determine if the layer should be initialized in quantized mode.
53
+
54
+The initialization code should look like this:
55
+
56
+.. code-block:: python
57
+
58
+ from torch import nn
59
+ from vllm.config import VllmConfig
60
+ from vllm.attention import Attention
61
+
62
+ class MyAttention(nn.Module):
63
+ def __init__(self, vllm_config: VllmConfig, prefix: str):
64
+ super().__init__()
65
+ self.attn = Attention(prefix=f"{prefix}.attn")
66
+
67
+ class MyDecoderLayer(nn.Module):
68
+ def __init__(self, vllm_config: VllmConfig, prefix: str):
69
+ super().__init__()
70
+ self.self_attn = MyAttention(prefix=f"{prefix}.self_attn")
71
+
72
+ class MyModel(nn.Module):
73
+ def __init__(self, vllm_config: VllmConfig, prefix: str):
74
+ super().__init__()
75
+ self.layers = nn.ModuleList(
76
+ [MyDecoderLayer(vllm_config, prefix=f"{prefix}.layers.{i}") for i in range(vllm_config.model_config.hf_config.num_hidden_layers)]
77
+ )
78
+
79
+ class MyModelForCausalLM(nn.Module):
80
+ def __init__(self, vllm_config: VllmConfig, prefix: str = ""):
81
+ super().__init__()
82
+ self.model = MyModel(vllm_config, prefix=f"{prefix}.model")
83
+
84
+Computation Code
85
+^^^^^^^^^^^^^^^^
86
+
87
+Rewrite the :meth:`~torch.nn.Module.forward` method of your model to remove any unnecessary code, such as training-specific code. Modify the input parameters to treat ``input_ids`` and ``positions`` as flattened tensors with a single batch size dimension, without a max-sequence length dimension.
88
+
89
+.. code-block:: python
90
+
91
+ def forward(
92
+ self,
93
+ input_ids: torch.Tensor,
94
+ positions: torch.Tensor,
95
+ kv_caches: List[torch.Tensor],
96
+ attn_metadata: AttentionMetadata,
97
+ ) -> torch.Tensor:
98
+ ...
99
100
.. note::
101
Currently, vLLM supports the basic multi-head attention mechanism and its variant with rotary positional embeddings.
102
If your model employs a different attention mechanism, you will need to implement a new attention layer in vLLM.
103
104
+For reference, check out the `LLAMA model <https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/llama.py>`__. vLLM already supports a large number of models. It is recommended to find a model similar to yours and adapt it to your model's architecture. Check out the `vLLM models <https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/models>`__ directory for more examples.
105
106
3. (Optional) Implement tensor parallelism and quantization support
107
-------------------------------------------------------------------