@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / e4fbb14414
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 07:00 UTC

[doc] update the code to add models (#10603)

Signed-off-by: youkaichao <youkaichao@gmail.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

youkaichao committed Nov 24, 2024 at 11:21 UTC e4fbb1441454847fdd871c9959b5cb05b5037aa2
1 file changed +57 -28
docs/source/models/adding_model.rst
+57 -28
@@ -38,41 +38,70 @@ For instance, vLLM's `OPT model <https://github.com/vllm-project/vllm/blob/main/
38 When copying the model code, make sure to review and adhere to the code's copyright and licensing terms.
39
40
41 -2. Rewrite the :code:`forward` methods
41 +2. Make your code compatible with vLLM
42 --------------------------------------
43
44 -Next, you need to rewrite the :meth:`~torch.nn.Module.forward` method of your model by following these steps:
45 -
46 -1. Remove any unnecessary code, such as the code only used for training.
47 -2. Change the input parameters:
48 -
49 -.. code-block:: diff
50 -
51 - def forward(
52 - self,
53 - input_ids: torch.Tensor,
54 - - attention_mask: Optional[torch.Tensor] = None,
55 - - position_ids: Optional[torch.LongTensor] = None,
56 - - past_key_values: Optional[List[torch.FloatTensor]] = None,
57 - - inputs_embeds: Optional[torch.FloatTensor] = None,
58 - - labels: Optional[torch.LongTensor] = None,
59 - - use_cache: Optional[bool] = None,
60 - - output_attentions: Optional[bool] = None,
61 - - output_hidden_states: Optional[bool] = None,
62 - - return_dict: Optional[bool] = None,
63 - - ) -> Union[Tuple, CausalLMOutputWithPast]:
64 - + positions: torch.Tensor,
65 - + kv_caches: List[torch.Tensor],
66 - + attn_metadata: AttentionMetadata,
67 - + ) -> Optional[SamplerOutput]:
68 -
69 -1. Update the code by considering that :code:`input_ids` and :code:`positions` are now flattened tensors.
70 -2. Replace the attention operation with either :code:`PagedAttention`, :code:`PagedAttentionWithRoPE`, or :code:`PagedAttentionWithALiBi` depending on the model's architecture.
44 +To ensure compatibility with vLLM, your model must meet the following requirements:
45 +
46 +Initialization Code
47 +^^^^^^^^^^^^^^^^^^^
48 +
49 +All vLLM modules within the model must include a ``prefix`` argument in their constructor. This ``prefix`` is typically the full name of the module in the model's state dictionary and is crucial for:
50 +
51 +* Runtime support: vLLM's attention operators are registered in a model's state by their full names. Each attention operator must have a unique prefix as its layer name to avoid conflicts.
52 +* Non-uniform quantization support: A quantized checkpoint can selectively quantize certain layers while keeping others in full precision. By providing the ``prefix`` during initialization, vLLM can match the current layer's ``prefix`` with the quantization configuration to determine if the layer should be initialized in quantized mode.
53 +
54 +The initialization code should look like this:
55 +
56 +.. code-block:: python
57 +
58 + from torch import nn
59 + from vllm.config import VllmConfig
60 + from vllm.attention import Attention
61 +
62 + class MyAttention(nn.Module):
63 + def __init__(self, vllm_config: VllmConfig, prefix: str):
64 + super().__init__()
65 + self.attn = Attention(prefix=f"{prefix}.attn")
66 +
67 + class MyDecoderLayer(nn.Module):
68 + def __init__(self, vllm_config: VllmConfig, prefix: str):
69 + super().__init__()
70 + self.self_attn = MyAttention(prefix=f"{prefix}.self_attn")
71 +
72 + class MyModel(nn.Module):
73 + def __init__(self, vllm_config: VllmConfig, prefix: str):
74 + super().__init__()
75 + self.layers = nn.ModuleList(
76 + [MyDecoderLayer(vllm_config, prefix=f"{prefix}.layers.{i}") for i in range(vllm_config.model_config.hf_config.num_hidden_layers)]
77 + )
78 +
79 + class MyModelForCausalLM(nn.Module):
80 + def __init__(self, vllm_config: VllmConfig, prefix: str = ""):
81 + super().__init__()
82 + self.model = MyModel(vllm_config, prefix=f"{prefix}.model")
83 +
84 +Computation Code
85 +^^^^^^^^^^^^^^^^
86 +
87 +Rewrite the :meth:`~torch.nn.Module.forward` method of your model to remove any unnecessary code, such as training-specific code. Modify the input parameters to treat ``input_ids`` and ``positions`` as flattened tensors with a single batch size dimension, without a max-sequence length dimension.
88 +
89 +.. code-block:: python
90 +
91 + def forward(
92 + self,
93 + input_ids: torch.Tensor,
94 + positions: torch.Tensor,
95 + kv_caches: List[torch.Tensor],
96 + attn_metadata: AttentionMetadata,
97 + ) -> torch.Tensor:
98 + ...
99
100 .. note::
101 Currently, vLLM supports the basic multi-head attention mechanism and its variant with rotary positional embeddings.
102 If your model employs a different attention mechanism, you will need to implement a new attention layer in vLLM.
103
104 +For reference, check out the `LLAMA model <https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/llama.py>`__. vLLM already supports a large number of models. It is recommended to find a model similar to yours and adapt it to your model's architecture. Check out the `vLLM models <https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/models>`__ directory for more examples.
105
106 3. (Optional) Implement tensor parallelism and quantization support
107 -------------------------------------------------------------------