A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 09:15 UTC
[Docs] Write the `Adding a New Model` section (#138)
Woosuk Kwon committed
Jun 5, 2023 at 20:01 UTC
456941cfe4313f5b89390e55fa4ed0b9771731ce
1 file changed
+88
-1
docs/source/models/adding_model.rst
+88
-1
@@ -3,5 +3,92 @@
3
Adding a New Model
4
==================
5
6
+This document provides a high-level guide on integrating a `HuggingFace Transformers <https://github.com/huggingface/transformers>`_ model into CacheFlow.
7
7
-Placeholder
8
+.. note::
9
+ The complexity of adding a new model depends heavily on the model's architecture.
10
+ The process is considerably straightforward if the model shares a similar architecture with an existing model in CacheFlow.
11
+ However, for models that include new operators (e.g., a new attention mechanism), the process can be a bit more complex.
12
+
13
+.. tip::
14
+ If you are encountering issues while integrating your model into CacheFlow, feel free to open an issue on our `GitHub <https://github.com/WoosukKwon/cacheflow/issues>`_ repository.
15
+ We will be happy to help you out!
16
+
17
+
18
+0. Fork the CacheFlow repository
19
+--------------------------------
20
+
21
+Start by forking our `GitHub <https://github.com/WoosukKwon/cacheflow/issues>`_ repository and then :ref:`build it from source <build_from_source>`.
22
+This gives you the ability to modify the codebase and test your model.
23
+
24
+
25
+1. Bring your model code
26
+------------------------
27
+
28
+Clone the PyTorch model code from the HuggingFace Transformers repository and put it into the `cacheflow/model_executor/models <https://github.com/WoosukKwon/cacheflow/tree/main/cacheflow/model_executor/models>`_ directory.
29
+For instance, CacheFlow's `OPT model <https://github.com/WoosukKwon/cacheflow/blob/main/cacheflow/model_executor/models/opt.py>`_ was adpated from the HuggingFace's `modeling_opt.py <https://github.com/huggingface/transformers/blob/main/src/transformers/models/opt/modeling_opt.py>`_ file.
30
+
31
+.. warning::
32
+ When copying the model code, make sure to review and adhere to the code's copyright and licensing terms.
33
+
34
+
35
+2. Rewrite the :code:`forward` methods
36
+--------------------------------------
37
+
38
+Next, you need to rewrite the :code:`forward` methods of your model by following these steps:
39
+
40
+1. Remove any unnecessary code, such as the code only used for training.
41
+2. Change the input parameters:
42
+
43
+.. code-block:: diff
44
+
45
+ def forward(
46
+ self,
47
+ input_ids: torch.Tensor,
48
+ - attention_mask: Optional[torch.Tensor] = None,
49
+ - position_ids: Optional[torch.LongTensor] = None,
50
+ - past_key_values: Optional[List[torch.FloatTensor]] = None,
51
+ - inputs_embeds: Optional[torch.FloatTensor] = None,
52
+ - labels: Optional[torch.LongTensor] = None,
53
+ - use_cache: Optional[bool] = None,
54
+ - output_attentions: Optional[bool] = None,
55
+ - output_hidden_states: Optional[bool] = None,
56
+ - return_dict: Optional[bool] = None,
57
+ -) -> Union[Tuple, CausalLMOutputWithPast]:
58
+ + positions: torch.Tensor,
59
+ + kv_caches: List[KVCache],
60
+ + input_metadata: InputMetadata,
61
+ + cache_events: Optional[List[torch.cuda.Event]],
62
+ +) -> Dict[int, SequenceOutputs]:
63
+
64
+3. Update the code by considering that :code:`input_ids` and :code:`positions` are now flattened tensors.
65
+4. Replace the attention operation with either :code:`GPTCacheFlowAttention` or :code:`GPTNeoXCacheFlowAttention`, depending on the model's architecture.
66
+
67
+.. note::
68
+ Currently, CacheFlow supports the basic multi-head attention mechanism and its variant with rotary positional embeddings.
69
+ If your model employs a different attention mechanism, you will need to implement a new attention layer in CacheFlow.
70
+
71
+
72
+3. (Optional) Implement tensor parallelism support
73
+--------------------------------------------------
74
+
75
+If your model is too large to fit into a single GPU, you can use tensor parallelism to manage it.
76
+To do this, substitute your model's linear and embedding layers with their tensor-parallel versions.
77
+For the embedding layer, you can simply replace :code:`nn.Embedding` with :code:`VocabParallelEmbedding`.
78
+When it comes to the linear layers, you should use either :code:`RowParallelLinear` or :code:`ColumnParallelLinear`.
79
+Typically, :code:`ColumnParallelLinear` is used for QKV linear layers and the first linear layers of the MLP blocks.
80
+For the remaining linear layers, :code:`RowParallelLinear` is used.
81
+
82
+
83
+4. Implement the weight loading logic
84
+-------------------------------------
85
+
86
+You now need to implement the :code:`load_weights` method in your :code:`*ForCausalLM` class.
87
+This method should load the weights from the HuggingFace's checkpoint file and assign them to the corresponding layers in your model.
88
+While the process is straightforward for most layers, the tensor-parallel layers necessitate some additional care as their weights should be partitioned to multiple GPUs.
89
+
90
+
91
+5. Register your model
92
+----------------------
93
+
94
+Finally, include your :code:`*ForCausalLM` class in `cacheflow/model_executor/models/__init__.py <https://github.com/WoosukKwon/cacheflow/blob/main/cacheflow/model_executor/models/__init__.py>`_ and register it to the :code:`_MODEL_REGISTRY` in `cacheflow/model_executor/model_loader.py <https://github.com/WoosukKwon/cacheflow/blob/main/cacheflow/model_executor/model_loader.py>`_.