@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 456941cfe4
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 09:15 UTC

[Docs] Write the `Adding a New Model` section (#138)

Woosuk Kwon committed Jun 5, 2023 at 20:01 UTC 456941cfe4313f5b89390e55fa4ed0b9771731ce
1 file changed +88 -1
docs/source/models/adding_model.rst
+88 -1
@@ -3,5 +3,92 @@
3 Adding a New Model
4 ==================
5
6 +This document provides a high-level guide on integrating a `HuggingFace Transformers <https://github.com/huggingface/transformers>`_ model into CacheFlow.
7
7 -Placeholder
8 +.. note::
9 + The complexity of adding a new model depends heavily on the model's architecture.
10 + The process is considerably straightforward if the model shares a similar architecture with an existing model in CacheFlow.
11 + However, for models that include new operators (e.g., a new attention mechanism), the process can be a bit more complex.
12 +
13 +.. tip::
14 + If you are encountering issues while integrating your model into CacheFlow, feel free to open an issue on our `GitHub <https://github.com/WoosukKwon/cacheflow/issues>`_ repository.
15 + We will be happy to help you out!
16 +
17 +
18 +0. Fork the CacheFlow repository
19 +--------------------------------
20 +
21 +Start by forking our `GitHub <https://github.com/WoosukKwon/cacheflow/issues>`_ repository and then :ref:`build it from source <build_from_source>`.
22 +This gives you the ability to modify the codebase and test your model.
23 +
24 +
25 +1. Bring your model code
26 +------------------------
27 +
28 +Clone the PyTorch model code from the HuggingFace Transformers repository and put it into the `cacheflow/model_executor/models <https://github.com/WoosukKwon/cacheflow/tree/main/cacheflow/model_executor/models>`_ directory.
29 +For instance, CacheFlow's `OPT model <https://github.com/WoosukKwon/cacheflow/blob/main/cacheflow/model_executor/models/opt.py>`_ was adpated from the HuggingFace's `modeling_opt.py <https://github.com/huggingface/transformers/blob/main/src/transformers/models/opt/modeling_opt.py>`_ file.
30 +
31 +.. warning::
32 + When copying the model code, make sure to review and adhere to the code's copyright and licensing terms.
33 +
34 +
35 +2. Rewrite the :code:`forward` methods
36 +--------------------------------------
37 +
38 +Next, you need to rewrite the :code:`forward` methods of your model by following these steps:
39 +
40 +1. Remove any unnecessary code, such as the code only used for training.
41 +2. Change the input parameters:
42 +
43 +.. code-block:: diff
44 +
45 + def forward(
46 + self,
47 + input_ids: torch.Tensor,
48 + - attention_mask: Optional[torch.Tensor] = None,
49 + - position_ids: Optional[torch.LongTensor] = None,
50 + - past_key_values: Optional[List[torch.FloatTensor]] = None,
51 + - inputs_embeds: Optional[torch.FloatTensor] = None,
52 + - labels: Optional[torch.LongTensor] = None,
53 + - use_cache: Optional[bool] = None,
54 + - output_attentions: Optional[bool] = None,
55 + - output_hidden_states: Optional[bool] = None,
56 + - return_dict: Optional[bool] = None,
57 + -) -> Union[Tuple, CausalLMOutputWithPast]:
58 + + positions: torch.Tensor,
59 + + kv_caches: List[KVCache],
60 + + input_metadata: InputMetadata,
61 + + cache_events: Optional[List[torch.cuda.Event]],
62 + +) -> Dict[int, SequenceOutputs]:
63 +
64 +3. Update the code by considering that :code:`input_ids` and :code:`positions` are now flattened tensors.
65 +4. Replace the attention operation with either :code:`GPTCacheFlowAttention` or :code:`GPTNeoXCacheFlowAttention`, depending on the model's architecture.
66 +
67 +.. note::
68 + Currently, CacheFlow supports the basic multi-head attention mechanism and its variant with rotary positional embeddings.
69 + If your model employs a different attention mechanism, you will need to implement a new attention layer in CacheFlow.
70 +
71 +
72 +3. (Optional) Implement tensor parallelism support
73 +--------------------------------------------------
74 +
75 +If your model is too large to fit into a single GPU, you can use tensor parallelism to manage it.
76 +To do this, substitute your model's linear and embedding layers with their tensor-parallel versions.
77 +For the embedding layer, you can simply replace :code:`nn.Embedding` with :code:`VocabParallelEmbedding`.
78 +When it comes to the linear layers, you should use either :code:`RowParallelLinear` or :code:`ColumnParallelLinear`.
79 +Typically, :code:`ColumnParallelLinear` is used for QKV linear layers and the first linear layers of the MLP blocks.
80 +For the remaining linear layers, :code:`RowParallelLinear` is used.
81 +
82 +
83 +4. Implement the weight loading logic
84 +-------------------------------------
85 +
86 +You now need to implement the :code:`load_weights` method in your :code:`*ForCausalLM` class.
87 +This method should load the weights from the HuggingFace's checkpoint file and assign them to the corresponding layers in your model.
88 +While the process is straightforward for most layers, the tensor-parallel layers necessitate some additional care as their weights should be partitioned to multiple GPUs.
89 +
90 +
91 +5. Register your model
92 +----------------------
93 +
94 +Finally, include your :code:`*ForCausalLM` class in `cacheflow/model_executor/models/__init__.py <https://github.com/WoosukKwon/cacheflow/blob/main/cacheflow/model_executor/models/__init__.py>`_ and register it to the :code:`_MODEL_REGISTRY` in `cacheflow/model_executor/model_loader.py <https://github.com/WoosukKwon/cacheflow/blob/main/cacheflow/model_executor/model_loader.py>`_.