A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 03:15 UTC
[Doc][2/N] Reorganize Models and Usage sections (#11755)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Cyrus Leung committed
Jan 6, 2025 at 21:40 UTC
ee77fdb5de42a6fead2b897d87d99d4b1e5650a9
45 files changed
+265
-238
.github/ISSUE_TEMPLATE/600-new-model.yml
+1
-1
@@ -9,7 +9,7 @@ body:
9
value: >
10
#### Before submitting an issue, please make sure the issue hasn't been already addressed by searching through [the existing and past issues](https://github.com/vllm-project/vllm/issues?q=is%3Aissue+sort%3Acreated-desc+).
11
12
- #### We also highly recommend you read https://docs.vllm.ai/en/latest/models/adding_model.html first to understand how to add a new model.
12
+ #### We also highly recommend you read https://docs.vllm.ai/en/latest/contributing/model/adding_model.html first to understand how to add a new model.
13
- type: textarea
14
attributes:
15
label: The model to consider.
docs/source/assets/features/disagg_prefill/abstraction.jpg
renamed
docs/source/assets/features/disagg_prefill/overview.jpg
renamed
docs/source/contributing/model/basic.md
new
+102
@@ -0,0 +1,102 @@
1
+(new-model-basic)=
2
+
3
+# Basic Implementation
4
+
5
+This guide walks you through the steps to implement a basic vLLM model.
6
+
7
+## 1. Bring your model code
8
+
9
+First, clone the PyTorch model code from the source repository.
10
+For instance, vLLM's [OPT model](gh-file:vllm/model_executor/models/opt.py) was adapted from
11
+HuggingFace's [modeling_opt.py](https://github.com/huggingface/transformers/blob/main/src/transformers/models/opt/modeling_opt.py) file.
12
+
13
+```{warning}
14
+Make sure to review and adhere to the original code's copyright and licensing terms!
15
+```
16
+
17
+## 2. Make your code compatible with vLLM
18
+
19
+To ensure compatibility with vLLM, your model must meet the following requirements:
20
+
21
+### Initialization Code
22
+
23
+All vLLM modules within the model must include a `prefix` argument in their constructor. This `prefix` is typically the full name of the module in the model's state dictionary and is crucial for:
24
+
25
+- Runtime support: vLLM's attention operators are registered in a model's state by their full names. Each attention operator must have a unique prefix as its layer name to avoid conflicts.
26
+- Non-uniform quantization support: A quantized checkpoint can selectively quantize certain layers while keeping others in full precision. By providing the `prefix` during initialization, vLLM can match the current layer's `prefix` with the quantization configuration to determine if the layer should be initialized in quantized mode.
27
+
28
+The initialization code should look like this:
29
+
30
+```python
31
+from torch import nn
32
+from vllm.config import VllmConfig
33
+from vllm.attention import Attention
34
+
35
+class MyAttention(nn.Module):
36
+ def __init__(self, vllm_config: VllmConfig, prefix: str):
37
+ super().__init__()
38
+ self.attn = Attention(prefix=f"{prefix}.attn")
39
+
40
+class MyDecoderLayer(nn.Module):
41
+ def __init__(self, vllm_config: VllmConfig, prefix: str):
42
+ super().__init__()
43
+ self.self_attn = MyAttention(prefix=f"{prefix}.self_attn")
44
+
45
+class MyModel(nn.Module):
46
+ def __init__(self, vllm_config: VllmConfig, prefix: str):
47
+ super().__init__()
48
+ self.layers = nn.ModuleList(
49
+ [MyDecoderLayer(vllm_config, prefix=f"{prefix}.layers.{i}") for i in range(vllm_config.model_config.hf_config.num_hidden_layers)]
50
+ )
51
+
52
+class MyModelForCausalLM(nn.Module):
53
+ def __init__(self, vllm_config: VllmConfig, prefix: str = ""):
54
+ super().__init__()
55
+ self.model = MyModel(vllm_config, prefix=f"{prefix}.model")
56
+```
57
+
58
+### Computation Code
59
+
60
+Rewrite the {meth}`~torch.nn.Module.forward` method of your model to remove any unnecessary code, such as training-specific code. Modify the input parameters to treat `input_ids` and `positions` as flattened tensors with a single batch size dimension, without a max-sequence length dimension.
61
+
62
+```python
63
+def forward(
64
+ self,
65
+ input_ids: torch.Tensor,
66
+ positions: torch.Tensor,
67
+ kv_caches: List[torch.Tensor],
68
+ attn_metadata: AttentionMetadata,
69
+) -> torch.Tensor:
70
+ ...
71
+```
72
+
73
+```{note}
74
+Currently, vLLM supports the basic multi-head attention mechanism and its variant with rotary positional embeddings.
75
+If your model employs a different attention mechanism, you will need to implement a new attention layer in vLLM.
76
+```
77
+
78
+For reference, check out our [Llama implementation](gh-file:vllm/model_executor/models/llama.py). vLLM already supports a large number of models. It is recommended to find a model similar to yours and adapt it to your model's architecture. Check out <gh-dir:vllm/model_executor/models> for more examples.
79
+
80
+## 3. (Optional) Implement tensor parallelism and quantization support
81
+
82
+If your model is too large to fit into a single GPU, you can use tensor parallelism to manage it.
83
+To do this, substitute your model's linear and embedding layers with their tensor-parallel versions.
84
+For the embedding layer, you can simply replace {class}`torch.nn.Embedding` with `VocabParallelEmbedding`. For the output LM head, you can use `ParallelLMHead`.
85
+When it comes to the linear layers, we provide the following options to parallelize them:
86
+
87
+- `ReplicatedLinear`: Replicates the inputs and weights across multiple GPUs. No memory saving.
88
+- `RowParallelLinear`: The input tensor is partitioned along the hidden dimension. The weight matrix is partitioned along the rows (input dimension). An *all-reduce* operation is performed after the matrix multiplication to reduce the results. Typically used for the second FFN layer and the output linear transformation of the attention layer.
89
+- `ColumnParallelLinear`: The input tensor is replicated. The weight matrix is partitioned along the columns (output dimension). The result is partitioned along the column dimension. Typically used for the first FFN layer and the separated QKV transformation of the attention layer in the original Transformer.
90
+- `MergedColumnParallelLinear`: Column-parallel linear that merges multiple `ColumnParallelLinear` operators. Typically used for the first FFN layer with weighted activation functions (e.g., SiLU). This class handles the sharded weight loading logic of multiple weight matrices.
91
+- `QKVParallelLinear`: Parallel linear layer for the query, key, and value projections of the multi-head and grouped-query attention mechanisms. When number of key/value heads are less than the world size, this class replicates the key/value heads properly. This class handles the weight loading and replication of the weight matrices.
92
+
93
+Note that all the linear layers above take `linear_method` as an input. vLLM will set this parameter according to different quantization schemes to support weight quantization.
94
+
95
+## 4. Implement the weight loading logic
96
+
97
+You now need to implement the `load_weights` method in your `*ForCausalLM` class.
98
+This method should load the weights from the HuggingFace's checkpoint file and assign them to the corresponding layers in your model. Specifically, for `MergedColumnParallelLinear` and `QKVParallelLinear` layers, if the original model has separated weight matrices, you need to load the different parts separately.
99
+
100
+## 5. Register your model
101
+
102
+See [this page](#new-model-registration) for instructions on how to register your new model to be used by vLLM.
docs/source/contributing/model/index.md
new
+26
@@ -0,0 +1,26 @@
1
+(new-model)=
2
+
3
+# Adding a New Model
4
+
5
+This section provides more information on how to integrate a [HuggingFace Transformers](https://github.com/huggingface/transformers) model into vLLM.
6
+
7
+```{toctree}
8
+:caption: Contents
9
+:maxdepth: 1
10
+
11
+basic
12
+registration
13
+multimodal
14
+```
15
+
16
+```{note}
17
+The complexity of adding a new model depends heavily on the model's architecture.
18
+The process is considerably straightforward if the model shares a similar architecture with an existing model in vLLM.
19
+However, for models that include new operators (e.g., a new attention mechanism), the process can be a bit more complex.
20
+```
21
+
22
+```{tip}
23
+If you are encountering issues while integrating your model into vLLM, feel free to open a [GitHub issue](https://github.com/vllm-project/vllm/issues)
24
+or ask on our [developer slack](https://slack.vllm.ai).
25
+We will be happy to help you out!
26
+```
docs/source/contributing/model/multimodal.md
renamed
+2
-6
@@ -2,15 +2,11 @@
2
3
# Enabling Multimodal Inputs
4
5
-This document walks you through the steps to extend a vLLM model so that it accepts [multi-modal inputs](#multimodal-inputs).
6
-
7
-```{seealso}
8
-[Adding a New Model](adding-a-new-model)
9
-```
5
+This document walks you through the steps to extend a basic model so that it accepts [multi-modal inputs](#multimodal-inputs).
6
7
## 1. Update the base vLLM model
8
13
-It is assumed that you have already implemented the model in vLLM according to [these steps](#adding-a-new-model).
9
+It is assumed that you have already implemented the model in vLLM according to [these steps](#new-model-basic).
10
Further update the model as follows:
11
12
- Implement the {class}`~vllm.model_executor.models.interfaces.SupportsMultiModal` interface.
docs/source/contributing/model/registration.md
new
+56
@@ -0,0 +1,56 @@
1
+(new-model-registration)=
2
+
3
+# Model Registration
4
+
5
+vLLM relies on a model registry to determine how to run each model.
6
+A list of pre-registered architectures can be found on the [Supported Models](#supported-models) page.
7
+
8
+If your model is not on this list, you must register it to vLLM.
9
+This page provides detailed instructions on how to do so.
10
+
11
+## Built-in models
12
+
13
+To add a model directly to the vLLM library, start by forking our [GitHub repository](https://github.com/vllm-project/vllm) and then [build it from source](#build-from-source).
14
+This gives you the ability to modify the codebase and test your model.
15
+
16
+After you have implemented your model (see [tutorial](#new-model-basic)), put it into the <gh-dir:vllm/model_executor/models> directory.
17
+Then, add your model class to `_VLLM_MODELS` in <gh-file:vllm/model_executor/models/registry.py> so that it is automatically registered upon importing vLLM.
18
+You should also include an example HuggingFace repository for this model in <gh-file:tests/models/registry.py> to run the unit tests.
19
+Finally, update the [Supported Models](#supported-models) documentation page to promote your model!
20
+
21
+```{important}
22
+The list of models in each section should be maintained in alphabetical order.
23
+```
24
+
25
+## Out-of-tree models
26
+
27
+You can load an external model using a plugin without modifying the vLLM codebase.
28
+
29
+```{seealso}
30
+[vLLM's Plugin System](#plugin-system)
31
+```
32
+
33
+To register the model, use the following code:
34
+
35
+```python
36
+from vllm import ModelRegistry
37
+from your_code import YourModelForCausalLM
38
+ModelRegistry.register_model("YourModelForCausalLM", YourModelForCausalLM)
39
+```
40
+
41
+If your model imports modules that initialize CUDA, consider lazy-importing it to avoid errors like `RuntimeError: Cannot re-initialize CUDA in forked subprocess`:
42
+
43
+```python
44
+from vllm import ModelRegistry
45
+
46
+ModelRegistry.register_model("YourModelForCausalLM", "your_code:YourModelForCausalLM")
47
+```
48
+
49
+```{important}
50
+If your model is a multimodal model, ensure the model class implements the {class}`~vllm.model_executor.models.interfaces.SupportsMultiModal` interface.
51
+Read more about that [here](#enabling-multimodal-inputs).
52
+```
53
+
54
+```{note}
55
+Although you can directly put these code snippets in your script using `vllm.LLM`, the recommended way is to place these snippets in a vLLM plugin. This ensures compatibility with various vLLM features like distributed inference and the API server.
56
+```
docs/source/design/automatic_prefix_caching.md
renamed
+4
-2
@@ -1,6 +1,8 @@
1
-# Implementation
1
+(design-automatic-prefix-caching)=
2
3
-The core idea of PagedAttention is to partition the KV cache of each request into KV Blocks. Each block contains the attention keys and values for a fixed number of tokens. The PagedAttention algorithm allows these blocks to be stored in non-contiguous physical memory so that we can eliminate memory fragmentation by allocating the memory on demand.
3
+# Automatic Prefix Caching
4
+
5
+The core idea of [PagedAttention](#design-paged-attention) is to partition the KV cache of each request into KV Blocks. Each block contains the attention keys and values for a fixed number of tokens. The PagedAttention algorithm allows these blocks to be stored in non-contiguous physical memory so that we can eliminate memory fragmentation by allocating the memory on demand.
6
7
To automatically cache the KV cache, we utilize the following key observation: Each KV block can be uniquely identified by the tokens within the block and the tokens in the prefix before the block.
8
docs/source/design/kernel/paged_attention.md
+2
@@ -1,3 +1,5 @@
1
+(design-paged-attention)=
2
+
3
# vLLM Paged Attention
4
5
- Currently, vLLM utilizes its own implementation of a multi-head query
docs/source/dev/offline_inference/offline_index.md
+1
@@ -1,6 +1,7 @@
1
# Offline Inference
2
3
```{toctree}
4
+:caption: Contents
5
:maxdepth: 1
6
7
llm
docs/source/features/automatic_prefix_caching.md
renamed
+4
-4
@@ -1,13 +1,13 @@
1
-(apc)=
1
+(automatic-prefix-caching)=
2
3
-# Introduction
3
+# Automatic Prefix Caching
4
5
-## What is Automatic Prefix Caching
5
+## Introduction
6
7
Automatic Prefix Caching (APC in short) caches the KV cache of existing queries, so that a new query can directly reuse the KV cache if it shares the same prefix with one of the existing queries, allowing the new query to skip the computation of the shared part.
8
9
```{note}
10
-Technical details on how vLLM implements APC are in the next page.
10
+Technical details on how vLLM implements APC can be found [here](#design-automatic-prefix-caching).
11
```
12
13
## Enabling APC in vLLM
docs/source/features/compatibility_matrix.md
renamed
+3
-3
@@ -32,7 +32,7 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
32
33
* - Feature
34
- [CP](#chunked-prefill)
35
- - [APC](#apc)
35
+ - [APC](#automatic-prefix-caching)
36
- [LoRA](#lora-adapter)
37
- <abbr title="Prompt Adapter">prmpt adptr</abbr>
38
- [SD](#spec_decode)
@@ -64,7 +64,7 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
64
-
65
-
66
-
67
- * - [APC](#apc)
67
+ * - [APC](#automatic-prefix-caching)
68
- ✅
69
-
70
-
@@ -345,7 +345,7 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
345
- ✅
346
- ✅
347
- ✅
348
- * - [APC](#apc)
348
+ * - [APC](#automatic-prefix-caching)
349
- [✗](gh-issue:3687)
350
- ✅
351
- ✅
docs/source/features/disagg_prefill.md
renamed
+2
-2
@@ -41,13 +41,13 @@ Key abstractions for disaggregated prefilling:
41
42
Here is a figure illustrating how the above 3 abstractions are organized:
43
44
-```{image} /assets/usage/disagg_prefill/abstraction.jpg
44
+```{image} /assets/features/disagg_prefill/abstraction.jpg
45
:alt: Disaggregated prefilling abstractions
46
```
47
48
The workflow of disaggregated prefilling is as follows:
49
50
-```{image} /assets/usage/disagg_prefill/overview.jpg
50
+```{image} /assets/features/disagg_prefill/overview.jpg
51
:alt: Disaggregated prefilling workflow
52
```
53
docs/source/features/lora.md
renamed
docs/source/features/multimodal_inputs.md
renamed
docs/source/features/quantization/auto_awq.md
renamed
docs/source/features/quantization/bnb.md
renamed
docs/source/features/quantization/fp8.md
renamed
docs/source/features/quantization/fp8_e4m3_kvcache.md
renamed
docs/source/features/quantization/fp8_e5m2_kvcache.md
renamed
docs/source/features/quantization/gguf.md
renamed
docs/source/features/quantization/index.md
new
+19
@@ -0,0 +1,19 @@
1
+(quantization-index)=
2
+
3
+# Quantization
4
+
5
+Quantization trades off model precision for smaller memory footprint, allowing large models to be run on a wider range of devices.
6
+
7
+```{toctree}
8
+:caption: Contents
9
+:maxdepth: 1
10
+
11
+supported_hardware
12
+auto_awq
13
+bnb
14
+gguf
15
+int8
16
+fp8
17
+fp8_e5m2_kvcache
18
+fp8_e4m3_kvcache
19
+```
docs/source/features/quantization/int8.md
renamed
docs/source/features/quantization/supported_hardware.md
renamed
+5
-5
@@ -1,6 +1,6 @@
1
-(supported-hardware-for-quantization)=
1
+(quantization-supported-hardware)=
2
3
-# Supported Hardware for Quantization Kernels
3
+# Supported Hardware
4
5
The table below shows the compatibility of various quantization implementations with different hardware platforms in vLLM:
6
@@ -120,12 +120,12 @@ The table below shows the compatibility of various quantization implementations
120
- ✗
121
```
122
123
-## Notes:
124
-
123
- Volta refers to SM 7.0, Turing to SM 7.5, Ampere to SM 8.0/8.6, Ada to SM 8.9, and Hopper to SM 9.0.
124
- "✅︎" indicates that the quantization method is supported on the specified hardware.
125
- "✗" indicates that the quantization method is not supported on the specified hardware.
126
129
-Please note that this compatibility chart may be subject to change as vLLM continues to evolve and expand its support for different hardware platforms and quantization methods.
127
+```{note}
128
+This compatibility chart is subject to change as vLLM continues to evolve and expand its support for different hardware platforms and quantization methods.
129
130
For the most up-to-date information on hardware support and quantization methods, please refer to <gh-dir:vllm/model_executor/layers/quantization> or consult with the vLLM development team.
131
+```
docs/source/features/spec_decode.md
renamed
docs/source/features/structured_outputs.md
renamed
docs/source/features/tool_calling.md
renamed
docs/source/index.md
+22
-44
@@ -79,6 +79,9 @@ serving/metrics
79
serving/integrations
80
serving/tensorizer
81
serving/runai_model_streamer
82
+serving/engine_args
83
+serving/env_vars
84
+serving/usage_stats
85
```
86
87
```{toctree}
@@ -88,53 +91,28 @@ serving/runai_model_streamer
91
models/supported_models
92
models/generative_models
93
models/pooling_models
91
-models/adding_model
92
-models/enabling_multimodal_inputs
94
```
95
96
```{toctree}
96
-:caption: Usage
97
+:caption: Features
98
:maxdepth: 1
99
99
-usage/lora
100
-usage/multimodal_inputs
101
-usage/tool_calling
102
-usage/structured_outputs
103
-usage/spec_decode
104
-usage/compatibility_matrix
105
-usage/performance
106
-usage/engine_args
107
-usage/env_vars
108
-usage/usage_stats
109
-usage/disagg_prefill
110
-```
111
-
112
-```{toctree}
113
-:caption: Quantization
114
-:maxdepth: 1
115
-
116
-quantization/supported_hardware
117
-quantization/auto_awq
118
-quantization/bnb
119
-quantization/gguf
120
-quantization/int8
121
-quantization/fp8
122
-quantization/fp8_e5m2_kvcache
123
-quantization/fp8_e4m3_kvcache
124
-```
125
-
126
-```{toctree}
127
-:caption: Automatic Prefix Caching
128
-:maxdepth: 1
129
-
130
-automatic_prefix_caching/apc
131
-automatic_prefix_caching/details
100
+features/quantization/index
101
+features/lora
102
+features/multimodal_inputs
103
+features/tool_calling
104
+features/structured_outputs
105
+features/automatic_prefix_caching
106
+features/disagg_prefill
107
+features/spec_decode
108
+features/compatibility_matrix
109
```
110
111
```{toctree}
112
:caption: Performance
113
:maxdepth: 1
114
115
+performance/optimization
116
performance/benchmarks
117
```
118
@@ -148,10 +126,8 @@ community/meetups
126
community/sponsors
127
```
128
151
-% API Documentation: API reference aimed at vllm library usage
152
-
129
```{toctree}
154
-:caption: API Documentation
130
+:caption: API Reference
131
:maxdepth: 2
132
133
dev/sampling_params
@@ -160,30 +136,32 @@ dev/offline_inference/offline_index
136
dev/engine/engine_index
137
```
138
163
-% Design: docs about vLLM internals
139
+% Design Documents: Details about vLLM internals
140
141
```{toctree}
166
-:caption: Design
142
+:caption: Design Documents
143
:maxdepth: 2
144
145
design/arch_overview
146
design/huggingface_integration
147
design/plugin_system
172
-design/input_processing/model_inputs_index
148
design/kernel/paged_attention
149
+design/input_processing/model_inputs_index
150
design/multimodal/multimodal_index
151
+design/automatic_prefix_caching
152
design/multiprocessing
153
```
154
178
-% For Developers: contributing to the vLLM project
155
+% Developer Guide: How to contribute to the vLLM project
156
157
```{toctree}
181
-:caption: For Developers
158
+:caption: Developer Guide
159
:maxdepth: 2
160
161
contributing/overview
162
contributing/profiling/profiling_index
163
contributing/dockerfile/dockerfile
164
+contributing/model/index
165
```
166
167
# Indices and tables
docs/source/models/adding_model.md
deleted
-155
@@ -1,155 +0,0 @@
1
-(adding-a-new-model)=
2
-
3
-# Adding a New Model
4
-
5
-This document provides a high-level guide on integrating a [HuggingFace Transformers](https://github.com/huggingface/transformers) model into vLLM.
6
-
7
-```{note}
8
-The complexity of adding a new model depends heavily on the model's architecture.
9
-The process is considerably straightforward if the model shares a similar architecture with an existing model in vLLM.
10
-However, for models that include new operators (e.g., a new attention mechanism), the process can be a bit more complex.
11
-```
12
-
13
-```{note}
14
-By default, vLLM models do not support multi-modal inputs. To enable multi-modal support,
15
-please follow [this guide](#enabling-multimodal-inputs) after implementing the model here.
16
-```
17
-
18
-```{tip}
19
-If you are encountering issues while integrating your model into vLLM, feel free to open an issue on our [GitHub](https://github.com/vllm-project/vllm/issues) repository.
20
-We will be happy to help you out!
21
-```
22
-
23
-## 0. Fork the vLLM repository
24
-
25
-Start by forking our [GitHub] repository and then [build it from source](#build-from-source).
26
-This gives you the ability to modify the codebase and test your model.
27
-
28
-```{tip}
29
-If you don't want to fork the repository and modify vLLM's codebase, please refer to the "Out-of-Tree Model Integration" section below.
30
-```
31
-
32
-## 1. Bring your model code
33
-
34
-Clone the PyTorch model code from the HuggingFace Transformers repository and put it into the <gh-dir:vllm/model_executor/models> directory.
35
-For instance, vLLM's [OPT model](gh-file:vllm/model_executor/models/opt.py) was adapted from the HuggingFace's [modeling_opt.py](https://github.com/huggingface/transformers/blob/main/src/transformers/models/opt/modeling_opt.py) file.
36
-
37
-```{warning}
38
-When copying the model code, make sure to review and adhere to the code's copyright and licensing terms.
39
-```
40
-
41
-## 2. Make your code compatible with vLLM
42
-
43
-To ensure compatibility with vLLM, your model must meet the following requirements:
44
-
45
-### Initialization Code
46
-
47
-All vLLM modules within the model must include a `prefix` argument in their constructor. This `prefix` is typically the full name of the module in the model's state dictionary and is crucial for:
48
-
49
-- Runtime support: vLLM's attention operators are registered in a model's state by their full names. Each attention operator must have a unique prefix as its layer name to avoid conflicts.
50
-- Non-uniform quantization support: A quantized checkpoint can selectively quantize certain layers while keeping others in full precision. By providing the `prefix` during initialization, vLLM can match the current layer's `prefix` with the quantization configuration to determine if the layer should be initialized in quantized mode.
51
-
52
-The initialization code should look like this:
53
-
54
-```python
55
-from torch import nn
56
-from vllm.config import VllmConfig
57
-from vllm.attention import Attention
58
-
59
-class MyAttention(nn.Module):
60
- def __init__(self, vllm_config: VllmConfig, prefix: str):
61
- super().__init__()
62
- self.attn = Attention(prefix=f"{prefix}.attn")
63
-
64
-class MyDecoderLayer(nn.Module):
65
- def __init__(self, vllm_config: VllmConfig, prefix: str):
66
- super().__init__()
67
- self.self_attn = MyAttention(prefix=f"{prefix}.self_attn")
68
-
69
-class MyModel(nn.Module):
70
- def __init__(self, vllm_config: VllmConfig, prefix: str):
71
- super().__init__()
72
- self.layers = nn.ModuleList(
73
- [MyDecoderLayer(vllm_config, prefix=f"{prefix}.layers.{i}") for i in range(vllm_config.model_config.hf_config.num_hidden_layers)]
74
- )
75
-
76
-class MyModelForCausalLM(nn.Module):
77
- def __init__(self, vllm_config: VllmConfig, prefix: str = ""):
78
- super().__init__()
79
- self.model = MyModel(vllm_config, prefix=f"{prefix}.model")
80
-```
81
-
82
-### Computation Code
83
-
84
-Rewrite the {meth}`~torch.nn.Module.forward` method of your model to remove any unnecessary code, such as training-specific code. Modify the input parameters to treat `input_ids` and `positions` as flattened tensors with a single batch size dimension, without a max-sequence length dimension.
85
-
86
-```python
87
-def forward(
88
- self,
89
- input_ids: torch.Tensor,
90
- positions: torch.Tensor,
91
- kv_caches: List[torch.Tensor],
92
- attn_metadata: AttentionMetadata,
93
-) -> torch.Tensor:
94
- ...
95
-```
96
-
97
-```{note}
98
-Currently, vLLM supports the basic multi-head attention mechanism and its variant with rotary positional embeddings.
99
-If your model employs a different attention mechanism, you will need to implement a new attention layer in vLLM.
100
-```
101
-
102
-For reference, check out our [Llama implementation](gh-file:vllm/model_executor/models/llama.py). vLLM already supports a large number of models. It is recommended to find a model similar to yours and adapt it to your model's architecture. Check out <gh-dir:vllm/model_executor/models> for more examples.
103
-
104
-## 3. (Optional) Implement tensor parallelism and quantization support
105
-
106
-If your model is too large to fit into a single GPU, you can use tensor parallelism to manage it.
107
-To do this, substitute your model's linear and embedding layers with their tensor-parallel versions.
108
-For the embedding layer, you can simply replace {class}`torch.nn.Embedding` with {code}`VocabParallelEmbedding`. For the output LM head, you can use {code}`ParallelLMHead`.
109
-When it comes to the linear layers, we provide the following options to parallelize them:
110
-
111
-- {code}`ReplicatedLinear`: Replicates the inputs and weights across multiple GPUs. No memory saving.
112
-- {code}`RowParallelLinear`: The input tensor is partitioned along the hidden dimension. The weight matrix is partitioned along the rows (input dimension). An *all-reduce* operation is performed after the matrix multiplication to reduce the results. Typically used for the second FFN layer and the output linear transformation of the attention layer.
113
-- {code}`ColumnParallelLinear`: The input tensor is replicated. The weight matrix is partitioned along the columns (output dimension). The result is partitioned along the column dimension. Typically used for the first FFN layer and the separated QKV transformation of the attention layer in the original Transformer.
114
-- {code}`MergedColumnParallelLinear`: Column-parallel linear that merges multiple {code}`ColumnParallelLinear` operators. Typically used for the first FFN layer with weighted activation functions (e.g., SiLU). This class handles the sharded weight loading logic of multiple weight matrices.
115
-- {code}`QKVParallelLinear`: Parallel linear layer for the query, key, and value projections of the multi-head and grouped-query attention mechanisms. When number of key/value heads are less than the world size, this class replicates the key/value heads properly. This class handles the weight loading and replication of the weight matrices.
116
-
117
-Note that all the linear layers above take {code}`linear_method` as an input. vLLM will set this parameter according to different quantization schemes to support weight quantization.
118
-
119
-## 4. Implement the weight loading logic
120
-
121
-You now need to implement the {code}`load_weights` method in your {code}`*ForCausalLM` class.
122
-This method should load the weights from the HuggingFace's checkpoint file and assign them to the corresponding layers in your model. Specifically, for {code}`MergedColumnParallelLinear` and {code}`QKVParallelLinear` layers, if the original model has separated weight matrices, you need to load the different parts separately.
123
-
124
-## 5. Register your model
125
-
126
-Finally, register your {code}`*ForCausalLM` class to the {code}`_VLLM_MODELS` in <gh-file:vllm/model_executor/models/registry.py>.
127
-
128
-## 6. Out-of-Tree Model Integration
129
-
130
-You can integrate a model without modifying the vLLM codebase. Steps 2, 3, and 4 are still required, but you can skip steps 1 and 5. Instead, write a plugin to register your model. For general introduction of the plugin system, see [plugin-system](#plugin-system).
131
-
132
-To register the model, use the following code:
133
-
134
-```python
135
-from vllm import ModelRegistry
136
-from your_code import YourModelForCausalLM
137
-ModelRegistry.register_model("YourModelForCausalLM", YourModelForCausalLM)
138
-```
139
-
140
-If your model imports modules that initialize CUDA, consider lazy-importing it to avoid errors like {code}`RuntimeError: Cannot re-initialize CUDA in forked subprocess`:
141
-
142
-```python
143
-from vllm import ModelRegistry
144
-
145
-ModelRegistry.register_model("YourModelForCausalLM", "your_code:YourModelForCausalLM")
146
-```
147
-
148
-```{important}
149
-If your model is a multimodal model, ensure the model class implements the {class}`~vllm.model_executor.models.interfaces.SupportsMultiModal` interface.
150
-Read more about that [here](#enabling-multimodal-inputs).
151
-```
152
-
153
-```{note}
154
-Although you can directly put these code snippets in your script using `vllm.LLM`, the recommended way is to place these snippets in a vLLM plugin. This ensures compatibility with various vLLM features like distributed inference and the API server.
155
-```
docs/source/models/supported_models.md
+1
-1
@@ -37,7 +37,7 @@ print(output)
37
If vLLM successfully returns text (for generative models) or hidden states (for pooling models), it indicates that your model is supported.
38
````
39
40
-Otherwise, please refer to [Adding a New Model](#adding-a-new-model) and [Enabling Multimodal Inputs](#enabling-multimodal-inputs) for instructions on how to implement your model in vLLM.
40
+Otherwise, please refer to [Adding a New Model](#new-model) for instructions on how to implement your model in vLLM.
41
Alternatively, you can [open an issue on GitHub](https://github.com/vllm-project/vllm/issues/new/choose) to request vLLM support.
42
43
### ModelScope
docs/source/performance/optimization.md
renamed
+2
-2
@@ -1,6 +1,6 @@
1
-(performance)=
1
+(optimization-and-tuning)=
2
3
-# Performance and Tuning
3
+# Optimization and Tuning
4
5
## Preemption
6
docs/source/serving/engine_args.md
renamed
docs/source/serving/env_vars.md
renamed
docs/source/serving/openai_compatible_server.md
+1
-1
@@ -217,7 +217,7 @@ you can use the [official OpenAI Python client](https://github.com/openai/openai
217
218
We support both [Vision](https://platform.openai.com/docs/guides/vision)- and
219
[Audio](https://platform.openai.com/docs/guides/audio?audio-generation-quickstart-example=audio-in)-related parameters;
220
-see our [Multimodal Inputs](../usage/multimodal_inputs.md) guide for more information.
220
+see our [Multimodal Inputs](#multimodal-inputs) guide for more information.
221
- *Note: `image_url.detail` parameter is not supported.*
222
223
Code example: <gh-file:examples/openai_chat_completion_client.py>
docs/source/serving/usage_stats.md
renamed
vllm/attention/backends/rocm_flash_attn.py
+1
-1
@@ -430,7 +430,7 @@ class ROCmFlashAttentionImpl(AttentionImpl):
430
Returns:
431
shape = [num_tokens, num_heads * head_size]
432
"""
433
- # Reminder: Please update docs/source/usage/compatibility_matrix.md
433
+ # Reminder: Please update docs/source/features/compatibility_matrix.md
434
# If the feature combo become valid
435
if attn_type != AttentionType.DECODER:
436
raise NotImplementedError("Encoder self-attention and "
vllm/config.py
+3
-3
@@ -644,7 +644,7 @@ class ModelConfig:
644
self.use_async_output_proc = False
645
return
646
647
- # Reminder: Please update docs/source/usage/compatibility_matrix.md
647
+ # Reminder: Please update docs/source/features/compatibility_matrix.md
648
# If the feature combo become valid
649
from vllm.platforms import current_platform
650
if not current_platform.is_async_output_supported(self.enforce_eager):
@@ -665,7 +665,7 @@ class ModelConfig:
665
if self.runner_type == "pooling":
666
self.use_async_output_proc = False
667
668
- # Reminder: Please update docs/source/usage/compatibility_matrix.md
668
+ # Reminder: Please update docs/source/features/compatibility_matrix.md
669
# If the feature combo become valid
670
if speculative_config:
671
logger.warning("Async output processing is not supported with"
@@ -2064,7 +2064,7 @@ class LoRAConfig:
2064
model_config.quantization)
2065
2066
def verify_with_scheduler_config(self, scheduler_config: SchedulerConfig):
2067
- # Reminder: Please update docs/source/usage/compatibility_matrix.md
2067
+ # Reminder: Please update docs/source/features/compatibility_matrix.md
2068
# If the feature combo become valid
2069
if scheduler_config.chunked_prefill_enabled:
2070
logger.warning("LoRA with chunked prefill is still experimental "
vllm/engine/arg_utils.py
+1
-1
@@ -1148,7 +1148,7 @@ class EngineArgs:
1148
disable_logprobs=self.disable_logprobs_during_spec_decoding,
1149
)
1150
1151
- # Reminder: Please update docs/source/usage/compatibility_matrix.md
1151
+ # Reminder: Please update docs/source/features/compatibility_matrix.md
1152
# If the feature combo become valid
1153
if self.num_scheduler_steps > 1:
1154
if speculative_config is not None:
vllm/engine/output_processor/multi_step.py
+1
-1
@@ -65,7 +65,7 @@ class MultiStepOutputProcessor(SequenceGroupOutputProcessor):
65
@staticmethod
66
@functools.lru_cache
67
def _log_prompt_logprob_unsupported_warning_once():
68
- # Reminder: Please update docs/source/usage/compatibility_matrix.md
68
+ # Reminder: Please update docs/source/features/compatibility_matrix.md
69
# If the feature combo become valid
70
logger.warning(
71
"Prompt logprob is not supported by multi step workers. "
vllm/executor/cpu_executor.py
+1
-1
@@ -22,7 +22,7 @@ class CPUExecutor(ExecutorBase):
22
23
def _init_executor(self) -> None:
24
assert self.device_config.device_type == "cpu"
25
- # Reminder: Please update docs/source/usage/compatibility_matrix.md
25
+ # Reminder: Please update docs/source/features/compatibility_matrix.md
26
# If the feature combo become valid
27
assert self.lora_config is None, "cpu backend doesn't support LoRA"
28
vllm/platforms/cpu.py
+1
-1
@@ -50,7 +50,7 @@ class CpuPlatform(Platform):
50
import vllm.envs as envs
51
from vllm.utils import GiB_bytes
52
model_config = vllm_config.model_config
53
- # Reminder: Please update docs/source/usage/compatibility_matrix.md
53
+ # Reminder: Please update docs/source/features/compatibility_matrix.md
54
# If the feature combo become valid
55
if not model_config.enforce_eager:
56
logger.warning(
vllm/spec_decode/spec_decode_worker.py
+1
-1
@@ -108,7 +108,7 @@ def create_spec_worker(*args, **kwargs) -> "SpecDecodeWorker":
108
return spec_decode_worker
109
110
111
-# Reminder: Please update docs/source/usage/compatibility_matrix.md
111
+# Reminder: Please update docs/source/features/compatibility_matrix.md
112
# If the feature combo become valid
113
class SpecDecodeWorker(LoraNotSupportedWorkerBase):
114
"""Worker which implements speculative decoding.
vllm/utils.py
+1
-1
@@ -58,7 +58,7 @@ logger = init_logger(__name__)
58
59
# Exception strings for non-implemented encoder/decoder scenarios
60
61
-# Reminder: Please update docs/source/usage/compatibility_matrix.md
61
+# Reminder: Please update docs/source/features/compatibility_matrix.md
62
# If the feature combo become valid
63
64
STR_NOT_IMPL_ENC_DEC_SWA = \
vllm/worker/multi_step_model_runner.py
+1
-1
@@ -822,7 +822,7 @@ def _pythonize_sampler_output(
822
823
for sgdx, (seq_group,
824
sample_result) in enumerate(zip(seq_groups, samples_list)):
825
- # Reminder: Please update docs/source/usage/compatibility_matrix.md
825
+ # Reminder: Please update docs/source/features/compatibility_matrix.md
826
# If the feature combo become valid
827
# (Check for Guided Decoding)
828
if seq_group.sampling_params.logits_processors:
vllm/worker/utils.py
+1
-1
@@ -13,7 +13,7 @@ def assert_enc_dec_mr_supported_scenario(
13
a supported scenario.
14
'''
15
16
- # Reminder: Please update docs/source/usage/compatibility_matrix.md
16
+ # Reminder: Please update docs/source/features/compatibility_matrix.md
17
# If the feature combo become valid
18
19
if enc_dec_mr.cache_config.enable_prefix_caching: