A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 6, 09:30 UTC
[PluggableLayer][MM] Add PluggableLayer for CustomQwen2Decoder (#37293)
Signed-off-by: Wangbei25 <wangbei41@huawie.com> Signed-off-by: Wangbei25 <wangbei41@huawei.com> Co-authored-by: Wangbei25 <wangbei41@huawie.com>
Wangbei25 committed
Mar 20, 2026 at 14:24 UTC
0674d1fee76102e05db6ebb415952e51a6cf81a8
2 files changed
+17
-4
docs/design/custom_op.md
+10
-3
@@ -51,11 +51,8 @@ For example:
51
**1. Attention:**
52
53
```python
54
---8<-- "vllm/model_executor/layers/attention/mm_encoder_attention.py:mm_encoder_attn"
55
-
54
--8<-- "vllm/model_executor/layers/mla.py:multi_head_latent_attention"
55
58
---8<-- "vllm/model_executor/models/deepencoder.py:rel_pos_attention"
56
```
57
58
**2. Activation:**
@@ -170,6 +167,16 @@ For example:
167
--8<-- "vllm/model_executor/layers/rotary_embedding/common.py:apply_rotary_emb"
168
```
169
170
+**12. Encoder:**
171
+
172
+```python
173
+--8<-- "vllm/model_executor/models/deepencoder2.py:qwen2_decoder"
174
+
175
+--8<-- "vllm/model_executor/layers/attention/mm_encoder_attention.py:mm_encoder_attn"
176
+
177
+--8<-- "vllm/model_executor/models/deepencoder.py:rel_pos_attention"
178
+```
179
+
180
## Guidelines for Implementing a New CustomOp
181
182
### Implement a New CustomOp in vLLM
vllm/model_executor/models/deepencoder2.py
+7
-1
@@ -14,14 +14,20 @@ import torch
14
import torch.nn as nn
15
import transformers
16
17
+from vllm.model_executor.custom_op import PluggableLayer
18
18
-class CustomQwen2Decoder(nn.Module):
19
+
20
+# --8<-- [start:qwen2_decoder]
21
+@PluggableLayer.register("qwen2_decoder")
22
+class CustomQwen2Decoder(PluggableLayer):
23
"""
24
Qwen2 visual encoder
25
non-causal attention + causal attention
26
token_type_ids ：0=non-causal, 1=causal
27
"""
28
29
+ # --8<-- [end:qwen2_decoder]
30
+
31
def __init__(
32
self,
33
decoder_layer: int = 24,