A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 05:30 UTC
Encoder model support for the Transformers backend (#25174)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Harry Mellor committed
Sep 19, 2025 at 19:15 UTC
12aed7e453aea713495bd7cced6f9e2bb78aaa79
4 files changed
+111
-24
docs/models/supported_models.md
+25
-12
@@ -17,9 +17,24 @@ These models are what we list in [supported-text-models][supported-text-models]
17
18
### Transformers
19
20
-vLLM also supports model implementations that are available in Transformers. This does not currently work for all models, but most decoder language models and common vision language models are supported! Vision-language models currently accept only image inputs. Support for video inputs will be added in future releases.
20
+vLLM also supports model implementations that are available in Transformers. You should expect the performance of a Transformers model implementation used in vLLM to be within <1% of the performance of a dedicated vLLM model implementation. We call this feature the "Transformers backend".
21
22
-To check if the modeling backend is Transformers, you can simply do this:
22
+Currently, the Transformers backend works for the following:
23
+
24
+- Modalities: embedding models, language models and vision-language models*
25
+- Architectures: encoder-only, decoder-only
26
+- Attention types: full attention and/or sliding attention
27
+
28
+_*Vision-language models currently accept only image inputs. Support for video inputs will be added in a future release._
29
+
30
+If the Transformers model implementation follows all the steps in [writing a custom model](#writing-custom-models) then, when used with the Transformers backend, it will be compatible with the following features of vLLM:
31
+
32
+- All the features listed in the [compatibility matrix](../features/compatibility_matrix.md#feature-x-feature)
33
+- Any combination of the following vLLM parallelisation schemes:
34
+ - Pipeline parallel
35
+ - Tensor parallel
36
+
37
+Checking if the modeling backend is Transformers is as simple as:
38
39
```python
40
from vllm import LLM
@@ -27,16 +42,12 @@ llm = LLM(model=...) # Name or path of your model
42
llm.apply_model(lambda model: print(type(model)))
43
```
44
30
-If it is `TransformersForCausalLM` or `TransformersForMultimodalLM` then it means it's based on Transformers!
45
+If the printed type starts with `Transformers...` then it's using the Transformers model implementation!
46
32
-!!! tip
33
- You can force the use of `TransformersForCausalLM` by setting `model_impl="transformers"` for [offline-inference](../serving/offline_inference.md) or `--model-impl transformers` for the [openai-compatible-server](../serving/openai_compatible_server.md).
34
-
35
-!!! note
36
- vLLM may not fully optimise the Transformers implementation so you may see degraded performance if comparing a native model to a Transformers model in vLLM.
47
+If a model has a vLLM implementation but you would prefer to use the Transformers implementation via the Transformers backend, set `model_impl="transformers"` for [offline inference](../serving/offline_inference.md) or `--model-impl transformers` for the [online serving](../serving/openai_compatible_server.md).
48
49
!!! note
39
- In case of vision language models if you are loading with `dtype="auto"`, vLLM loads the whole model with config's `dtype` if it exists. In contrast the native Transformers will respect the `dtype` attribute of each backbone in the model. That might cause a slight difference in performance.
50
+ For vision-language models, if you are loading with `dtype="auto"`, vLLM loads the whole model with config's `dtype` if it exists. In contrast the native Transformers will respect the `dtype` attribute of each backbone in the model. That might cause a slight difference in performance.
51
52
#### Custom models
53
@@ -66,10 +77,11 @@ This section details the necessary modifications to make to a Transformers compa
77
To make your model compatible with the Transformers backend, it needs:
78
79
1. `kwargs` passed down through all modules from `MyModel` to `MyAttention`.
80
+ 1. If your model is encoder-only, you must also add `is_causal = False` to `MyAttention`.
81
2. `MyAttention` must use `ALL_ATTENTION_FUNCTIONS` to call attention.
82
3. `MyModel` must contain `_supports_attention_backend = True`.
83
72
-<details>
84
+<details class="code">
85
<summary>modeling_my_model.py</summary>
86
87
```python
@@ -78,6 +90,7 @@ from transformers import PreTrainedModel
90
from torch import nn
91
92
class MyAttention(nn.Module):
93
+ is_causal = False # Only do this for encoder-only models
94
95
def forward(self, hidden_states, **kwargs):
96
...
@@ -101,13 +114,13 @@ Here is what happens in the background when this model is loaded:
114
115
1. The config is loaded.
116
2. `MyModel` Python class is loaded from the `auto_map` in config, and we check that the model `is_backend_compatible()`.
104
-3. `MyModel` is loaded into `TransformersForCausalLM` or `TransformersForMultimodalLM` (see <gh-file:vllm/model_executor/models/transformers.py>) which sets `self.config._attn_implementation = "vllm"` so that vLLM's attention layer is used.
117
+3. `MyModel` is loaded into one of the Transformers backend classes in <gh-file:vllm/model_executor/models/transformers.py> which sets `self.config._attn_implementation = "vllm"` so that vLLM's attention layer is used.
118
119
That's it!
120
121
For your model to be compatible with vLLM's tensor parallel and/or pipeline parallel features, you must add `base_model_tp_plan` and/or `base_model_pp_plan` to your model's config class:
122
110
-<details>
123
+<details class="code">
124
<summary>configuration_my_model.py</summary>
125
126
```python
tests/models/test_transformers.py
+35
-1
@@ -9,7 +9,7 @@ from vllm.platforms import current_platform
9
10
from ..conftest import HfRunner, VllmRunner
11
from ..utils import multi_gpu_test, prep_prompts
12
-from .utils import check_logprobs_close
12
+from .utils import check_embeddings_close, check_logprobs_close
13
14
15
def check_implementation(
@@ -165,6 +165,40 @@ def test_embed_loading(vllm_runner, model):
165
assert model_config.using_transformers_backend()
166
167
168
+@pytest.mark.parametrize(
169
+ "model",
170
+ [
171
+ # Encoder model
172
+ "BAAI/bge-base-en-v1.5",
173
+ ])
174
+def test_embed_correctness(hf_runner, vllm_runner, example_prompts, model):
175
+ import transformers
176
+ from packaging.version import Version
177
+ installed = Version(transformers.__version__)
178
+ required = Version("4.57.0.dev0")
179
+ if installed < required:
180
+ pytest.skip("Encoder models with the Transformers backend require "
181
+ f"transformers>={required}, but got {installed}")
182
+
183
+ with vllm_runner(model, max_model_len=512,
184
+ model_impl="transformers") as vllm_model:
185
+ model_config = vllm_model.llm.llm_engine.model_config
186
+ assert model_config.using_transformers_backend()
187
+
188
+ vllm_outputs = vllm_model.embed(example_prompts)
189
+
190
+ with hf_runner(model, is_sentence_transformer=True) as hf_model:
191
+ hf_outputs = hf_model.encode(example_prompts)
192
+
193
+ check_embeddings_close(
194
+ embeddings_0_lst=hf_outputs,
195
+ embeddings_1_lst=vllm_outputs,
196
+ name_0="hf",
197
+ name_1="vllm",
198
+ tol=1e-2,
199
+ )
200
+
201
+
202
@pytest.mark.parametrize(
203
"model",
204
["jason9693/Qwen2.5-1.5B-apeach"],
vllm/attention/backends/abstract.py
+4
-4
@@ -23,14 +23,14 @@ class AttentionType:
23
Attention type.
24
Use string to be compatible with `torch.compile`.
25
"""
26
- # Decoder attention between previous layer Q/K/V
26
DECODER = "decoder"
28
- # Encoder attention between previous layer Q/K/V for encoder-decoder
27
+ """Decoder attention between previous layer Q/K/V."""
28
ENCODER = "encoder"
30
- # Encoder attention between previous layer Q/K/V
29
+ """Encoder attention between previous layer Q/K/V for encoder-decoder."""
30
ENCODER_ONLY = "encoder_only"
32
- # Attention between dec. Q and enc. K/V for encoder-decoder
31
+ """Encoder attention between previous layer Q/K/V."""
32
ENCODER_DECODER = "encoder_decoder"
33
+ """Attention between dec. Q and enc. K/V for encoder-decoder."""
34
35
36
class AttentionBackend(ABC):
vllm/model_executor/models/transformers.py
+47
-7
@@ -27,7 +27,7 @@ from transformers import (AutoModel, BatchFeature, PretrainedConfig,
27
PreTrainedModel)
28
from transformers.modeling_utils import ALL_ATTENTION_FUNCTIONS
29
30
-from vllm.attention import Attention
30
+from vllm.attention import Attention, AttentionType
31
from vllm.compilation.decorators import support_torch_compile
32
from vllm.config import (CacheConfig, DeviceConfig, ModelConfig,
33
ParallelConfig, VllmConfig)
@@ -452,8 +452,9 @@ class TransformersBase(nn.Module, SupportsQuant, SupportsLoRA, SupportsPP):
452
self.pp_rank = self.pp_group.rank_in_group
453
self.tp_size = get_tensor_model_parallel_world_size()
454
455
- # To be updated in child classes for use in `load_weights`
456
- self.skip_prefixes: Optional[list[str]] = None
455
+ # Weights to skip in `self.load_weights`
456
+ self.skip_prefixes: list[str] = []
457
+ self.skip_substrs: list[str] = []
458
459
# Set correct attn and init on "meta" to delay allocating GPU tensors
460
# TODO: @raushan, use the public `model.set_attn_implementation()`
@@ -596,7 +597,10 @@ class TransformersBase(nn.Module, SupportsQuant, SupportsLoRA, SupportsPP):
597
598
_tensor_parallel(self.model)
599
599
- def create_attention_instances(self) -> dict[int, Attention]:
600
+ def create_attention_instances(
601
+ self,
602
+ attn_type: AttentionType = AttentionType.DECODER
603
+ ) -> dict[int, Attention]:
604
"""
605
Create `Attention` instances to inform KV cache allocation.
606
"""
@@ -625,7 +629,8 @@ class TransformersBase(nn.Module, SupportsQuant, SupportsLoRA, SupportsPP):
629
cache_config=self.cache_config,
630
quant_config=self.quant_config,
631
per_layer_sliding_window=per_layer_sliding_window,
628
- prefix=f"{i}.attn")
632
+ prefix=f"{i}.attn",
633
+ attn_type=attn_type)
634
return attention_instances
635
636
def init_parameters(self, module: nn.Module):
@@ -685,7 +690,11 @@ class TransformersBase(nn.Module, SupportsQuant, SupportsLoRA, SupportsPP):
690
691
def load_weights(self, weights: Iterable[tuple[str,
692
torch.Tensor]]) -> set[str]:
688
- loader = AutoWeightsLoader(self, skip_prefixes=self.skip_prefixes)
693
+ loader = AutoWeightsLoader(
694
+ self,
695
+ skip_prefixes=self.skip_prefixes,
696
+ skip_substrs=self.skip_substrs,
697
+ )
698
return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
699
700
@@ -700,6 +709,37 @@ class TransformersModel(TransformersBase):
709
"model.score": "score",
710
})
711
712
+ def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
713
+ super().__init__(vllm_config=vllm_config, prefix=prefix)
714
+
715
+ # Some encoder models have the position_ids buffer in the checkpoint
716
+ # vLLM will always pass position_ids as an argument, so we skip loading
717
+ # the buffer if it exists
718
+ self.skip_substrs.append("position_ids")
719
+
720
+ def create_attention_instances(
721
+ self, attn_type: AttentionType = AttentionType.DECODER):
722
+ # TODO(hmellor): Better way to detect encoder models
723
+ # In encoder models, the attention layers will have `is_causal=False`
724
+ is_encoder = lambda m: not getattr(m, "is_causal", True)
725
+ # vLLM does not support encoder-decoder models, so if any encoder layer
726
+ # is found, we assume the whole model is an encoder model
727
+ if any(is_encoder(m) for m in self.model.modules()):
728
+ attn_type = AttentionType.ENCODER_ONLY
729
+
730
+ # Check minimum transformers version for encoder models support
731
+ if attn_type == AttentionType.ENCODER_ONLY:
732
+ import transformers
733
+ from packaging.version import Version
734
+ installed = Version(transformers.__version__)
735
+ required = Version("4.57.0.dev0")
736
+ if installed < required:
737
+ raise ValueError(
738
+ "Encoder models with the Transformers backend require "
739
+ f"transformers>={required}, but got {installed}")
740
+
741
+ return super().create_attention_instances(attn_type)
742
+
743
744
@support_torch_compile(enable_if=can_enable_torch_compile)
745
class TransformersForCausalLM(TransformersBase):
@@ -710,7 +750,7 @@ class TransformersForCausalLM(TransformersBase):
750
# Tell `TransformersBase.load_weights` to skip
751
# `lm_head` if the model has tied word embeddings
752
if self.text_config.tie_word_embeddings:
713
- self.skip_prefixes = ["lm_head."]
753
+ self.skip_prefixes.append("lm_head.")
754
755
if get_pp_group().is_last_rank:
756
self.unpadded_vocab_size = self.text_config.vocab_size