A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 16:00 UTC
Rename fallback model and refactor supported models section (#15829)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Harry Mellor committed
Apr 1, 2025 at 06:49 UTC
a76f547e11ebfc1a5cbe1e5e167e2c5c5691a081
6 files changed
+80
-60
docs/source/index.md
+1
-1
@@ -77,9 +77,9 @@ getting_started/v1_user_guide
77
:caption: Models
78
:maxdepth: 1
79
80
+models/supported_models
81
models/generative_models
82
models/pooling_models
82
-models/supported_models
83
models/extensions/index
84
:::
85
docs/source/models/supported_models.md
+70
-49
@@ -1,55 +1,28 @@
1
(supported-models)=
2
3
-# List of Supported Models
3
+# Supported Models
4
5
-vLLM supports generative and pooling models across various tasks.
5
+vLLM supports [generative](generative-models) and [pooling](pooling-models) models across various tasks.
6
If a model supports more than one task, you can set the task via the `--task` argument.
7
8
For each task, we list the model architectures that have been implemented in vLLM.
9
Alongside each architecture, we include some popular models that use it.
10
11
-## Loading a Model
12
-
13
-### HuggingFace Hub
14
-
15
-By default, vLLM loads models from [HuggingFace (HF) Hub](https://huggingface.co/models).
16
-
17
-To determine whether a given model is natively supported, you can check the `config.json` file inside the HF repository.
18
-If the `"architectures"` field contains a model architecture listed below, then it should be natively supported.
19
-
20
-Models do not _need_ to be natively supported to be used in vLLM.
21
-The <project:#transformers-fallback> enables you to run models directly using their Transformers implementation (or even remote code on the Hugging Face Model Hub!).
22
-
23
-:::{tip}
24
-The easiest way to check if your model is really supported at runtime is to run the program below:
25
-
26
-```python
27
-from vllm import LLM
28
-
29
-# For generative models (task=generate) only
30
-llm = LLM(model=..., task="generate") # Name or path of your model
31
-output = llm.generate("Hello, my name is")
32
-print(output)
11
+## Model Implementation
12
34
-# For pooling models (task={embed,classify,reward,score}) only
35
-llm = LLM(model=..., task="embed") # Name or path of your model
36
-output = llm.encode("Hello, my name is")
37
-print(output)
38
-```
13
+### vLLM
14
40
-If vLLM successfully returns text (for generative models) or hidden states (for pooling models), it indicates that your model is supported.
41
-:::
15
+If vLLM natively supports a model, its implementation can be found in <gh-file:vllm/model_executor/models>.
16
43
-Otherwise, please refer to [Adding a New Model](#new-model) for instructions on how to implement your model in vLLM.
44
-Alternatively, you can [open an issue on GitHub](https://github.com/vllm-project/vllm/issues/new/choose) to request vLLM support.
17
+These models are what we list in <project:#supported-text-models> and <project:#supported-mm-models>.
18
46
-(transformers-fallback)=
19
+(transformers-backend)=
20
48
-### Transformers fallback
21
+### Transformers
22
50
-vLLM can fallback to model implementations that are available in Transformers. This does not work for all models for now, but most decoder language models are supported, and vision language model support is planned!
23
+vLLM also supports model implementations that are available in Transformers. This does not currently work for all models, but most decoder language models are supported, and vision language model support is planned!
24
52
-To check if the backend is Transformers, you can simply do this:
25
+To check if the modeling backend is Transformers, you can simply do this:
26
27
```python
28
from vllm import LLM
@@ -69,16 +42,15 @@ vLLM may not fully optimise the Transformers implementation so you may see degra
42
43
#### Supported features
44
72
-The Transformers fallback explicitly supports the following features:
45
+The Transformers modeling backend explicitly supports the following features:
46
47
- <project:#quantization-index> (except GGUF)
48
- <project:#lora-adapter>
49
- <project:#distributed-serving>
50
78
-#### Remote code
51
+#### Remote Code
52
80
-Earlier we mentioned that the Transformers fallback enables you to run remote code models directly in vLLM.
81
-If you are interested in this feature, this section is for you!
53
+If your model is neither supported natively by vLLM or Transformers, you can still run it in vLLM!
54
55
Simply set `trust_remote_code=True` and vLLM will run any model on the Model Hub that is compatible with Transformers.
56
Provided that the model writer implements their model in a compatible way, this means that you can run new models before they are officially supported in Transformers or vLLM!
@@ -89,7 +61,7 @@ llm = LLM(model=..., task="generate", trust_remote_code=True) # Name or path of
61
llm.apply_model(lambda model: print(model.__class__))
62
```
63
92
-To make your model compatible with the Transformers fallback, it needs:
64
+To make your model compatible with the Transformers backend, it needs:
65
66
```{code-block} python
67
:caption: modeling_my_model.py
@@ -121,7 +93,9 @@ Here is what happens in the background:
93
2. `MyModel` Python class is loaded from the `auto_map`, and we check that the model `_supports_attention_backend`.
94
3. The `TransformersForCausalLM` backend is used. See <gh-file:vllm/model_executor/models/transformers.py>, which leverage `self.config._attn_implementation = "vllm"`, thus the need to use `ALL_ATTENTION_FUNCTION`.
95
124
-To make your model compatible with tensor parallel, it needs:
96
+That's it!
97
+
98
+For your model to be compatible with vLLM's tensor parallel and/or pipeline parallel features, you must add `base_model_tp_plan` and/or `base_model_pp_plan` to your model's config class:
99
100
```{code-block} python
101
:caption: configuration_my_model.py
@@ -130,20 +104,65 @@ from transformers import PretrainedConfig
104
105
class MyConfig(PretrainedConfig):
106
base_model_tp_plan = {
133
- "layers.*.self_attn.q_proj": "colwise",
134
- ...
107
+ "layers.*.self_attn.k_proj": "colwise",
108
+ "layers.*.self_attn.v_proj": "colwise",
109
+ "layers.*.self_attn.o_proj": "rowwise",
110
+ "layers.*.mlp.gate_proj": "colwise",
111
+ "layers.*.mlp.up_proj": "colwise",
112
+ "layers.*.mlp.down_proj": "rowwise",
113
+ }
114
+ base_model_pp_plan = {
115
+ "embed_tokens": (["input_ids"], ["inputs_embeds"]),
116
+ "layers": (["hidden_states", "attention_mask"], ["hidden_states"]),
117
+ "norm": (["hidden_states"], ["hidden_states"]),
118
}
119
```
120
121
+- `base_model_tp_plan` is a `dict` that maps fully qualified layer name patterns to tensor parallel styles (currently only `"colwise"` and `"rowwise"` are supported).
122
+- `base_model_pp_plan` is a `dict` that maps direct child layer names to `tuple`s of `list`s of `str`s:
123
+ * You only need to do this for layers which are not present on all pipeline stages
124
+ * vLLM assumes that there will be only one `nn.ModuleList`, which is distributed across the pipeline stages
125
+ * The `list` in the first element of the `tuple` contains the names of the input arguments
126
+ * The `list` in the last element of the `tuple` contains the names of the variables the layer outputs to in your modeling code
127
+
128
+## Loading a Model
129
+
130
+### Hugging Face Hub
131
+
132
+By default, vLLM loads models from [Hugging Face (HF) Hub](https://huggingface.co/models).
133
+
134
+To determine whether a given model is natively supported, you can check the `config.json` file inside the HF repository.
135
+If the `"architectures"` field contains a model architecture listed below, then it should be natively supported.
136
+
137
+Models do not _need_ to be natively supported to be used in vLLM.
138
+The <project:#transformers-backend> enables you to run models directly using their Transformers implementation (or even remote code on the Hugging Face Model Hub!).
139
+
140
:::{tip}
139
-`base_model_tp_plan` is a `dict` that maps fully qualified layer name patterns to tensor parallel styles (currently only `"colwise"` and `"rowwise"` are supported).
141
+The easiest way to check if your model is really supported at runtime is to run the program below:
142
+
143
+```python
144
+from vllm import LLM
145
+
146
+# For generative models (task=generate) only
147
+llm = LLM(model=..., task="generate") # Name or path of your model
148
+output = llm.generate("Hello, my name is")
149
+print(output)
150
+
151
+# For pooling models (task={embed,classify,reward,score}) only
152
+llm = LLM(model=..., task="embed") # Name or path of your model
153
+output = llm.encode("Hello, my name is")
154
+print(output)
155
+```
156
+
157
+If vLLM successfully returns text (for generative models) or hidden states (for pooling models), it indicates that your model is supported.
158
:::
159
142
-That's it!
160
+Otherwise, please refer to [Adding a New Model](#new-model) for instructions on how to implement your model in vLLM.
161
+Alternatively, you can [open an issue on GitHub](https://github.com/vllm-project/vllm/issues/new/choose) to request vLLM support.
162
163
### ModelScope
164
146
-To use models from [ModelScope](https://www.modelscope.cn) instead of HuggingFace Hub, set an environment variable:
165
+To use models from [ModelScope](https://www.modelscope.cn) instead of Hugging Face Hub, set an environment variable:
166
167
```shell
168
export VLLM_USE_MODELSCOPE=True
@@ -165,6 +184,8 @@ output = llm.encode("Hello, my name is")
184
print(output)
185
```
186
187
+(supported-text-models)=
188
+
189
## List of Text-only Language Models
190
191
### Generative Models
@@ -1066,7 +1087,7 @@ At vLLM, we are committed to facilitating the integration and support of third-p
1087
2. **Best-Effort Consistency**: While we aim to maintain a level of consistency between the models implemented in vLLM and other frameworks like transformers, complete alignment is not always feasible. Factors like acceleration techniques and the use of low-precision computations can introduce discrepancies. Our commitment is to ensure that the implemented models are functional and produce sensible results.
1088
1089
:::{tip}
1069
- When comparing the output of `model.generate` from HuggingFace Transformers with the output of `llm.generate` from vLLM, note that the former reads the model's generation config file (i.e., [generation_config.json](https://github.com/huggingface/transformers/blob/19dabe96362803fb0a9ae7073d03533966598b17/src/transformers/generation/utils.py#L1945)) and applies the default parameters for generation, while the latter only uses the parameters passed to the function. Ensure all sampling parameters are identical when comparing outputs.
1090
+ When comparing the output of `model.generate` from Hugging Face Transformers with the output of `llm.generate` from vLLM, note that the former reads the model's generation config file (i.e., [generation_config.json](https://github.com/huggingface/transformers/blob/19dabe96362803fb0a9ae7073d03533966598b17/src/transformers/generation/utils.py#L1945)) and applies the default parameters for generation, while the latter only uses the parameters passed to the function. Ensure all sampling parameters are identical when comparing outputs.
1091
:::
1092
1093
3. **Issue Resolution and Model Updates**: Users are encouraged to report any bugs or issues they encounter with third-party models. Proposed fixes should be submitted via PRs, with a clear explanation of the problem and the rationale behind the proposed solution. If a fix for one model impacts another, we rely on the community to highlight and address these cross-model dependencies. Note: for bugfix PRs, it is good etiquette to inform the original author to seek their feedback.
tests/models/registry.py
+2
-2
@@ -346,7 +346,7 @@ _SPECULATIVE_DECODING_EXAMPLE_MODELS = {
346
trust_remote_code=True),
347
}
348
349
-_FALLBACK_MODEL = {
349
+_TRANSFORMERS_MODELS = {
350
"TransformersForCausalLM": _HfExamplesInfo("ArthurZ/Ilama-3.2-1B", trust_remote_code=True), # noqa: E501
351
}
352
@@ -356,7 +356,7 @@ _EXAMPLE_MODELS = {
356
**_CROSS_ENCODER_EXAMPLE_MODELS,
357
**_MULTIMODAL_EXAMPLE_MODELS,
358
**_SPECULATIVE_DECODING_EXAMPLE_MODELS,
359
- **_FALLBACK_MODEL,
359
+ **_TRANSFORMERS_MODELS,
360
}
361
362
vllm/model_executor/model_loader/utils.py
+3
-4
@@ -39,8 +39,8 @@ def is_transformers_impl_compatible(
39
return mod.is_backend_compatible()
40
41
42
-def resolve_transformers_fallback(model_config: ModelConfig,
43
- architectures: list[str]):
42
+def resolve_transformers_arch(model_config: ModelConfig,
43
+ architectures: list[str]):
44
for i, arch in enumerate(architectures):
45
if arch == "TransformersForCausalLM":
46
continue
@@ -101,8 +101,7 @@ def get_model_architecture(
101
for arch in architectures)
102
if (not is_vllm_supported
103
or model_config.model_impl == ModelImpl.TRANSFORMERS):
104
- architectures = resolve_transformers_fallback(model_config,
105
- architectures)
104
+ architectures = resolve_transformers_arch(model_config, architectures)
105
106
model_cls, arch = ModelRegistry.resolve_model_cls(architectures)
107
if model_config.task == "embed":
vllm/model_executor/models/registry.py
+3
-3
@@ -202,7 +202,7 @@ _SPECULATIVE_DECODING_MODELS = {
202
"MLPSpeculatorPreTrainedModel": ("mlp_speculator", "MLPSpeculator"),
203
}
204
205
-_FALLBACK_MODEL = {
205
+_TRANSFORMERS_MODELS = {
206
"TransformersForCausalLM": ("transformers", "TransformersForCausalLM"),
207
}
208
# yapf: enable
@@ -213,7 +213,7 @@ _VLLM_MODELS = {
213
**_CROSS_ENCODER_MODELS,
214
**_MULTIMODAL_MODELS,
215
**_SPECULATIVE_DECODING_MODELS,
216
- **_FALLBACK_MODEL,
216
+ **_TRANSFORMERS_MODELS,
217
}
218
219
# This variable is used as the args for subprocess.run(). We
@@ -427,7 +427,7 @@ class _ModelRegistry:
427
normalized_arch = list(
428
filter(lambda model: model in self.models, architectures))
429
430
- # make sure Transformers fallback are put at the last
430
+ # make sure Transformers backend is put at the last as a fallback
431
if len(normalized_arch) != len(architectures):
432
normalized_arch.append("TransformersForCausalLM")
433
return normalized_arch
vllm/model_executor/models/transformers.py
+1
-1
@@ -401,7 +401,7 @@ class TransformersForCausalLM(nn.Module, SupportsQuant, SupportsLoRA,
401
self.make_empty_intermediate_tensors = (
402
self.model.make_empty_intermediate_tensors)
403
404
- # FIXME(Isotr0py): Don't use any weights mapper for Transformers fallback,
404
+ # FIXME(Isotr0py): Don't use any weights mapper for Transformers backend,
405
# this makes thing complicated. We need to remove this mapper after refactor
406
# `TransformersModel` in the future.
407
@property