@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 5f3cd7f7f2

Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 13, 04:30 UTC

[Docs] Update the name of `Transformers backend` -> `Transformers modeling backend` (#28725)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Harry Mellor committed Nov 14, 2025 at 16:34 UTC 5f3cd7f7f20a8e4445d70cbd1f5475175ef391e3

16 files changed +46 -43

.github/CODEOWNERS

+1 -1

@@ -57,7 +57,7 @@ CMakeLists.txt @tlrmchlsmth @LucasWilkinson

57 /tests/v1/kv_connector @ApostaC

58 /tests/v1/offloading @ApostaC

60 -# Transformers backend

60 +# Transformers modeling backend

61 /vllm/model_executor/models/transformers @hmellor

62 /tests/models/test_transformers.py @hmellor

docs/contributing/model/README.md

+1 -1

@@ -1,7 +1,7 @@

1 # Summary

3 !!! important

4 - Many decoder language models can now be automatically loaded using the [Transformers backend](../../models/supported_models.md#transformers) without having to implement them in vLLM. See if `vllm serve <model>` works first!

4 + Many decoder language models can now be automatically loaded using the [Transformers modeling backend](../../models/supported_models.md#transformers) without having to implement them in vLLM. See if `vllm serve <model>` works first!

6 vLLM models are specialized [PyTorch](https://pytorch.org/) models that take advantage of various [features](../../features/README.md#compatibility-matrix) to optimize their performance.

docs/deployment/frameworks/hf_inference_endpoints.md

+2 -2

@@ -156,7 +156,7 @@ In this guide, we demonstrate manual deployment using the [`rednote-hilab/dots.o

156

157 ## Advanced Deployment Details

158

159 -With the [transformers backend integration](https://blog.vllm.ai/2025/04/11/transformers-backend.html), vLLM now offers Day 0 support for any model compatible with `transformers`. This means you can deploy such models immediately, leveraging vLLM’s optimized inference without additional backend modifications.

159 +With the [Transformers modeling backend integration](https://blog.vllm.ai/2025/04/11/transformers-backend.html), vLLM now offers Day 0 support for any model compatible with `transformers`. This means you can deploy such models immediately, leveraging vLLM’s optimized inference without additional backend modifications.

160

161 Hugging Face Inference Endpoints provides a fully managed environment for serving models via vLLM. You can deploy models without configuring servers, installing dependencies, or managing clusters. Endpoints also support deployment across multiple cloud providers (AWS, Azure, GCP) without the need for separate accounts.

162

@@ -167,4 +167,4 @@ The platform integrates seamlessly with the Hugging Face Hub, allowing you to de

167 - Explore the [Inference Endpoints](https://endpoints.huggingface.co/catalog) model catalog

168 - Read the Inference Endpoints [documentation](https://huggingface.co/docs/inference-endpoints/en/index)

169 - Learn about [Inference Endpoints engines](https://huggingface.co/docs/inference-endpoints/en/engines/vllm)

170 -- Understand the [transformers backend integration](https://blog.vllm.ai/2025/04/11/transformers-backend.html)

170 +- Understand the [Transformers modeling backend integration](https://blog.vllm.ai/2025/04/11/transformers-backend.html)

docs/models/supported_models.md

+13 -13

@@ -15,9 +15,9 @@ These models are what we list in [supported text models](#list-of-text-only-lang

16 ### Transformers

18 -vLLM also supports model implementations that are available in Transformers. You should expect the performance of a Transformers model implementation used in vLLM to be within <5% of the performance of a dedicated vLLM model implementation. We call this feature the "Transformers backend".

18 +vLLM also supports model implementations that are available in Transformers. You should expect the performance of a Transformers model implementation used in vLLM to be within <5% of the performance of a dedicated vLLM model implementation. We call this feature the "Transformers modeling backend".

20 -Currently, the Transformers backend works for the following:

20 +Currently, the Transformers modeling backend works for the following:

22 - Modalities: embedding models, language models and vision-language models*

23 - Architectures: encoder-only, decoder-only, mixture-of-experts

@@ -25,7 +25,7 @@ Currently, the Transformers backend works for the following:

26 _*Vision-language models currently accept only image inputs. Support for video inputs will be added in a future release._

28 -If the Transformers model implementation follows all the steps in [writing a custom model](#writing-custom-models) then, when used with the Transformers backend, it will be compatible with the following features of vLLM:

28 +If the Transformers model implementation follows all the steps in [writing a custom model](#writing-custom-models) then, when used with the Transformers modeling backend, it will be compatible with the following features of vLLM:

30 - All the features listed in the [compatibility matrix](../features/README.md#feature-x-feature)

31 - Any combination of the following vLLM parallelisation schemes:

@@ -44,7 +44,7 @@ llm.apply_model(lambda model: print(type(model)))

45 If the printed type starts with `Transformers...` then it's using the Transformers model implementation!

47 -If a model has a vLLM implementation but you would prefer to use the Transformers implementation via the Transformers backend, set `model_impl="transformers"` for [offline inference](../serving/offline_inference.md) or `--model-impl transformers` for the [online serving](../serving/openai_compatible_server.md).

47 +If a model has a vLLM implementation but you would prefer to use the Transformers implementation via the Transformers modeling backend, set `model_impl="transformers"` for [offline inference](../serving/offline_inference.md) or `--model-impl transformers` for the [online serving](../serving/openai_compatible_server.md).

49 !!! note

50 For vision-language models, if you are loading with `dtype="auto"`, vLLM loads the whole model with config's `dtype` if it exists. In contrast the native Transformers will respect the `dtype` attribute of each backbone in the model. That might cause a slight difference in performance.

@@ -53,12 +53,12 @@ If a model has a vLLM implementation but you would prefer to use the Transformer

54 If a model is neither supported natively by vLLM nor Transformers, it can still be used in vLLM!

56 -For a model to be compatible with the Transformers backend for vLLM it must:

56 +For a model to be compatible with the Transformers modeling backend for vLLM it must:

58 - be a Transformers compatible custom model (see [Transformers - Customizing models](https://huggingface.co/docs/transformers/en/custom_models)):

59 - The model directory must have the correct structure (e.g. `config.json` is present).

60 - `config.json` must contain `auto_map.AutoModel`.

61 -- be a Transformers backend for vLLM compatible model (see [Writing custom models](#writing-custom-models)):

61 +- be a Transformers modeling backend for vLLM compatible model (see [Writing custom models](#writing-custom-models)):

62 - Customisation should be done in the base model (e.g. in `MyModel`, not `MyModelForCausalLM`).

64 If the compatible model is:

@@ -66,13 +66,13 @@ If the compatible model is:

66 - on the Hugging Face Model Hub, simply set `trust_remote_code=True` for [offline-inference](../serving/offline_inference.md) or `--trust-remote-code` for the [openai-compatible-server](../serving/openai_compatible_server.md).

67 - in a local directory, simply pass directory path to `model=<MODEL_DIR>` for [offline-inference](../serving/offline_inference.md) or `vllm serve <MODEL_DIR>` for the [openai-compatible-server](../serving/openai_compatible_server.md).

69 -This means that, with the Transformers backend for vLLM, new models can be used before they are officially supported in Transformers or vLLM!

69 +This means that, with the Transformers modeling backend for vLLM, new models can be used before they are officially supported in Transformers or vLLM!

71 #### Writing custom models

73 -This section details the necessary modifications to make to a Transformers compatible custom model that make it compatible with the Transformers backend for vLLM. (We assume that a Transformers compatible custom model has already been created, see [Transformers - Customizing models](https://huggingface.co/docs/transformers/en/custom_models)).

73 +This section details the necessary modifications to make to a Transformers compatible custom model that make it compatible with the Transformers modeling backend for vLLM. (We assume that a Transformers compatible custom model has already been created, see [Transformers - Customizing models](https://huggingface.co/docs/transformers/en/custom_models)).

75 -To make your model compatible with the Transformers backend, it needs:

75 +To make your model compatible with the Transformers modeling backend, it needs:

77 1. `kwargs` passed down through all modules from `MyModel` to `MyAttention`.

78 - If your model is encoder-only:

@@ -134,7 +134,7 @@ Here is what happens in the background when this model is loaded:

134

135 1. The config is loaded.

136 2. `MyModel` Python class is loaded from the `auto_map` in config, and we check that the model `is_backend_compatible()`.

137 -3. `MyModel` is loaded into one of the Transformers backend classes in [vllm/model_executor/models/transformers](../../vllm/model_executor/models/transformers) which sets `self.config._attn_implementation = "vllm"` so that vLLM's attention layer is used.

137 +3. `MyModel` is loaded into one of the Transformers modeling backend classes in [vllm/model_executor/models/transformers](../../vllm/model_executor/models/transformers) which sets `self.config._attn_implementation = "vllm"` so that vLLM's attention layer is used.

138

139 That's it!

140

@@ -182,7 +182,7 @@ To determine whether a given model is natively supported, you can check the `con

182 If the `"architectures"` field contains a model architecture listed below, then it should be natively supported.

183

184 Models do not _need_ to be natively supported to be used in vLLM.

185 -The [Transformers backend](#transformers) enables you to run models directly using their Transformers implementation (or even remote code on the Hugging Face Model Hub!).

185 +The [Transformers modeling backend](#transformers) enables you to run models directly using their Transformers implementation (or even remote code on the Hugging Face Model Hub!).

186

187 !!! tip

188 The easiest way to check if your model is really supported at runtime is to run the program below:

@@ -451,7 +451,7 @@ th {

452 | `LongcatFlashForCausalLM` | LongCat-Flash | `meituan-longcat/LongCat-Flash-Chat`, `meituan-longcat/LongCat-Flash-Chat-FP8` | ✅︎ | ✅︎ |

453

454 -Some models are supported only via the [Transformers backend](#transformers). The purpose of the table below is to acknowledge models which we officially support in this way. The logs will say that the Transformers backend is being used, and you will see no warning that this is fallback behaviour. This means that, if you have issues with any of the models listed below, please [make an issue](https://github.com/vllm-project/vllm/issues/new/choose) and we'll do our best to fix it!

454 +Some models are supported only via the [Transformers modeling backend](#transformers). The purpose of the table below is to acknowledge models which we officially support in this way. The logs will say that the Transformers modeling backend is being used, and you will see no warning that this is fallback behaviour. This means that, if you have issues with any of the models listed below, please [make an issue](https://github.com/vllm-project/vllm/issues/new/choose) and we'll do our best to fix it!

455

457 |--------------|--------|-------------------|----------------------|---------------------------|

@@ -720,7 +720,7 @@ These models primarily accept the [`LLM.generate`](./generative_models.md#llmgen

720 | `TarsierForConditionalGeneration` | Tarsier | T + IE+ | `omni-search/Tarsier-7b`, `omni-search/Tarsier-34b` | | ✅︎ |

722

723 -Some models are supported only via the [Transformers backend](#transformers). The purpose of the table below is to acknowledge models which we officially support in this way. The logs will say that the Transformers backend is being used, and you will see no warning that this is fallback behaviour. This means that, if you have issues with any of the models listed below, please [make an issue](https://github.com/vllm-project/vllm/issues/new/choose) and we'll do our best to fix it!

723 +Some models are supported only via the [Transformers modeling backend](#transformers). The purpose of the table below is to acknowledge models which we officially support in this way. The logs will say that the Transformers modeling backend is being used, and you will see no warning that this is fallback behaviour. This means that, if you have issues with any of the models listed below, please [make an issue](https://github.com/vllm-project/vllm/issues/new/choose) and we'll do our best to fix it!

724

726 |--------------|--------|--------|-------------------|-----------------------------|-----------------------------------------|

tests/models/test_transformers.py

+2 -2

@@ -1,6 +1,6 @@

1 # SPDX-License-Identifier: Apache-2.0

2 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project

3 -"""Test the functionality of the Transformers backend."""

3 +"""Test the functionality of the Transformers modeling backend."""

5 from typing import Any

@@ -85,7 +85,7 @@ def test_models(

85 required = Version("5.0.0.dev")

86 if model == "allenai/OLMoE-1B-7B-0924" and installed < required:

87 pytest.skip(

88 - "MoE models with the Transformers backend require "

88 + "MoE models with the Transformers modeling backend require "

89 f"transformers>={required}, but got {installed}"

90 )

vllm/config/model.py

+4 -4

@@ -732,7 +732,7 @@ class ModelConfig:

732 return self

733

734 def _get_transformers_backend_cls(self) -> str:

735 - """Determine which Transformers backend class will be used if

735 + """Determine which Transformers modeling backend class will be used if

736 `model_impl` is set to `transformers` or `auto`."""

737 cls = "Transformers"

738 # If 'hf_config != hf_text_config' it's a nested config, i.e. multimodal

@@ -746,8 +746,8 @@ class ModelConfig:

746 # User specified value take precedence

747 if self.runner != "auto":

748 runner = self.runner

749 - # Only consider Transformers backend pooling classes if we're wrapping an

750 - # architecture that defaults to pooling. Otherwise, we return the LM class

749 + # Only consider Transformers modeling backend pooling classes if we're wrapping

750 + # an architecture that defaults to pooling. Otherwise, we return the LM class

751 # and use adapters.

752 if runner == "pooling" and task in {"embed", "classify"}:

753 if task == "embed":

@@ -759,7 +759,7 @@ class ModelConfig:

759 return cls

760

761 def using_transformers_backend(self) -> bool:

762 - """Check if the model is using the Transformers backend class."""

762 + """Check if the model is using the Transformers modeling backend class."""

763 used_cls = self._model_info.architecture

764 transformers_backend_cls = self._get_transformers_backend_cls()

765 return used_cls == transformers_backend_cls

vllm/lora/layers/base_linear.py

+1 -1

@@ -121,7 +121,7 @@ class BaseLinearLayerWithLoRA(BaseLayerWithLoRA):

121 def apply(self, x: torch.Tensor, bias: torch.Tensor | None = None) -> torch.Tensor:

122 output = self.base_layer.quant_method.apply(self.base_layer, x, bias)

123

124 - # In transformers backend, x and output have extra batch dimension like

124 + # In Transformers modeling backend, x and output have extra batch dimension like

125 # (1, seq_len, hidden_dim), while punica expects (seq_len, hidden_dim),

126 # therefore we need to flatten the batch dimensions.

127 if x.ndim == 3 and output.ndim == 3:

vllm/model_executor/models/adapters.py

+2 -2

@@ -429,7 +429,7 @@ def load_weights_using_from_2_way_softmax(

429 if text_config.tie_word_embeddings:

430 # embed_tokens is the assumed name for input embeddings. If the model does not

431 # have this attribute, we fallback to get_input_embeddings(), which is used by

432 - # the Transformers backend.

432 + # the Transformers modeling backend.

433 embed_tokens = (

434 model.model.embed_tokens

435 if hasattr(model.model, "embed_tokens")

@@ -487,7 +487,7 @@ def load_weights_no_post_processing(model, weights: Iterable[tuple[str, torch.Te

487 if text_config.tie_word_embeddings:

488 # embed_tokens is the assumed name for input embeddings. If the model does not

489 # have this attribute, we fallback to get_input_embeddings(), which is used by

490 - # the Transformers backend.

490 + # the Transformers modeling backend.

491 embed_tokens = (

492 model.model.embed_tokens

493 if hasattr(model.model, "embed_tokens")

vllm/model_executor/models/transformers/__init__.py

+2 -2

@@ -120,8 +120,8 @@ def __getattr__(name: str):

120 """Handle imports of non-existent classes with a helpful error message."""

121 if name not in globals():

122 raise AttributeError(

123 - "The Transformers backend does not currently have a class to handle "

124 - f"the requested model type: {name}. Please open an issue at "

123 + "The Transformers modeling backend does not currently have a class to "

124 + f"handle the requested model type: {name}. Please open an issue at "

125 "https://github.com/vllm-project/vllm/issues/new"

126 )

127 return globals()[name]

vllm/model_executor/models/transformers/base.py

+5 -4

@@ -14,7 +14,7 @@

14 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

15 # See the License for the specific language governing permissions and

16 # limitations under the License.

17 -"""Transformers backend base class."""

17 +"""Transformers modeling backend base class."""

19 from collections.abc import Iterable

20 from typing import TYPE_CHECKING

@@ -118,7 +118,7 @@ class Base(nn.Module, VllmModel, SupportsQuant, SupportsLoRA, SupportsPP):

118

119 def __init__(self, *, vllm_config: "VllmConfig", prefix: str = ""):

120 super().__init__()

121 - logger.info("Using Transformers backend.")

121 + logger.info("Using Transformers modeling backend.")

122

123 self.config = vllm_config.model_config.hf_config

124 self.text_config = self.config.get_text_config()

@@ -147,7 +147,8 @@ class Base(nn.Module, VllmModel, SupportsQuant, SupportsLoRA, SupportsPP):

147 # Check for unsupported quantization methods.

148 if quant_method_name == "mxfp4":

149 raise NotImplementedError(

150 - "Transformers backend does not support MXFP4 quantization yet."

150 + "Transformers modeling backend does "

151 + "not support MXFP4 quantization yet."

152 )

153 # Skip loading extra bias for GPTQ models.

154 if "gptq" in quant_method_name:

@@ -458,6 +459,6 @@ class Base(nn.Module, VllmModel, SupportsQuant, SupportsLoRA, SupportsPP):

459 required = Version(min_version)

460 if installed < required:

461 raise ImportError(

461 - f"Transformers backend requires transformers>={required} "

462 + f"Transformers modeling backend requires transformers>={required} "

463 f"for {feature}, but got {installed}"

464 )

vllm/model_executor/models/transformers/causal.py

+1 -1

@@ -14,7 +14,7 @@

14 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

15 # See the License for the specific language governing permissions and

16 # limitations under the License.

17 -"""Transformers backend mixin for causal language models."""

17 +"""Transformers modeling backend mixin for causal language models."""

19 from typing import TYPE_CHECKING

vllm/model_executor/models/transformers/legacy.py

+1 -1

@@ -14,7 +14,7 @@

14 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

15 # See the License for the specific language governing permissions and

16 # limitations under the License.

17 -"""Transformers backend mixin for legacy models."""

17 +"""Transformers modeling backend mixin for legacy models."""

19 from typing import TYPE_CHECKING

vllm/model_executor/models/transformers/moe.py

+2 -2

@@ -14,7 +14,7 @@

14 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

15 # See the License for the specific language governing permissions and

16 # limitations under the License.

17 -"""Transformers backend mixin for Mixture of Experts (MoE) models."""

17 +"""Transformers modeling backend mixin for Mixture of Experts (MoE) models."""

19 from typing import TYPE_CHECKING, Any

@@ -39,7 +39,7 @@ if TYPE_CHECKING:

40 @CustomOp.register("transformers_fused_moe")

41 class TransformersFusedMoE(FusedMoE):

42 - """Custom FusedMoE for the Transformers backend."""

42 + """Custom FusedMoE for the Transformers modeling backend."""

44 def __init__(self, *args, **kwargs):

45 super().__init__(*args, **kwargs)

vllm/model_executor/models/transformers/multimodal.py

+7 -5

@@ -14,7 +14,7 @@

14 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

15 # See the License for the specific language governing permissions and

16 # limitations under the License.

17 -"""Transformers backend mixin for multi-modal models."""

17 +"""Transformers modeling backend mixin for multi-modal models."""

19 from collections.abc import Mapping

20 from typing import TYPE_CHECKING

@@ -310,9 +310,9 @@ class MultiModalMixin(SupportsMultiModal, SupportsMRoPE):

310 return model_output

311

312 def get_language_model(self) -> torch.nn.Module:

313 - """Transformers backend multimodal classes do not contain a separate vLLM

314 - language model class. Therefore, in order to return a language model vLLM class,

315 - we use a wrapper to give `self` the same interface as a text model."""

313 + """Transformers modeling backend multimodal classes do not contain a separate

314 + vLLM language model class. Therefore, in order to return a language model vLLM

315 + class, we use a wrapper to give `self` the same interface as a text model."""

316

317 # Exclude self and object

318 bases = self.__class__.mro()[1:-1]

@@ -385,7 +385,9 @@ class MultiModalMixin(SupportsMultiModal, SupportsMRoPE):

385 for k, v in kwargs.items()

386 if k not in {"image_grid_thw", "video_grid_thw"}

387 ):

388 - raise NotImplementedError("Transformers backend only supports images.")

388 + raise NotImplementedError(

389 + "Transformers modeling backend only supports images."

390 + )

391

392 image_grid_thw = kwargs.get("image_grid_thw", [])

393 video_grid_thw = kwargs.get("video_grid_thw", [])

vllm/model_executor/models/transformers/pooling.py

+1 -1

@@ -14,7 +14,7 @@

14 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

15 # See the License for the specific language governing permissions and

16 # limitations under the License.

17 -"""Transformers backend mixins for pooling models."""

17 +"""Transformers modeling backend mixins for pooling models."""

19 from typing import TYPE_CHECKING

vllm/model_executor/models/transformers/utils.py

+1 -1

@@ -14,7 +14,7 @@

14 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

15 # See the License for the specific language governing permissions and

16 # limitations under the License.

17 -"""Transformers backend utilities."""

17 +"""Transformers modeling backend utilities."""

19 from contextlib import contextmanager

20 from pathlib import Path