@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / a1a2aaadb9
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 00:45 UTC

[Model]: Add `transformers` backend support (#11330)

# Adds support for `transformers` as a backend Following https://github.com/huggingface/transformers/pull/35235, a bunch of models should already be supported, we are ramping up support for more models. Thanks @Isotr0py for the TP support, and @hmellor for his help as well! This includes: - `trust_remote_code=True` support: any model on the hub, if it implements attention the correct way can be natively supported!! - tensor parallel support --------- Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by: Isotr0py <2037008807@qq.com> Co-authored-by: Isotr0py <41363108+Isotr0py@users.noreply.github.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Isotr0py <2037008807@qq.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Co-authored-by: Michael Goin <mgoin64@gmail.com> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>

Arthur committed Feb 3, 2025 at 14:30 UTC a1a2aaadb9122f05667140e39cf67e5736c8b6d6
11 files changed +528 -9
.buildkite/test-pipeline.yaml
+2
@@ -349,6 +349,7 @@ steps:
349 - vllm/
350 - tests/models
351 commands:
352 + - pytest -v -s models/test_transformers.py
353 - pytest -v -s models/test_registry.py
354 - pytest -v -s models/test_initialization.py
355
@@ -485,6 +486,7 @@ steps:
486 - VLLM_TEST_SAME_HOST=1 torchrun --nproc-per-node=4 distributed/test_same_node.py | grep 'Same node test passed'
487 - TARGET_TEST_SUITE=L4 pytest basic_correctness/ -v -s -m 'distributed(num_gpus=2)'
488 # Avoid importing model tests that cause CUDA reinitialization error
489 + - pytest models/test_transformers.py -v -s -m 'distributed(num_gpus=2)'
490 - pytest models/encoder_decoder/language/test_bart.py -v -s -m 'distributed(num_gpus=2)'
491 - pytest models/encoder_decoder/vision_language/test_broadcast.py -v -s -m 'distributed(num_gpus=2)'
492 - pytest models/decoder_only/vision_language/test_models.py -v -s -m 'distributed(num_gpus=2)'
docs/source/models/supported_models.md
+76
@@ -40,6 +40,82 @@ If vLLM successfully returns text (for generative models) or hidden states (for
40 Otherwise, please refer to [Adding a New Model](#new-model) for instructions on how to implement your model in vLLM.
41 Alternatively, you can [open an issue on GitHub](https://github.com/vllm-project/vllm/issues/new/choose) to request vLLM support.
42
43 +### Transformers fallback
44 +
45 +After the merge of <gh-pr:11330>, `vllm` can fallback to models that are available in `transformers`. This does not work for all models for now, but most decoder language models are supported, and vision language model support is planned!
46 +
47 +To check if the backend is `transformers`, you can simply do this:
48 +
49 +```python
50 +from vllm import LLM
51 +llm = LLM(model=..., task="generate") # Name or path of your model
52 +llm.apply_model(lambda model: print(model.__class__))
53 +```
54 +
55 +If it is `TransformersModel` then it means it's based on `transformers`!
56 +
57 +#### Supported features
58 +
59 +##### LORA and quantization
60 +
61 +Both are not supported yet! Make sure to open an issue and we'll work on this together with the `transformers` team!
62 +
63 +Usually `transformers` model load weights via the `load_adapters` API, that depends on PEFT. We need to work a bit to either use this api (for now this would result in some weights not being marked as loaded) or replace modules accordingly.
64 +
65 +Hints as to how this would look like:
66 +
67 +```python
68 +class TransformersModel(nn.Module, SupportsLoRA):
69 + def __init__(*):
70 + ...
71 + self.model.load_adapter(vllm_config.load_config.model_loader_extra_config["qlora_adapter_name_or_path"])
72 +```
73 +
74 +Blocker is that you need to specify supported lora layers, when we would ideally want to load whatever is inside the checkpoint!
75 +
76 +##### Remote code
77 +
78 +This fallback also means that any model on the hub that can be used in `transformers` with `trust_remote_code=True` that correctly implements attention can be used in production!
79 +
80 +```python
81 +from vllm import LLM
82 +llm = LLM(model=..., task="generate", trust_remote_code=True) # Name or path of your model
83 +llm.apply_model(lambda model: print(model.__class__))
84 +```
85 +
86 +A model just needs the following two things:
87 +
88 +```python
89 +from transformers import PreTrainedModel
90 +from torch import nn
91 +
92 +class MyAttention(nn.Module):
93 +
94 + def forward(self, hidden_states, **kwargs): # <- kwargs are required
95 +
96 + ...
97 + attention_interface = attention_interface = ALL_ATTENTION_FUNCTIONS[self.config._attn_implementation]
98 + attn_output, attn_weights = attention_interface(
99 + self,
100 + query_states,
101 + key_states,
102 + value_states,
103 + **kwargs,
104 + )
105 + ...
106 +
107 +class MyModel(PreTrainedModel):
108 + _supports_attention_backend = True
109 +```
110 +
111 +Here is what happens in the background:
112 +
113 +1. The config is loaded
114 +2. `MyModel` python class is loaded from the `auto_map`, and we check that the model `_supports_attention_backend`.
115 +3. The `TransformersModel` backend is used. See `/model_executors/models/transformers`, which leverage `self.config._attn_implementation = "vllm"`, thus the need to use `ALL_ATTENTION_FUNCTION`.
116 +
117 +That's it!
118 +
119 ### ModelScope
120
121 To use models from [ModelScope](https://www.modelscope.cn) instead of HuggingFace Hub, set an environment variable:
requirements-common.txt
+1 -1
@@ -5,7 +5,7 @@ requests >= 2.26.0
5 tqdm
6 blake3
7 py-cpuinfo
8 -transformers >= 4.48.2 # Required for Bamba.
8 +transformers >= 4.48.2 # Required for Bamba model and Transformers backend.
9 tokenizers >= 0.19.1 # Required for Llama 3.
10 protobuf # Required by LlamaTokenizer.
11 fastapi >= 0.107.0, < 0.113.0; python_version < '3.9'
tests/models/registry.py
+5
@@ -281,12 +281,17 @@ _SPECULATIVE_DECODING_EXAMPLE_MODELS = {
281 speculative_model="ibm-fms/llama-160m-accelerator"), # noqa: E501
282 }
283
284 +_FALLBACK_MODEL = {
285 + "TransformersModel": _HfExamplesInfo("ArthurZ/Ilama-3.2-1B", trust_remote_code=True), # noqa: E501
286 +}
287 +
288 _EXAMPLE_MODELS = {
289 **_TEXT_GENERATION_EXAMPLE_MODELS,
290 **_EMBEDDING_EXAMPLE_MODELS,
291 **_CROSS_ENCODER_EXAMPLE_MODELS,
292 **_MULTIMODAL_EXAMPLE_MODELS,
293 **_SPECULATIVE_DECODING_EXAMPLE_MODELS,
294 + **_FALLBACK_MODEL,
295 }
296
297
tests/models/test_oot_registration.py
+3 -1
@@ -15,7 +15,9 @@ def test_plugin(dummy_opt_path):
15 os.environ["VLLM_PLUGINS"] = ""
16 with pytest.raises(Exception) as excinfo:
17 LLM(model=dummy_opt_path, load_format="dummy")
18 - assert "are not supported for now" in str(excinfo.value)
18 + error_msg = "has no vLLM implementation and " \
19 + "the Transformers implementation is not compatible with vLLM."
20 + assert (error_msg in str(excinfo.value))
21
22
23 @fork_new_process_for_each_test
tests/models/test_transformers.py new
+75
@@ -0,0 +1,75 @@
1 +"""Test the functionality of the Transformers backend.
2 +
3 +Run `pytest tests/models/test_transformers.py`.
4 +"""
5 +from contextlib import nullcontext
6 +from typing import Type
7 +
8 +import pytest
9 +
10 +from ..conftest import HfRunner, VllmRunner
11 +from ..utils import multi_gpu_test
12 +from .utils import check_logprobs_close
13 +
14 +
15 +def check_implementation(
16 + hf_runner: Type[HfRunner],
17 + vllm_runner: Type[VllmRunner],
18 + example_prompts: list[str],
19 + model: str,
20 + **kwargs,
21 +):
22 + max_tokens = 32
23 + num_logprobs = 5
24 +
25 + with vllm_runner(model, **kwargs) as vllm_model:
26 + vllm_outputs = vllm_model.generate_greedy_logprobs(
27 + example_prompts, max_tokens, num_logprobs)
28 +
29 + with hf_runner(model) as hf_model:
30 + hf_outputs = hf_model.generate_greedy_logprobs_limit(
31 + example_prompts, max_tokens, num_logprobs)
32 +
33 + check_logprobs_close(
34 + outputs_0_lst=hf_outputs,
35 + outputs_1_lst=vllm_outputs,
36 + name_0="hf",
37 + name_1="vllm",
38 + )
39 +
40 +
41 +@pytest.mark.parametrize(
42 + "model,model_impl",
43 + [
44 + ("meta-llama/Llama-3.2-1B-Instruct", "transformers"),
45 + ("openai-community/gpt2", "transformers"),
46 + ("ArthurZ/Ilama-3.2-1B", "auto"), # CUSTOM CODE
47 + ("meta-llama/Llama-3.2-1B-Instruct", "auto"),
48 + ]) # trust_remote_code=True by default
49 +def test_models(hf_runner, vllm_runner, example_prompts, model,
50 + model_impl) -> None:
51 +
52 + maybe_raises = nullcontext()
53 + if model == "openai-community/gpt2" and model_impl == "transformers":
54 + # Model is not backend compatible
55 + maybe_raises = pytest.raises(
56 + ValueError,
57 + match="The Transformers implementation.*not compatible with vLLM")
58 +
59 + with maybe_raises:
60 + check_implementation(hf_runner,
61 + vllm_runner,
62 + example_prompts,
63 + model,
64 + model_impl=model_impl)
65 +
66 +
67 +@multi_gpu_test(num_gpus=2)
68 +def test_distributed(
69 + hf_runner,
70 + vllm_runner,
71 + example_prompts,
72 +):
73 + kwargs = {"model_impl": "transformers", "tensor_parallel_size": 2}
74 + check_implementation(hf_runner, vllm_runner, example_prompts,
75 + "meta-llama/Llama-3.2-1B-Instruct", **kwargs)
vllm/config.py
+14
@@ -83,6 +83,12 @@ class SupportsHash(Protocol):
83 ...
84
85
86 +class ModelImpl(str, enum.Enum):
87 + AUTO = "auto"
88 + VLLM = "vllm"
89 + TRANSFORMERS = "transformers"
90 +
91 +
92 class ModelConfig:
93 """Configuration for the model.
94
@@ -167,6 +173,12 @@ class ModelConfig:
173 `logits_processors` extra completion argument. Defaults to None,
174 which allows no processors.
175 generation_config: Configuration parameter file for generation.
176 + model_impl: Which implementation of the model to use:
177 + "auto" will try to use the vLLM implementation if it exists and
178 + fall back to the Transformers implementation if no vLLM
179 + implementation is available.
180 + "vllm" will use the vLLM model implementation.
181 + "transformers" will use the Transformers model implementation.
182 override_generation_config: Override the generation config with the
183 given config.
184 """
@@ -230,6 +242,7 @@ class ModelConfig:
242 generation_config: Optional[str] = None,
243 enable_sleep_mode: bool = False,
244 override_generation_config: Optional[Dict[str, Any]] = None,
245 + model_impl: Union[str, ModelImpl] = ModelImpl.AUTO,
246 ) -> None:
247 self.model = model
248 self.tokenizer = tokenizer
@@ -241,6 +254,7 @@ class ModelConfig:
254 self.code_revision = code_revision
255 self.rope_scaling = rope_scaling
256 self.rope_theta = rope_theta
257 + self.model_impl = model_impl
258
259 if hf_overrides is None:
260 hf_overrides = {}
vllm/engine/arg_utils.py
+18 -4
@@ -13,10 +13,10 @@ import vllm.envs as envs
13 from vllm.config import (CacheConfig, CompilationConfig, ConfigFormat,
14 DecodingConfig, DeviceConfig, HfOverrides,
15 KVTransferConfig, LoadConfig, LoadFormat, LoRAConfig,
16 - ModelConfig, ObservabilityConfig, ParallelConfig,
17 - PoolerConfig, PromptAdapterConfig, SchedulerConfig,
18 - SpeculativeConfig, TaskOption, TokenizerPoolConfig,
19 - VllmConfig)
16 + ModelConfig, ModelImpl, ObservabilityConfig,
17 + ParallelConfig, PoolerConfig, PromptAdapterConfig,
18 + SchedulerConfig, SpeculativeConfig, TaskOption,
19 + TokenizerPoolConfig, VllmConfig)
20 from vllm.executor.executor_base import ExecutorBase
21 from vllm.logger import init_logger
22 from vllm.model_executor.layers.quantization import QUANTIZATION_METHODS
@@ -199,6 +199,7 @@ class EngineArgs:
199 generation_config: Optional[str] = None
200 override_generation_config: Optional[Dict[str, Any]] = None
201 enable_sleep_mode: bool = False
202 + model_impl: str = "auto"
203
204 calculate_kv_scales: Optional[bool] = None
205
@@ -378,6 +379,18 @@ class EngineArgs:
379 'qualified names that can be passed with the `logits_processors` '
380 'extra completion argument. Defaults to None, which allows no '
381 'processors.')
382 + parser.add_argument(
383 + '--model-impl',
384 + type=str,
385 + default=EngineArgs.model_impl,
386 + choices=[f.value for f in ModelImpl],
387 + help='Which implementation of the model to use.\n\n'
388 + '* "auto" will try to use the vLLM implementation if it exists '
389 + 'and fall back to the Transformers implementation if no vLLM '
390 + 'implementation is available.\n'
391 + '* "vllm" will use the vLLM model implementation.\n'
392 + '* "transformers" will use the Transformers model '
393 + 'implementation.\n')
394 # Parallel arguments
395 parser.add_argument(
396 '--distributed-executor-backend',
@@ -1017,6 +1030,7 @@ class EngineArgs:
1030 generation_config=self.generation_config,
1031 override_generation_config=self.override_generation_config,
1032 enable_sleep_mode=self.enable_sleep_mode,
1033 + model_impl=self.model_impl,
1034 )
1035
1036 def create_load_config(self) -> LoadConfig:
vllm/model_executor/model_loader/utils.py
+59 -2
@@ -2,17 +2,22 @@
2 """Utilities for selecting and loading models."""
3 import contextlib
4 from dataclasses import dataclass, field
5 -from typing import Dict, List, Tuple, Type
5 +from typing import Dict, List, Optional, Tuple, Type
6
7 import torch
8 +import transformers
9 from torch import nn
10 +from transformers.dynamic_module_utils import get_class_from_dynamic_module
11
10 -from vllm.config import ModelConfig
12 +from vllm.config import ModelConfig, ModelImpl
13 +from vllm.logger import init_logger
14 from vllm.model_executor.models import ModelRegistry
15 from vllm.model_executor.models.adapters import (as_classification_model,
16 as_embedding_model,
17 as_reward_model)
18
19 +logger = init_logger(__name__)
20 +
21
22 @contextlib.contextmanager
23 def set_default_torch_dtype(dtype: torch.dtype):
@@ -23,6 +28,50 @@ def set_default_torch_dtype(dtype: torch.dtype):
28 torch.set_default_dtype(old_dtype)
29
30
31 +def is_transformers_impl_compatible(
32 + arch: str,
33 + module: Optional[transformers.PreTrainedModel] = None) -> bool:
34 + mod = module or getattr(transformers, arch, None)
35 + if mod is None:
36 + return False
37 + if hasattr(mod, "supports_backend"):
38 + return mod.is_backend_compatible()
39 + else:
40 + return mod._supports_flex_attn
41 +
42 +
43 +def resolve_transformers_fallback(model_config: ModelConfig,
44 + architectures: list[str]):
45 + for i, arch in enumerate(architectures):
46 + if arch == "TransformersModel":
47 + continue
48 + custom_module = None
49 + auto_map = getattr(model_config.hf_config, "auto_map", None)
50 + if auto_map is not None and "AutoModel" in auto_map:
51 + custom_module = get_class_from_dynamic_module(
52 + model_config.hf_config.auto_map["AutoModel"],
53 + model_config.model)
54 + # TODO(Isotr0py): Further clean up these raises.
55 + # perhaps handled them in _ModelRegistry._raise_for_unsupported?
56 + if model_config.model_impl == ModelImpl.TRANSFORMERS:
57 + if not is_transformers_impl_compatible(arch, custom_module):
58 + raise ValueError(
59 + f"The Transformers implementation of {arch} is not "
60 + "compatible with vLLM.")
61 + architectures[i] = "TransformersModel"
62 + if model_config.model_impl == ModelImpl.AUTO:
63 + if not is_transformers_impl_compatible(arch, custom_module):
64 + raise ValueError(
65 + f"{arch} has no vLLM implementation and the Transformers "
66 + "implementation is not compatible with vLLM.")
67 + logger.warning(
68 + "%s has no vLLM implementation, falling back to Transformers "
69 + "implementation. Some features may not be supported and "
70 + "performance may not be optimal.", arch)
71 + architectures[i] = "TransformersModel"
72 + return architectures
73 +
74 +
75 def get_model_architecture(
76 model_config: ModelConfig) -> Tuple[Type[nn.Module], str]:
77 architectures = getattr(model_config.hf_config, "architectures", [])
@@ -38,6 +87,14 @@ def get_model_architecture(
87 and "MixtralForCausalLM" in architectures):
88 architectures = ["QuantMixtralForCausalLM"]
89
90 + vllm_supported_archs = ModelRegistry.get_supported_archs()
91 + is_vllm_supported = any(arch in vllm_supported_archs
92 + for arch in architectures)
93 + if (not is_vllm_supported
94 + or model_config.model_impl == ModelImpl.TRANSFORMERS):
95 + architectures = resolve_transformers_fallback(model_config,
96 + architectures)
97 +
98 model_cls, arch = ModelRegistry.resolve_model_cls(architectures)
99 if model_config.task == "embed":
100 model_cls = as_embedding_model(model_cls)
vllm/model_executor/models/registry.py
+11 -1
@@ -184,6 +184,10 @@ _SPECULATIVE_DECODING_MODELS = {
184 "MedusaModel": ("medusa", "Medusa"),
185 "MLPSpeculatorPreTrainedModel": ("mlp_speculator", "MLPSpeculator"),
186 }
187 +
188 +_FALLBACK_MODEL = {
189 + "TransformersModel": ("transformers", "TransformersModel"),
190 +}
191 # yapf: enable
192
193 _VLLM_MODELS = {
@@ -192,6 +196,7 @@ _VLLM_MODELS = {
196 **_CROSS_ENCODER_MODELS,
197 **_MULTIMODAL_MODELS,
198 **_SPECULATIVE_DECODING_MODELS,
199 + **_FALLBACK_MODEL,
200 }
201
202
@@ -378,7 +383,12 @@ class _ModelRegistry:
383 if not architectures:
384 logger.warning("No model architectures are specified")
385
381 - return architectures
386 + normalized_arch = []
387 + for model in architectures:
388 + if model not in self.models:
389 + model = "TransformersModel"
390 + normalized_arch.append(model)
391 + return normalized_arch
392
393 def inspect_model_cls(
394 self,
vllm/model_executor/models/transformers.py new
+264
@@ -0,0 +1,264 @@
1 +# Copyright 2024 The vLLM team.
2 +#
3 +# Licensed under the Apache License, Version 2.0 (the "License");
4 +# you may not use this file except in compliance with the License.
5 +# You may obtain a copy of the License at
6 +#
7 +# http://www.apache.org/licenses/LICENSE-2.0
8 +#
9 +# Unless required by applicable law or agreed to in writing, software
10 +# distributed under the License is distributed on an "AS IS" BASIS,
11 +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 +# See the License for the specific language governing permissions and
13 +# limitations under the License.
14 +"""Wrapper around `transformers` models"""
15 +import re
16 +from typing import Iterable, List, Optional, Set, Tuple, Union
17 +
18 +import torch
19 +from torch import nn
20 +from transformers import AutoModel, PreTrainedModel
21 +from transformers.modeling_utils import ALL_ATTENTION_FUNCTIONS
22 +
23 +from vllm.attention import Attention, AttentionMetadata
24 +from vllm.config import VllmConfig
25 +from vllm.distributed import get_tensor_model_parallel_world_size
26 +from vllm.distributed.utils import divide
27 +from vllm.logger import init_logger
28 +from vllm.model_executor.layers.linear import (ColumnParallelLinear,
29 + RowParallelLinear)
30 +from vllm.model_executor.layers.logits_processor import LogitsProcessor
31 +from vllm.model_executor.layers.sampler import SamplerOutput, get_sampler
32 +from vllm.model_executor.layers.vocab_parallel_embedding import (
33 + ParallelLMHead, VocabParallelEmbedding)
34 +from vllm.model_executor.model_loader.weight_utils import default_weight_loader
35 +from vllm.model_executor.sampling_metadata import SamplingMetadata
36 +from vllm.sequence import IntermediateTensors
37 +
38 +from .utils import maybe_prefix
39 +
40 +logger = init_logger(__name__)
41 +
42 +
43 +def vllm_flash_attention_forward(
44 + # Transformers args
45 + module: torch.nn.Module,
46 + query: torch.Tensor,
47 + key: torch.Tensor,
48 + value: torch.Tensor,
49 + attention_mask: torch.Tensor,
50 + # Transformers kwargs
51 + scaling: float = None,
52 + # vLLM kwargs
53 + attn_metadata: AttentionMetadata = None,
54 + attention_instances: list[Attention] = None,
55 + **kwargs):
56 + self_attn = attention_instances[module.layer_idx]
57 + if scaling is not None:
58 + self_attn.impl.scale = float(scaling)
59 + hidden = query.shape[-2]
60 + query, key, value = (x.transpose(1, 2) for x in (query, key, value))
61 + query, key, value = (x.reshape(hidden, -1) for x in (query, key, value))
62 + return self_attn.forward(
63 + query,
64 + key,
65 + value,
66 + kv_cache=None, # argument not used
67 + attn_metadata=attn_metadata), None
68 +
69 +
70 +ALL_ATTENTION_FUNCTIONS["vllm"] = vllm_flash_attention_forward
71 +
72 +
73 +# Linear Layer that is compatible with transformers internal forward
74 +# TODO: This is a temporary solution, we should find a better way to integrate
75 +class HFColumnParallelLinear(ColumnParallelLinear):
76 +
77 + def forward(self, input: torch.Tensor) -> torch.Tensor:
78 + return super().forward(input)[0]
79 +
80 +
81 +class HFRowParallelLinear(RowParallelLinear):
82 +
83 + def forward(self, input: torch.Tensor) -> torch.Tensor:
84 + return super().forward(input)[0]
85 +
86 +
87 +def replace_tp_linear_class(orig_module: nn.Linear,
88 + style: str,
89 + quant_config=None):
90 + """
91 + In model configurations, we use a neutral type (string) to specify parallel
92 + styles, here we use it to translate nn.Linear into vllm-style tp Linear.
93 +
94 + Quant config is not supported yet
95 + """
96 +
97 + if not isinstance(style, str):
98 + raise ValueError(
99 + f"Unsupported parallel style type {type(style)}, expected str")
100 +
101 + input_size = orig_module.in_features
102 + output_size = orig_module.out_features
103 + bias = orig_module.bias is not None
104 +
105 + if style == "colwise":
106 + return HFColumnParallelLinear(
107 + input_size,
108 + output_size,
109 + bias,
110 + )
111 + elif style == "rowwise":
112 + return HFRowParallelLinear(
113 + input_size,
114 + output_size,
115 + bias,
116 + )
117 + # We don't consider colwise_rep since it's used in lm_head
118 + else:
119 + raise ValueError(f"Unsupported parallel style value: {style}")
120 +
121 +
122 +class TransformersModel(nn.Module):
123 + embedding_padding_modules = ["lm_head"]
124 + embedding_modules = ["embed_tokens"
125 + ] # TODO transformers will have a util to get it
126 +
127 + def __init__(self, *, vllm_config: VllmConfig, prefix: str = "") -> None:
128 + super().__init__()
129 + logger.info("Using Transformers backend.")
130 +
131 + self.vllm_config = vllm_config
132 + config = vllm_config.model_config.hf_config
133 + cache_config = vllm_config.cache_config
134 + quant_config = vllm_config.quant_config
135 + self.quant_config = quant_config
136 + self.config = config
137 + self.vocab_size = config.vocab_size
138 + self.unpadded_vocab_size = config.vocab_size
139 +
140 + self.model: PreTrainedModel = AutoModel.from_config(
141 + self.config,
142 + attn_implementation="vllm",
143 + torch_dtype=vllm_config.model_config.dtype,
144 + trust_remote_code=vllm_config.model_config.trust_remote_code,
145 + )
146 + prefix = self.model.base_model_prefix
147 +
148 + # MLP modifications
149 + self.tensor_parallelize(self.model)
150 +
151 + # Attention modifications (assumes 1 attention op per hidden layer)
152 + tp_size = get_tensor_model_parallel_world_size()
153 + self.attention_instances = [
154 + Attention(
155 + num_heads=divide(config.num_attention_heads, tp_size),
156 + head_size=config.head_dim,
157 + # NOTE: We use Llama scale as default, if it's set by
158 + # Transformers, it's updated in vllm_flash_attention_forward
159 + scale=config.head_dim**-0.5,
160 + num_kv_heads=divide(config.num_key_value_heads, tp_size),
161 + cache_config=cache_config,
162 + quant_config=None,
163 + prefix=f"{i}.attn") for i in range(config.num_hidden_layers)
164 + ]
165 +
166 + # Model modifications
167 + self.replace_vocab_embed_class(self.model)
168 +
169 + # ForCausalLM modifications
170 + self.lm_head = ParallelLMHead(config.vocab_size,
171 + config.hidden_size,
172 + quant_config=None,
173 + prefix=maybe_prefix(prefix, "lm_head"))
174 + if config.tie_word_embeddings:
175 + self.lm_head.weight = self.model.get_input_embeddings().weight
176 +
177 + logit_scale = getattr(config, "logit_scale", 1.0)
178 + self.logits_processor = LogitsProcessor(self.unpadded_vocab_size,
179 + config.vocab_size, logit_scale)
180 + self.sampler = get_sampler()
181 +
182 + def log_replacement(self, name: str, old_module: nn.Module,
183 + new_module: nn.Module):
184 + logger.debug("%s: %s -> %s", name, old_module, new_module)
185 +
186 + def tensor_parallelize(self, module: nn.Module, prefix: str = ""):
187 + if (self.config.base_model_tp_plan is None
188 + and self.vllm_config.parallel_config.tensor_parallel_size > 1):
189 + raise ValueError(
190 + "Trying to run tensor parallelization but the model does not "
191 + "support it yet!")
192 +
193 + for child_name, child_module in module.named_children():
194 + qual_name = prefix + child_name
195 + for pattern, style in self.config.base_model_tp_plan.items():
196 + if re.match(pattern, qual_name) and isinstance(
197 + child_module, nn.Linear):
198 + new_module = replace_tp_linear_class(
199 + child_module, style, self.quant_config)
200 + setattr(module, child_name, new_module)
201 + self.log_replacement(qual_name, child_module, new_module)
202 + else:
203 + self.tensor_parallelize(child_module, prefix=f"{qual_name}.")
204 +
205 + def replace_vocab_embed_class(self, module: nn.Module):
206 + # Use native set input embeddings
207 + new_module = VocabParallelEmbedding(
208 + self.vocab_size,
209 + self.config.hidden_size,
210 + org_num_embeddings=self.config.vocab_size,
211 + quant_config=None,
212 + )
213 + self.log_replacement("input embedding",
214 + self.model.get_input_embeddings(), new_module)
215 + self.model.set_input_embeddings(new_module)
216 +
217 + def forward(
218 + self,
219 + input_ids: torch.Tensor,
220 + positions: torch.Tensor,
221 + kv_caches: List[torch.Tensor], # argument not used
222 + attn_metadata: AttentionMetadata,
223 + intermediate_tensors: Optional[IntermediateTensors] = None,
224 + inputs_embeds: Optional[torch.Tensor] = None,
225 + ) -> Union[torch.Tensor, IntermediateTensors]:
226 + model_output = self.model(
227 + input_ids[None, ...],
228 + use_cache=False,
229 + position_ids=positions[None, ...],
230 + attn_metadata=attn_metadata,
231 + intermediate_tensors=intermediate_tensors,
232 + attention_instances=self.attention_instances,
233 + return_dict=False)[0][0, ...] # we remove batch dimension for now
234 + return model_output
235 +
236 + def compute_logits(
237 + self,
238 + hidden_states: torch.Tensor,
239 + sampling_metadata: SamplingMetadata,
240 + ) -> Optional[torch.Tensor]:
241 + logits = self.logits_processor(self.lm_head, hidden_states,
242 + sampling_metadata)
243 + return logits
244 +
245 + def sample(self, logits: torch.Tensor,
246 + sampling_metadata: SamplingMetadata) -> Optional[SamplerOutput]:
247 +
248 + next_tokens = self.sampler(logits, sampling_metadata)
249 + return next_tokens
250 +
251 + def load_weights(self, weights: Iterable[Tuple[str,
252 + torch.Tensor]]) -> Set[str]:
253 + params_dict = dict(self.named_parameters())
254 + loaded_params: Set[str] = set()
255 + for name, loaded_weight in weights:
256 + if name not in params_dict:
257 + name = f"{self.model.base_model_prefix}.{name}"
258 + if name in params_dict:
259 + param = params_dict[name]
260 + weight_loader = getattr(param, "weight_loader",
261 + default_weight_loader)
262 + weight_loader(param, loaded_weight)
263 + loaded_params.add(name)
264 + return loaded_params