@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 504ac53d18
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 07:00 UTC

[misc] error early for old-style class (#10304)

Signed-off-by: youkaichao <youkaichao@gmail.com>

youkaichao committed Nov 13, 2024 at 18:55 UTC 504ac53d18fc057d2a98741fa27d89df9054422d
21 files changed +75 -63
docs/source/design/class_hierarchy.rst
+39
@@ -26,6 +26,45 @@ There are several important design choices behind this class hierarchy:
26
27 2. **Uniformity**: The model runner needs a unified interface to create and initialize the model. vLLM supports more than 50 types of popular open-source models. Each model has its own initialization logic. If the constructor signature varies with models, the model runner does not know how to call the constructor accordingly, without complicated and error-prone inspection logic. By making the constructor of the model class uniform, the model runner can easily create and initialize the model without knowing the specific model type. This is also useful for composing models. Vision-language models often consist of a vision model and a language model. By making the constructor uniform, we can easily create a vision model and a language model and compose them into a vision-language model.
28
29 +.. note::
30 +
31 + To support this change, all vLLM models' signatures have been updated to:
32 +
33 + .. code-block:: python
34 +
35 + def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
36 +
37 + To avoid accidentally passing incorrect arguments, the constructor is now keyword-only. This ensures that the constructor will raise an error if old configurations are passed. vLLM developers have already made this change for all models within vLLM. For out-of-tree registered models, developers need to update their models, for example by adding shim code to adapt the old constructor signature to the new one:
38 +
39 + .. code-block:: python
40 +
41 + class MyOldModel(nn.Module):
42 + def __init__(
43 + self,
44 + config,
45 + cache_config: Optional[CacheConfig] = None,
46 + quant_config: Optional[QuantizationConfig] = None,
47 + lora_config: Optional[LoRAConfig] = None,
48 + prefix: str = "",
49 + ) -> None:
50 + ...
51 +
52 + from vllm.config import VllmConfig
53 + class MyNewModel(MyOldModel):
54 + def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
55 + config = vllm_config.model_config.hf_config
56 + cache_config = vllm_config.cache_config
57 + quant_config = vllm_config.quant_config
58 + lora_config = vllm_config.lora_config
59 + super().__init__(config, cache_config, quant_config, lora_config, prefix)
60 +
61 + if __version__ >= "0.6.4":
62 + MyModel = MyNewModel
63 + else:
64 + MyModel = MyOldModel
65 +
66 + This way, the model can work with both old and new versions of vLLM.
67 +
68 3. **Sharding and Quantization at Initialization**: Certain features require changing the model weights. For example, tensor parallelism needs to shard the model weights, and quantization needs to quantize the model weights. There are two possible ways to implement this feature. One way is to change the model weights after the model is initialized. The other way is to change the model weights during the model initialization. vLLM chooses the latter. The first approach is not scalable to large models. Suppose we want to run a 405B model (with roughly 810GB weights) with 16 H100 80GB GPUs. Ideally, every GPU should only load 50GB weights. If we change the model weights after the model is initialized, we need to load the full 810GB weights to every GPU and then shard the weights, leading to a huge memory overhead. Instead, if we shard the weights during the model initialization, every layer will only create a shard of the weights it needs, leading to a much smaller memory overhead. The same idea applies to quantization. Note that we also add an additional argument ``prefix`` to the model's constructor so that the model can initialize itself differently based on the prefix. This is useful for non-uniform quantization, where different parts of the model are quantized differently. The ``prefix`` is usually an empty string for the top-level model and a string like ``"vision"`` or ``"language"`` for the sub-models. In general, it matches the name of the module's state dict in the checkpoint file.
69
70 One disadvantage of this design is that it is hard to write unit tests for individual components in vLLM because every component needs to be initialized by a complete config object. We solve this problem by providing a default initialization function that creates a default config object with all fields set to ``None``. If the component we want to test only cares about a few fields in the config object, we can create a default config object and set the fields we care about. This way, we can test the component in isolation. Note that many tests in vLLM are end-to-end tests that test the whole system, so this is not a big problem.
vllm/model_executor/model_loader/loader.py
+15 -2
@@ -4,6 +4,7 @@ import copy
4 import dataclasses
5 import fnmatch
6 import glob
7 +import inspect
8 import json
9 import math
10 import os
@@ -88,11 +89,23 @@ def device_loading_context(module: torch.nn.Module,
89 logger = init_logger(__name__)
90
91
91 -def _initialize_model(vllm_config: VllmConfig) -> nn.Module:
92 +def _initialize_model(vllm_config: VllmConfig, prefix: str = "") -> nn.Module:
93 """Initialize a model with the given configurations."""
94 model_config = vllm_config.model_config
95 model_class, _ = get_model_architecture(model_config)
95 - return model_class(vllm_config=vllm_config)
96 + signatures = inspect.signature(model_class.__init__)
97 + # collect all kw-only parameters
98 + kw_only_params = [
99 + param.name for param in signatures.parameters.values()
100 + if param.kind == inspect.Parameter.KEYWORD_ONLY
101 + ]
102 + assert "vllm_config" in kw_only_params and "prefix" in kw_only_params, \
103 + ("vLLM model class must accept `vllm_config` and `prefix` as kw-only "
104 + "arguments. Possibly you have an old-style model class registered from "
105 + "out of tree and it is used for new vLLM version. "
106 + "Please check https://docs.vllm.ai/en/latest/design/class_hierarchy.html "
107 + "for the design and update the model class accordingly.")
108 + return model_class(vllm_config=vllm_config, prefix=prefix)
109
110
111 class BaseModelLoader(ABC):
vllm/model_executor/models/arctic.py
+1 -1
@@ -415,7 +415,7 @@ class ArcticModel(nn.Module):
415
416 class ArcticForCausalLM(nn.Module, SupportsPP):
417
418 - def __init__(self, vllm_config: VllmConfig, prefix: str = "") -> None:
418 + def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
419 super().__init__()
420 config = vllm_config.model_config.hf_config
421 quant_config = vllm_config.quant_config
vllm/model_executor/models/bloom.py
+1 -5
@@ -281,11 +281,7 @@ class BloomModel(nn.Module):
281
282 class BloomForCausalLM(nn.Module, SupportsPP):
283
284 - def __init__(
285 - self,
286 - vllm_config: VllmConfig,
287 - prefix: str = "",
288 - ):
284 + def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
285 super().__init__()
286 config = vllm_config.model_config.hf_config
287 quant_config = vllm_config.quant_config
vllm/model_executor/models/chatglm.py
+1 -5
@@ -593,11 +593,7 @@ class ChatGLMForCausalLM(nn.Module, SupportsLoRA, SupportsPP,
593 embedding_modules = {}
594 embedding_padding_modules = []
595
596 - def __init__(
597 - self,
598 - vllm_config: VllmConfig,
599 - prefix: str = "",
600 - ):
596 + def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
597 super().__init__()
598 config = vllm_config.model_config.hf_config
599 quant_config = vllm_config.quant_config
vllm/model_executor/models/dbrx.py
+1 -5
@@ -350,11 +350,7 @@ class DbrxModel(nn.Module):
350
351 class DbrxForCausalLM(nn.Module, SupportsPP):
352
353 - def __init__(
354 - self,
355 - vllm_config: VllmConfig,
356 - prefix: str = "",
357 - ):
353 + def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
354 super().__init__()
355 config = vllm_config.model_config.hf_config
356 quant_config = vllm_config.quant_config
vllm/model_executor/models/eagle.py
+1 -1
@@ -36,7 +36,7 @@ class EAGLE(nn.Module):
36 in the draft checkpoint (using key token_map). Also, the draft config
37 needs to have truncated_vocab_size (=k) as an attribute."""
38
39 - def __init__(self, vllm_config: VllmConfig, prefix: str = "") -> None:
39 + def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
40 super().__init__()
41 config = vllm_config.model_config.hf_config
42 self.config = config
vllm/model_executor/models/falcon.py
+1 -5
@@ -401,11 +401,7 @@ class FalconForCausalLM(nn.Module, SupportsPP):
401 ".dense_4h_to_h.",
402 ]
403
404 - def __init__(
405 - self,
406 - vllm_config: VllmConfig,
407 - prefix: str = "",
408 - ):
404 + def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
405 super().__init__()
406 config = vllm_config.model_config.hf_config
407 quant_config = vllm_config.quant_config
vllm/model_executor/models/fuyu.py
+1 -1
@@ -225,7 +225,7 @@ def input_mapper_for_fuyu(ctx: InputContext, data: object):
225 @INPUT_REGISTRY.register_input_processor(input_processor_for_fuyu)
226 class FuyuForCausalLM(nn.Module, SupportsMultiModal, SupportsPP):
227
228 - def __init__(self, vllm_config: VllmConfig, prefix: str = "") -> None:
228 + def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
229 super().__init__()
230 config = vllm_config.model_config.hf_config
231 quant_config = vllm_config.quant_config
vllm/model_executor/models/gpt2.py
+1 -5
@@ -242,11 +242,7 @@ class GPT2Model(nn.Module):
242
243 class GPT2LMHeadModel(nn.Module, SupportsPP):
244
245 - def __init__(
246 - self,
247 - vllm_config: VllmConfig,
248 - prefix: str = "",
249 - ):
245 + def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
246 super().__init__()
247 config = vllm_config.model_config.hf_config
248 quant_config = vllm_config.quant_config
vllm/model_executor/models/gpt_bigcode.py
+1 -5
@@ -257,11 +257,7 @@ class GPTBigCodeForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
257
258 embedding_padding_modules = []
259
260 - def __init__(
261 - self,
262 - vllm_config: VllmConfig,
263 - prefix: str = "",
264 - ):
260 + def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
261 super().__init__()
262 config = vllm_config.model_config.hf_config
263 quant_config = vllm_config.quant_config
vllm/model_executor/models/gpt_j.py
+1 -5
@@ -229,11 +229,7 @@ class GPTJModel(nn.Module):
229
230 class GPTJForCausalLM(nn.Module, SupportsPP):
231
232 - def __init__(
233 - self,
234 - vllm_config: VllmConfig,
235 - prefix: str = "",
236 - ):
232 + def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
233 super().__init__()
234 config = vllm_config.model_config.hf_config
235 quant_config = vllm_config.quant_config
vllm/model_executor/models/gpt_neox.py
+1 -5
@@ -242,11 +242,7 @@ class GPTNeoXModel(nn.Module):
242
243 class GPTNeoXForCausalLM(nn.Module, SupportsPP):
244
245 - def __init__(
246 - self,
247 - vllm_config: VllmConfig,
248 - prefix: str = "",
249 - ):
245 + def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
246 super().__init__()
247 config = vllm_config.model_config.hf_config
248 quant_config = vllm_config.quant_config
vllm/model_executor/models/internvl.py
+1 -1
@@ -409,7 +409,7 @@ input_pipeline = InternVLInputPipeline(IMG_START, IMG_END, IMG_CONTEXT)
409 @INPUT_REGISTRY.register_input_processor(input_pipeline.input_processor)
410 class InternVLChatModel(nn.Module, SupportsMultiModal, SupportsPP):
411
412 - def __init__(self, vllm_config: VllmConfig, prefix: str = "") -> None:
412 + def __init__(self, *, vllm_config: VllmConfig, prefix: str = "") -> None:
413 super().__init__()
414
415 config = vllm_config.model_config.hf_config
vllm/model_executor/models/jais.py
+1 -5
@@ -286,11 +286,7 @@ class JAISModel(nn.Module):
286
287 class JAISLMHeadModel(nn.Module, SupportsPP):
288
289 - def __init__(
290 - self,
291 - vllm_config: VllmConfig,
292 - prefix: str = "",
293 - ):
289 + def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
290 super().__init__()
291 config = vllm_config.model_config.hf_config
292 quant_config = vllm_config.quant_config
vllm/model_executor/models/llava.py
+1 -1
@@ -259,7 +259,7 @@ def init_vision_tower_for_llava(
259 @INPUT_REGISTRY.register_input_processor(input_processor_for_llava)
260 class LlavaForConditionalGeneration(nn.Module, SupportsMultiModal, SupportsPP):
261
262 - def __init__(self, vllm_config: VllmConfig, prefix: str = "") -> None:
262 + def __init__(self, *, vllm_config: VllmConfig, prefix: str = "") -> None:
263 super().__init__()
264
265 config = vllm_config.model_config.hf_config
vllm/model_executor/models/llava_next.py
+1 -1
@@ -281,7 +281,7 @@ def input_processor_for_llava_next(ctx: InputContext,
281 class LlavaNextForConditionalGeneration(nn.Module, SupportsMultiModal,
282 SupportsPP):
283
284 - def __init__(self, vllm_config: VllmConfig, prefix: str = "") -> None:
284 + def __init__(self, *, vllm_config: VllmConfig, prefix: str = "") -> None:
285 super().__init__()
286 config = vllm_config.model_config.hf_config
287 quant_config = vllm_config.quant_config
vllm/model_executor/models/llava_next_video.py
+1 -1
@@ -253,7 +253,7 @@ class LlavaNextMultiModalProjector(nn.Module):
253 class LlavaNextVideoForConditionalGeneration(nn.Module, SupportsMultiModal,
254 SupportsPP):
255
256 - def __init__(self, vllm_config: VllmConfig, prefix: str = "") -> None:
256 + def __init__(self, *, vllm_config: VllmConfig, prefix: str = "") -> None:
257 super().__init__()
258 config = vllm_config.model_config.hf_config
259 quant_config = vllm_config.quant_config
vllm/model_executor/models/llava_onevision.py
+1 -1
@@ -404,7 +404,7 @@ class LlavaOnevisionMultiModalProjector(nn.Module):
404 class LlavaOnevisionForConditionalGeneration(nn.Module, SupportsMultiModal,
405 SupportsPP):
406
407 - def __init__(self, vllm_config: VllmConfig, prefix: str = "") -> None:
407 + def __init__(self, *, vllm_config: VllmConfig, prefix: str = "") -> None:
408 super().__init__()
409 config = vllm_config.model_config.hf_config
410 quant_config = vllm_config.quant_config
vllm/model_executor/models/medusa.py
+1 -1
@@ -44,7 +44,7 @@ class Medusa(nn.Module):
44 in the draft checkpoint (using key token_map). Also, the draft config
45 needs to have truncated_vocab_size (=k) as an attribute."""
46
47 - def __init__(self, vllm_config: VllmConfig, prefix: str = "") -> None:
47 + def __init__(self, *, vllm_config: VllmConfig, prefix: str = "") -> None:
48 config = vllm_config.model_config.hf_config
49 super().__init__()
50 self.config = config
vllm/model_executor/models/utils.py
+3 -7
@@ -14,7 +14,6 @@ from vllm.attention.selector import (_Backend, backend_name_to_enum,
14 from vllm.config import VllmConfig
15 from vllm.logger import init_logger
16 from vllm.model_executor.model_loader.weight_utils import default_weight_loader
17 -from vllm.model_executor.models import ModelRegistry
17 from vllm.multimodal import MultiModalPlaceholderMap, NestedTensors
18 from vllm.platforms import current_platform
19 from vllm.sequence import IntermediateTensors
@@ -240,12 +239,9 @@ def init_vllm_registered_model(
239 Helper function to initialize an inner model registered to vLLM,
240 based on the arguments passed to the outer vLLM model.
241 """
243 - model_class, _ = ModelRegistry.resolve_model_cls(hf_config.architectures)
244 -
245 - return model_class(
246 - vllm_config=vllm_config.with_hf_config(hf_config),
247 - prefix=prefix,
248 - )
242 + from vllm.model_executor.model_loader.loader import _initialize_model
243 + vllm_config = vllm_config.with_hf_config(hf_config)
244 + return _initialize_model(vllm_config, prefix)
245
246
247 @overload