A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 10:45 UTC
[Models] Add remaining model PP support (#7168)
Signed-off-by: Muralidhar Andoorveedu <muralidhar.andoorveedu@centml.ai> Signed-off-by: Murali Andoorveedu <muralidhar.andoorveedu@centml.ai> Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
Murali Andoorveedu committed
Oct 3, 2024 at 19:56 UTC
0f6d7a9a347944bffd2204cbf9686299e9dd6557
69 files changed
+2583
-1342
.buildkite/test-pipeline.yaml
+3
-1
@@ -146,7 +146,9 @@ steps:
146
source_file_dependencies:
147
- vllm/
148
- tests/test_regression
149
- command: pytest -v -s test_regression.py
149
+ commands:
150
+ - pip install modelscope
151
+ - pytest -v -s test_regression.py
152
working_dir: "/vllm-workspace/tests" # optional
153
154
- label: Engine Test # 10min
docs/source/models/supported_models.rst
+81
-16
@@ -12,201 +12,249 @@ Alongside each architecture, we include some popular models that use it.
12
Decoder-only Language Models
13
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
14
.. list-table::
15
- :widths: 25 25 50 5
15
+ :widths: 25 25 50 5 5
16
:header-rows: 1
17
18
* - Architecture
19
- Models
20
- Example HuggingFace Models
21
- :ref:`LoRA <lora>`
22
+ - :ref:`PP <distributed_serving>`
23
* - :code:`AquilaForCausalLM`
24
- Aquila, Aquila2
25
- :code:`BAAI/Aquila-7B`, :code:`BAAI/AquilaChat-7B`, etc.
26
- ✅︎
27
+ - ✅︎
28
* - :code:`ArcticForCausalLM`
29
- Arctic
30
- :code:`Snowflake/snowflake-arctic-base`, :code:`Snowflake/snowflake-arctic-instruct`, etc.
31
-
32
+ - ✅︎
33
* - :code:`BaiChuanForCausalLM`
34
- Baichuan2, Baichuan
35
- :code:`baichuan-inc/Baichuan2-13B-Chat`, :code:`baichuan-inc/Baichuan-7B`, etc.
36
- ✅︎
37
+ - ✅︎
38
* - :code:`BloomForCausalLM`
39
- BLOOM, BLOOMZ, BLOOMChat
40
- :code:`bigscience/bloom`, :code:`bigscience/bloomz`, etc.
41
-
42
+ - ✅︎
43
* - :code:`ChatGLMModel`
44
- ChatGLM
45
- :code:`THUDM/chatglm2-6b`, :code:`THUDM/chatglm3-6b`, etc.
46
- ✅︎
47
+ - ✅︎
48
* - :code:`CohereForCausalLM`
49
- Command-R
50
- :code:`CohereForAI/c4ai-command-r-v01`, etc.
45
- -
51
+ - ✅︎
52
+ - ✅︎
53
* - :code:`DbrxForCausalLM`
54
- DBRX
55
- :code:`databricks/dbrx-base`, :code:`databricks/dbrx-instruct`, etc.
56
-
57
+ - ✅︎
58
* - :code:`DeciLMForCausalLM`
59
- DeciLM
60
- :code:`Deci/DeciLM-7B`, :code:`Deci/DeciLM-7B-instruct`, etc.
61
-
62
+ - ✅︎
63
* - :code:`DeepseekForCausalLM`
64
- DeepSeek
65
- :code:`deepseek-ai/deepseek-llm-67b-base`, :code:`deepseek-ai/deepseek-llm-7b-chat` etc.
66
-
67
+ - ✅︎
68
* - :code:`DeepseekV2ForCausalLM`
69
- DeepSeek-V2
70
- :code:`deepseek-ai/DeepSeek-V2`, :code:`deepseek-ai/DeepSeek-V2-Chat` etc.
71
-
72
+ - ✅︎
73
* - :code:`ExaoneForCausalLM`
74
- EXAONE-3
75
- :code:`LGAI-EXAONE/EXAONE-3.0-7.8B-Instruct`, etc.
76
- ✅︎
77
+ - ✅︎
78
* - :code:`FalconForCausalLM`
79
- Falcon
80
- :code:`tiiuae/falcon-7b`, :code:`tiiuae/falcon-40b`, :code:`tiiuae/falcon-rw-7b`, etc.
81
-
82
+ - ✅︎
83
* - :code:`GemmaForCausalLM`
84
- Gemma
85
- :code:`google/gemma-2b`, :code:`google/gemma-7b`, etc.
86
- ✅︎
87
+ - ✅︎
88
* - :code:`Gemma2ForCausalLM`
89
- Gemma2
90
- :code:`google/gemma-2-9b`, :code:`google/gemma-2-27b`, etc.
91
- ✅︎
92
+ - ✅︎
93
* - :code:`GPT2LMHeadModel`
94
- GPT-2
95
- :code:`gpt2`, :code:`gpt2-xl`, etc.
96
-
97
+ - ✅︎
98
* - :code:`GPTBigCodeForCausalLM`
99
- StarCoder, SantaCoder, WizardCoder
100
- :code:`bigcode/starcoder`, :code:`bigcode/gpt_bigcode-santacoder`, :code:`WizardLM/WizardCoder-15B-V1.0`, etc.
101
- ✅︎
102
+ - ✅︎
103
* - :code:`GPTJForCausalLM`
104
- GPT-J
105
- :code:`EleutherAI/gpt-j-6b`, :code:`nomic-ai/gpt4all-j`, etc.
106
-
107
+ - ✅︎
108
* - :code:`GPTNeoXForCausalLM`
109
- GPT-NeoX, Pythia, OpenAssistant, Dolly V2, StableLM
110
- :code:`EleutherAI/gpt-neox-20b`, :code:`EleutherAI/pythia-12b`, :code:`OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5`, :code:`databricks/dolly-v2-12b`, :code:`stabilityai/stablelm-tuned-alpha-7b`, etc.
111
-
112
+ - ✅︎
113
* - :code:`GraniteForCausalLM`
114
- PowerLM
115
- :code:`ibm/PowerLM-3b` etc.
116
- ✅︎
117
+ - ✅︎
118
* - :code:`GraniteMoeForCausalLM`
119
- PowerMoE
120
- :code:`ibm/PowerMoE-3b` etc.
121
- ✅︎
122
+ - ✅︎
123
* - :code:`InternLMForCausalLM`
124
- InternLM
125
- :code:`internlm/internlm-7b`, :code:`internlm/internlm-chat-7b`, etc.
126
- ✅︎
127
+ - ✅︎
128
* - :code:`InternLM2ForCausalLM`
129
- InternLM2
130
- :code:`internlm/internlm2-7b`, :code:`internlm/internlm2-chat-7b`, etc.
131
-
132
+ - ✅︎
133
* - :code:`JAISLMHeadModel`
134
- Jais
135
- :code:`core42/jais-13b`, :code:`core42/jais-13b-chat`, :code:`core42/jais-30b-v3`, :code:`core42/jais-30b-chat-v3`, etc.
136
-
137
+ - ✅︎
138
* - :code:`JambaForCausalLM`
139
- Jamba
140
- :code:`ai21labs/AI21-Jamba-1.5-Large`, :code:`ai21labs/AI21-Jamba-1.5-Mini`, :code:`ai21labs/Jamba-v0.1`, etc.
141
- ✅︎
142
+ -
143
* - :code:`LlamaForCausalLM`
144
- Llama 3.1, Llama 3, Llama 2, LLaMA, Yi
145
- :code:`meta-llama/Meta-Llama-3.1-405B-Instruct`, :code:`meta-llama/Meta-Llama-3.1-70B`, :code:`meta-llama/Meta-Llama-3-70B-Instruct`, :code:`meta-llama/Llama-2-70b-hf`, :code:`01-ai/Yi-34B`, etc.
146
- ✅︎
147
+ - ✅︎
148
* - :code:`MiniCPMForCausalLM`
149
- MiniCPM
150
- :code:`openbmb/MiniCPM-2B-sft-bf16`, :code:`openbmb/MiniCPM-2B-dpo-bf16`, etc.
125
- -
151
+ - ✅︎
152
+ - ✅︎
153
* - :code:`MiniCPM3ForCausalLM`
154
- MiniCPM3
155
- :code:`openbmb/MiniCPM3-4B`, etc.
129
- -
156
+ - ✅︎
157
+ - ✅︎
158
* - :code:`MistralForCausalLM`
159
- Mistral, Mistral-Instruct
160
- :code:`mistralai/Mistral-7B-v0.1`, :code:`mistralai/Mistral-7B-Instruct-v0.1`, etc.
161
- ✅︎
162
+ - ✅︎
163
* - :code:`MixtralForCausalLM`
164
- Mixtral-8x7B, Mixtral-8x7B-Instruct
165
- :code:`mistralai/Mixtral-8x7B-v0.1`, :code:`mistralai/Mixtral-8x7B-Instruct-v0.1`, :code:`mistral-community/Mixtral-8x22B-v0.1`, etc.
166
- ✅︎
167
+ - ✅︎
168
* - :code:`MPTForCausalLM`
169
- MPT, MPT-Instruct, MPT-Chat, MPT-StoryWriter
170
- :code:`mosaicml/mpt-7b`, :code:`mosaicml/mpt-7b-storywriter`, :code:`mosaicml/mpt-30b`, etc.
171
-
172
+ - ✅︎
173
* - :code:`NemotronForCausalLM`
174
- Nemotron-3, Nemotron-4, Minitron
175
- :code:`nvidia/Minitron-8B-Base`, :code:`mgoin/Nemotron-4-340B-Base-hf-FP8`, etc.
176
- ✅︎
146
- * - :code:`OLMoEForCausalLM`
147
- - OLMoE
148
- - :code:`allenai/OLMoE-1B-7B-0924`, :code:`allenai/OLMoE-1B-7B-0924-Instruct`, etc.
149
- -
177
+ - ✅︎
178
* - :code:`OLMoForCausalLM`
179
- OLMo
180
- :code:`allenai/OLMo-1B-hf`, :code:`allenai/OLMo-7B-hf`, etc.
181
-
182
+ - ✅︎
183
+ * - :code:`OLMoEForCausalLM`
184
+ - OLMoE
185
+ - :code:`allenai/OLMoE-1B-7B-0924`, :code:`allenai/OLMoE-1B-7B-0924-Instruct`, etc.
186
+ - ✅︎
187
+ - ✅︎
188
* - :code:`OPTForCausalLM`
189
- OPT, OPT-IML
190
- :code:`facebook/opt-66b`, :code:`facebook/opt-iml-max-30b`, etc.
191
-
192
+ - ✅︎
193
* - :code:`OrionForCausalLM`
194
- Orion
195
- :code:`OrionStarAI/Orion-14B-Base`, :code:`OrionStarAI/Orion-14B-Chat`, etc.
196
-
197
+ - ✅︎
198
* - :code:`PhiForCausalLM`
199
- Phi
200
- :code:`microsoft/phi-1_5`, :code:`microsoft/phi-2`, etc.
201
- ✅︎
202
+ - ✅︎
203
* - :code:`Phi3ForCausalLM`
204
- Phi-3
205
- :code:`microsoft/Phi-3-mini-4k-instruct`, :code:`microsoft/Phi-3-mini-128k-instruct`, :code:`microsoft/Phi-3-medium-128k-instruct`, etc.
169
- -
206
+ - ✅︎
207
+ - ✅︎
208
* - :code:`Phi3SmallForCausalLM`
209
- Phi-3-Small
210
- :code:`microsoft/Phi-3-small-8k-instruct`, :code:`microsoft/Phi-3-small-128k-instruct`, etc.
211
-
212
+ - ✅︎
213
* - :code:`PhiMoEForCausalLM`
214
- Phi-3.5-MoE
215
- :code:`microsoft/Phi-3.5-MoE-instruct`, etc.
177
- -
216
+ - ✅︎
217
+ - ✅︎
218
* - :code:`PersimmonForCausalLM`
219
- Persimmon
220
- :code:`adept/persimmon-8b-base`, :code:`adept/persimmon-8b-chat`, etc.
221
-
222
+ - ✅︎
223
* - :code:`QWenLMHeadModel`
224
- Qwen
225
- :code:`Qwen/Qwen-7B`, :code:`Qwen/Qwen-7B-Chat`, etc.
226
-
227
+ - ✅︎
228
* - :code:`Qwen2ForCausalLM`
229
- Qwen2
230
- :code:`Qwen/Qwen2-beta-7B`, :code:`Qwen/Qwen2-beta-7B-Chat`, etc.
231
- ✅︎
232
+ - ✅︎
233
* - :code:`Qwen2MoeForCausalLM`
234
- Qwen2MoE
235
- :code:`Qwen/Qwen1.5-MoE-A2.7B`, :code:`Qwen/Qwen1.5-MoE-A2.7B-Chat`, etc.
236
-
237
+ - ✅︎
238
* - :code:`StableLmForCausalLM`
239
- StableLM
240
- :code:`stabilityai/stablelm-3b-4e1t`, :code:`stabilityai/stablelm-base-alpha-7b-v2`, etc.
241
-
242
+ - ✅︎
243
* - :code:`Starcoder2ForCausalLM`
244
- Starcoder2
245
- :code:`bigcode/starcoder2-3b`, :code:`bigcode/starcoder2-7b`, :code:`bigcode/starcoder2-15b`, etc.
246
-
247
+ - ✅︎
248
* - :code:`SolarForCausalLM`
203
- - EXAONE-3
249
+ - Solar Pro
250
- :code:`upstage/solar-pro-preview-instruct`, etc.
205
- -
251
+ - ✅︎
252
+ - ✅︎
253
* - :code:`XverseForCausalLM`
207
- - Xverse
254
+ - XVERSE
255
- :code:`xverse/XVERSE-7B-Chat`, :code:`xverse/XVERSE-13B-Chat`, :code:`xverse/XVERSE-65B-Chat`, etc.
209
- -
256
+ - ✅︎
257
+ - ✅︎
258
259
.. note::
260
Currently, the ROCm version of vLLM supports Mistral and Mixtral only for context lengths up to 4096.
@@ -217,7 +265,7 @@ Multimodal Language Models
265
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
266
267
.. list-table::
220
- :widths: 25 25 25 25 5
268
+ :widths: 25 25 25 25 5 5
269
:header-rows: 1
270
271
* - Architecture
@@ -225,86 +273,103 @@ Multimodal Language Models
273
- Modalities
274
- Example HuggingFace Models
275
- :ref:`LoRA <lora>`
276
+ - :ref:`PP <distributed_serving>`
277
* - :code:`Blip2ForConditionalGeneration`
278
- BLIP-2
279
- Image\ :sup:`E`
280
- :code:`Salesforce/blip2-opt-2.7b`, :code:`Salesforce/blip2-opt-6.7b`, etc.
281
-
282
+ - ✅︎
283
* - :code:`ChameleonForConditionalGeneration`
284
- Chameleon
285
- Image
286
- :code:`facebook/chameleon-7b` etc.
287
-
288
+ - ✅︎
289
* - :code:`FuyuForCausalLM`
290
- Fuyu
291
- Image
292
- :code:`adept/fuyu-8b` etc.
293
-
294
+ - ✅︎
295
* - :code:`InternVLChatModel`
296
- InternVL2
297
- Image\ :sup:`E+`
298
- :code:`OpenGVLab/InternVL2-4B`, :code:`OpenGVLab/InternVL2-8B`, etc.
299
-
300
+ - ✅︎
301
* - :code:`LlavaForConditionalGeneration`
302
- LLaVA-1.5
303
- Image\ :sup:`E+`
304
- :code:`llava-hf/llava-1.5-7b-hf`, :code:`llava-hf/llava-1.5-13b-hf`, etc.
305
-
306
+ - ✅︎
307
* - :code:`LlavaNextForConditionalGeneration`
308
- LLaVA-NeXT
309
- Image\ :sup:`E+`
310
- :code:`llava-hf/llava-v1.6-mistral-7b-hf`, :code:`llava-hf/llava-v1.6-vicuna-7b-hf`, etc.
311
-
312
+ - ✅︎
313
* - :code:`LlavaNextVideoForConditionalGeneration`
314
- LLaVA-NeXT-Video
315
- Video
316
- :code:`llava-hf/LLaVA-NeXT-Video-7B-hf`, etc.
317
-
318
+ - ✅︎
319
* - :code:`LlavaOnevisionForConditionalGeneration`
320
- LLaVA-Onevision
321
- Image\ :sup:`+` / Video
322
- :code:`llava-hf/llava-onevision-qwen2-7b-ov-hf`, :code:`llava-hf/llava-onevision-qwen2-0.5b-ov-hf`, etc.
323
-
324
+ - ✅︎
325
* - :code:`MiniCPMV`
326
- MiniCPM-V
327
- Image\ :sup:`+`
328
- :code:`openbmb/MiniCPM-V-2` (see note), :code:`openbmb/MiniCPM-Llama3-V-2_5`, :code:`openbmb/MiniCPM-V-2_6`, etc.
272
- -
329
+ - ✅︎
330
+ - ✅︎
331
* - :code:`MllamaForConditionalGeneration`
332
- Llama 3.2
333
- Image
334
- :code:`meta-llama/Llama-3.2-90B-Vision-Instruct`, :code:`meta-llama/Llama-3.2-11B-Vision`, etc.
335
-
336
+ -
337
* - :code:`PaliGemmaForConditionalGeneration`
338
- PaliGemma
339
- Image\ :sup:`E`
340
- :code:`google/paligemma-3b-pt-224`, :code:`google/paligemma-3b-mix-224`, etc.
341
-
342
+ - ✅︎
343
* - :code:`Phi3VForCausalLM`
344
- Phi-3-Vision, Phi-3.5-Vision
345
- Image\ :sup:`E+`
346
- :code:`microsoft/Phi-3-vision-128k-instruct`, :code:`microsoft/Phi-3.5-vision-instruct` etc.
347
-
348
+ - ✅︎
349
* - :code:`PixtralForConditionalGeneration`
350
- Pixtral
351
- Image\ :sup:`+`
352
- :code:`mistralai/Pixtral-12B-2409`
353
-
354
+ - ✅︎
355
* - :code:`QWenLMHeadModel`
356
- Qwen-VL
357
- Image\ :sup:`E+`
358
- :code:`Qwen/Qwen-VL`, :code:`Qwen/Qwen-VL-Chat`, etc.
359
-
360
+ - ✅︎
361
* - :code:`Qwen2VLForConditionalGeneration`
362
- Qwen2-VL
363
- Image\ :sup:`E+` / Video\ :sup:`+`
364
- :code:`Qwen/Qwen2-VL-2B-Instruct`, :code:`Qwen/Qwen2-VL-7B-Instruct`, :code:`Qwen/Qwen2-VL-72B-Instruct`, etc.
365
-
366
+ - ✅︎
367
* - :code:`UltravoxModel`
368
- Ultravox
369
- Audio\ :sup:`E+`
370
- :code:`fixie-ai/ultravox-v0_3`
371
-
372
+ - ✅︎
373
374
| :sup:`E` Pre-computed embeddings can be inputted for this modality.
375
| :sup:`+` Multiple items can be inputted per text prompt for this modality.
requirements-test.txt
+2
-2
@@ -10,8 +10,8 @@ pytest-shard
10
awscli
11
einops # required for MPT, qwen-vl and Mamba
12
httpx
13
-librosa # required for audio test
14
-opencv-python # required for video test
13
+librosa # required for audio tests
14
+opencv-python # required for video tests
15
peft
16
requests
17
ray[adag]==2.35
tests/distributed/test_pipeline_parallel.py
+240
-57
@@ -6,6 +6,8 @@ WARNING: This test runs in both single-node (4 GPUs) and multi-node
6
to fail.
7
"""
8
import os
9
+from dataclasses import dataclass
10
+from typing import List, NamedTuple, Optional
11
12
import pytest
13
@@ -18,49 +20,256 @@ logger = init_logger("test_pipeline_parallel")
20
VLLM_MULTI_NODE = os.getenv("VLLM_MULTI_NODE", "0") == "1"
21
22
23
+class ParallelSetup(NamedTuple):
24
+ tp_size: int
25
+ pp_size: int
26
+ eager_mode: bool
27
+ chunked_prefill: bool
28
+
29
+
30
+@dataclass
31
+class PPTestSettings:
32
+ parallel_setups: List[ParallelSetup]
33
+ distributed_backends: List[str]
34
+ trust_remote_code: bool
35
+ tokenizer_mode: Optional[str]
36
+
37
+ @staticmethod
38
+ def detailed(
39
+ *,
40
+ tp_base: int = 1,
41
+ pp_base: int = 2,
42
+ trust_remote_code: bool = False,
43
+ tokenizer_mode: Optional[str] = None,
44
+ ):
45
+ return PPTestSettings(
46
+ parallel_setups=[
47
+ ParallelSetup(tp_size=tp_base,
48
+ pp_size=pp_base,
49
+ eager_mode=False,
50
+ chunked_prefill=False),
51
+ ParallelSetup(tp_size=tp_base,
52
+ pp_size=2 * pp_base,
53
+ eager_mode=False,
54
+ chunked_prefill=True),
55
+ ParallelSetup(tp_size=tp_base,
56
+ pp_size=2 * pp_base,
57
+ eager_mode=True,
58
+ chunked_prefill=False),
59
+ ParallelSetup(tp_size=2 * tp_base,
60
+ pp_size=pp_base,
61
+ eager_mode=False,
62
+ chunked_prefill=True),
63
+ ParallelSetup(tp_size=2 * tp_base,
64
+ pp_size=pp_base,
65
+ eager_mode=True,
66
+ chunked_prefill=False),
67
+ ],
68
+ distributed_backends=["mp", "ray"],
69
+ trust_remote_code=trust_remote_code,
70
+ tokenizer_mode=tokenizer_mode,
71
+ )
72
+
73
+ @staticmethod
74
+ def fast(
75
+ *,
76
+ tp_base: int = 1,
77
+ pp_base: int = 2,
78
+ trust_remote_code: bool = False,
79
+ tokenizer_mode: Optional[str] = None,
80
+ ):
81
+ return PPTestSettings(
82
+ parallel_setups=[
83
+ ParallelSetup(tp_size=tp_base,
84
+ pp_size=pp_base,
85
+ eager_mode=True,
86
+ chunked_prefill=False),
87
+ ],
88
+ distributed_backends=["mp"],
89
+ trust_remote_code=trust_remote_code,
90
+ tokenizer_mode=tokenizer_mode,
91
+ )
92
+
93
+ def iter_params(self, model_name: str):
94
+ for parallel_setup in self.parallel_setups:
95
+ for distributed_backend in self.distributed_backends:
96
+ yield (model_name, parallel_setup, distributed_backend,
97
+ self.trust_remote_code, self.tokenizer_mode)
98
+
99
+
100
+# yapf: disable
101
+GENERATION_MODEL_SETTINGS = {
102
+ # [DETAILED TESTS]
103
+ "meta-llama/Meta-Llama-3-8B": PPTestSettings.detailed(),
104
+ # [FAST TESTS]
105
+ # Uses Llama
106
+ # "BAAI/AquilaChat-7B": PPTestSettings.fast(),
107
+ # TODO: Test on larger GPU
108
+ # "Snowflake/snowflake-arctic-instruct": PPTestSettings.fast(trust_remote_code=True), # noqa: E501
109
+ "baichuan-inc/Baichuan-7B": PPTestSettings.fast(trust_remote_code=True),
110
+ "baichuan-inc/Baichuan2-13B-Chat": PPTestSettings.fast(trust_remote_code=True), # noqa: E501
111
+ "bigscience/bloomz-1b1": PPTestSettings.fast(),
112
+ "THUDM/chatglm3-6b": PPTestSettings.fast(trust_remote_code=True),
113
+ "CohereForAI/c4ai-command-r-v01": PPTestSettings.fast(tp_base=2, trust_remote_code=True), # noqa: E501
114
+ # TODO: Test on larger GPU
115
+ # "databricks/dbrx-instruct": PPTestSettings.fast(),
116
+ "Deci/DeciLM-7B-instruct": PPTestSettings.fast(trust_remote_code=True),
117
+ "deepseek-ai/deepseek-llm-7b-chat": PPTestSettings.fast(),
118
+ "deepseek-ai/DeepSeek-V2-Lite-Chat": PPTestSettings.fast(trust_remote_code=True), # noqa: E501
119
+ "LGAI-EXAONE/EXAONE-3.0-7.8B-Instruct": PPTestSettings.fast(),
120
+ "tiiuae/falcon-7b": PPTestSettings.fast(),
121
+ "google/gemma-2b": PPTestSettings.fast(),
122
+ "google/gemma-2-9b": PPTestSettings.fast(),
123
+ "gpt2": PPTestSettings.fast(),
124
+ "bigcode/starcoder": PPTestSettings.fast(),
125
+ "EleutherAI/gpt-j-6b": PPTestSettings.fast(),
126
+ "EleutherAI/pythia-12b": PPTestSettings.fast(),
127
+ "ibm/PowerLM-3b": PPTestSettings.fast(),
128
+ "ibm/PowerMoE-3b": PPTestSettings.fast(),
129
+ # Uses Llama
130
+ # "internlm/internlm-chat-7b": PPTestSettings.fast(),
131
+ "internlm/internlm2-chat-7b": PPTestSettings.fast(trust_remote_code=True),
132
+ "core42/jais-13b-chat": PPTestSettings.fast(),
133
+ # TODO: Implement PP
134
+ # "ai21labs/AI21-Jamba-1.5-Mini": PPTestSettings.fast(),
135
+ "openbmb/MiniCPM-2B-sft-bf16": PPTestSettings.fast(trust_remote_code=True),
136
+ "openbmb/MiniCPM3-4B": PPTestSettings.fast(trust_remote_code=True),
137
+ # Uses Llama
138
+ # "mistralai/Mistral-7B-Instruct-v0.1": PPTestSettings.fast(),
139
+ "mistralai/Mixtral-8x7B-Instruct-v0.1": PPTestSettings.fast(tp_base=4),
140
+ "mosaicml/mpt-7b": PPTestSettings.fast(),
141
+ "nvidia/Minitron-8B-Base": PPTestSettings.fast(),
142
+ "allenai/OLMoE-1B-7B-0924-Instruct": PPTestSettings.fast(),
143
+ "allenai/OLMo-1B-hf": PPTestSettings.fast(),
144
+ "facebook/opt-iml-max-1.3b": PPTestSettings.fast(),
145
+ "OrionStarAI/Orion-14B-Chat": PPTestSettings.fast(trust_remote_code=True),
146
+ "microsoft/phi-2": PPTestSettings.fast(),
147
+ "microsoft/Phi-3-mini-4k-instruct": PPTestSettings.fast(),
148
+ "microsoft/Phi-3-small-8k-instruct": PPTestSettings.fast(trust_remote_code=True), # noqa: E501
149
+ # FIXME: https://github.com/vllm-project/vllm/issues/8553
150
+ # "microsoft/Phi-3.5-MoE-instruct": PPTestSettings.fast(trust_remote_code=True), # noqa: E501
151
+ "adept/persimmon-8b-chat": PPTestSettings.fast(),
152
+ "Qwen/Qwen-7B-Chat": PPTestSettings.fast(trust_remote_code=True),
153
+ "Qwen/Qwen2-beta-7B-Chat": PPTestSettings.fast(),
154
+ "Qwen/Qwen1.5-MoE-A2.7B-Chat": PPTestSettings.fast(),
155
+ "stabilityai/stablelm-3b-4e1t": PPTestSettings.fast(),
156
+ "bigcode/starcoder2-3b": PPTestSettings.fast(),
157
+ "upstage/solar-pro-preview-instruct": PPTestSettings.fast(tp_base=2),
158
+ # FIXME: Cannot load tokenizer in latest transformers version
159
+ # "xverse/XVERSE-7B-Chat": PPTestSettings.fast(trust_remote_code=True),
160
+}
161
+
162
+EMBEDDING_MODEL_SETTINGS = { # type: ignore[var-annotated]
163
+ # [FAST TESTS]
164
+ # Uses Llama
165
+ # "intfloat/e5-mistral-7b-instruct": PPTestSettings.fast(),
166
+}
167
+
168
+MULTIMODAL_MODEL_SETTINGS = {
169
+ # [FAST TESTS]
170
+ "Salesforce/blip2-opt-2.7b": PPTestSettings.fast(),
171
+ "facebook/chameleon-7b": PPTestSettings.fast(),
172
+ "adept/fuyu-8b": PPTestSettings.fast(),
173
+ "OpenGVLab/InternVL2-1B": PPTestSettings.fast(trust_remote_code=True),
174
+ "llava-hf/llava-1.5-7b-hf": PPTestSettings.fast(),
175
+ "llava-hf/llava-v1.6-mistral-7b-hf": PPTestSettings.fast(),
176
+ "llava-hf/LLaVA-NeXT-Video-7B-hf": PPTestSettings.fast(),
177
+ "llava-hf/llava-onevision-qwen2-0.5b-ov-hf": PPTestSettings.fast(),
178
+ "openbmb/MiniCPM-Llama3-V-2_5": PPTestSettings.fast(trust_remote_code=True),
179
+ # TODO: Implement PP
180
+ # "meta-llama/Llama-3.2-11B-Vision-Instruct": PPTestSettings.fast(),
181
+ "microsoft/Phi-3-vision-128k-instruct": PPTestSettings.fast(trust_remote_code=True), # noqa: E501
182
+ "mistralai/Pixtral-12B-2409": PPTestSettings.fast(tp_base=2, tokenizer_mode="mistral"), # noqa: E501
183
+ "Qwen/Qwen-VL-Chat": PPTestSettings.fast(trust_remote_code=True),
184
+ "Qwen/Qwen2-VL-2B-Instruct": PPTestSettings.fast(),
185
+ "fixie-ai/ultravox-v0_3": PPTestSettings.fast(),
186
+}
187
+
188
+CONDITIONAL_GENERATION_MODEL_SETTINGS = { # type: ignore[var-annotated]
189
+ # [FAST TESTS]
190
+ # TODO: Implement PP
191
+ # "facebook/bart-base": PPTestSettings.fast(),
192
+}
193
+# yapf: enable
194
+
195
+MODEL_SETTINGS = {
196
+ **GENERATION_MODEL_SETTINGS,
197
+ **EMBEDDING_MODEL_SETTINGS,
198
+ **MULTIMODAL_MODEL_SETTINGS,
199
+}
200
+
201
+# You can update this on your local machine to run specific tests
202
+TEST_MODELS = [
203
+ "meta-llama/Meta-Llama-3-8B",
204
+ "facebook/chameleon-7b",
205
+ "OpenGVLab/InternVL2-1B",
206
+ "microsoft/Phi-3-vision-128k-instruct",
207
+ "mistralai/Pixtral-12B-2409",
208
+ "fixie-ai/ultravox-v0_3",
209
+]
210
+
211
+
212
@pytest.mark.parametrize(
22
- ("TP_SIZE, PP_SIZE, EAGER_MODE, CHUNKED_PREFILL, TRUST_REMOTE_CODE, "
23
- "MODEL_NAME, DIST_BACKEND"),
213
+ ("model_name", "parallel_setup", "distributed_backend",
214
+ "trust_remote_code", "tokenizer_mode"),
215
[
25
- (2, 2, 0, 1, 0, "meta-llama/Meta-Llama-3-8B", "mp"),
26
- (2, 2, 1, 0, 0, "meta-llama/Meta-Llama-3-8B", "mp"),
27
- (1, 3, 0, 0, 0, "meta-llama/Meta-Llama-3-8B", "mp"),
28
- (1, 4, 0, 1, 0, "meta-llama/Meta-Llama-3-8B", "mp"),
29
- (1, 4, 1, 0, 0, "meta-llama/Meta-Llama-3-8B", "mp"),
30
- (1, 3, 0, 0, 0, "meta-llama/Meta-Llama-3-8B", "ray"),
31
- (1, 4, 0, 1, 0, "meta-llama/Meta-Llama-3-8B", "ray"),
32
- (1, 4, 1, 0, 0, "meta-llama/Meta-Llama-3-8B", "ray"),
33
- (2, 2, 1, 0, 0, "meta-llama/Meta-Llama-3-8B", "ray"),
34
- (2, 2, 0, 1, 0, "meta-llama/Meta-Llama-3-8B", "ray"),
35
- # NOTE: InternVL2 multi-node tests are flaky,
36
- # use mp backend to skip the multi-node tests
37
- (1, 2, 1, 1, 1, "OpenGVLab/InternVL2-1B", "mp"),
38
- (1, 2, 1, 1, 1, "OpenGVLab/InternVL2-2B", "mp"),
39
- (1, 2, 1, 0, 1, "OpenGVLab/InternVL2-4B", "mp"),
40
- (1, 2, 0, 1, 0, "Qwen/Qwen2-VL-2B-Instruct", "mp"),
41
- # TP only models
42
- (2, 1, 1, 0, 0, "adept/fuyu-8b", "mp"),
216
+ params for model_name, settings in MODEL_SETTINGS.items()
217
+ for params in settings.iter_params(model_name)
218
+ if model_name in TEST_MODELS
219
],
220
)
221
@fork_new_process_for_each_test
46
-def test_compare_tp(TP_SIZE, PP_SIZE, EAGER_MODE, CHUNKED_PREFILL,
47
- TRUST_REMOTE_CODE, MODEL_NAME, DIST_BACKEND):
48
- if VLLM_MULTI_NODE and DIST_BACKEND == "mp":
222
+def test_compare_tp(model_name: str, parallel_setup: ParallelSetup,
223
+ distributed_backend: str, trust_remote_code: bool,
224
+ tokenizer_mode: Optional[str], num_gpus_available):
225
+ tp_size, pp_size, eager_mode, chunked_prefill = parallel_setup
226
+
227
+ if num_gpus_available < tp_size:
228
+ pytest.skip(f"Need at least {tp_size} GPUs to run the test")
229
+ if VLLM_MULTI_NODE and distributed_backend == "mp":
230
pytest.skip("Skipping multi-node pipeline parallel test for "
231
"multiprocessing distributed backend")
232
52
- pp_args = [
233
+ common_args = [
234
# use half precision for speed and memory savings in CI environment
235
"--dtype",
236
"float16",
237
"--max-model-len",
57
- "8192",
238
+ "2048",
239
+ "--max-num-seqs",
240
+ "8",
241
+ ]
242
+ if chunked_prefill:
243
+ common_args.append("--enable-chunked-prefill")
244
+ if eager_mode:
245
+ common_args.append("--enforce-eager")
246
+ if trust_remote_code:
247
+ common_args.append("--trust-remote-code")
248
+ if tokenizer_mode:
249
+ common_args.extend(["--tokenizer-mode", tokenizer_mode])
250
+
251
+ if (distributed_backend == "ray" and tp_size == 2 and pp_size == 2
252
+ and chunked_prefill):
253
+ # Test Ray ADAG for a subset of the tests
254
+ pp_env = {
255
+ "VLLM_USE_RAY_COMPILED_DAG": "1",
256
+ "VLLM_USE_RAY_SPMD_WORKER": "1",
257
+ "VLLM_USE_RAY_COMPILED_DAG_NCCL_CHANNEL": "1",
258
+ }
259
+ # Temporary. Currently when zeromq + SPMD is used, it does not properly
260
+ # terminate because of aDAG issue.
261
+ common_args.append("--disable-frontend-multiprocessing")
262
+ else:
263
+ pp_env = None
264
+
265
+ pp_args = [
266
+ *common_args,
267
"--pipeline-parallel-size",
59
- str(PP_SIZE),
268
+ str(pp_size),
269
"--tensor-parallel-size",
61
- str(TP_SIZE),
270
+ str(tp_size),
271
"--distributed-executor-backend",
63
- DIST_BACKEND,
272
+ distributed_backend,
273
]
274
275
# compare without pipeline parallelism
@@ -69,41 +278,15 @@ def test_compare_tp(TP_SIZE, PP_SIZE, EAGER_MODE, CHUNKED_PREFILL,
278
# schedule all workers in a node other than the head node,
279
# which can cause the test to fail.
280
tp_args = [
72
- # use half precision for speed and memory savings in CI environment
73
- "--dtype",
74
- "float16",
75
- "--max-model-len",
76
- "8192",
281
+ *common_args,
282
"--tensor-parallel-size",
78
- str(max(TP_SIZE, 2)), # We only use 2 GPUs in the CI.
283
+ str(tp_size),
284
"--distributed-executor-backend",
285
"mp",
286
]
82
- if CHUNKED_PREFILL:
83
- pp_args.append("--enable-chunked-prefill")
84
- tp_args.append("--enable-chunked-prefill")
85
- if EAGER_MODE:
86
- pp_args.append("--enforce-eager")
87
- tp_args.append("--enforce-eager")
88
- if TRUST_REMOTE_CODE:
89
- pp_args.append("--trust-remote-code")
90
- tp_args.append("--trust-remote-code")
91
- pp_env = None
92
- if (DIST_BACKEND == "ray" and TP_SIZE == 2 and PP_SIZE == 2
93
- and CHUNKED_PREFILL):
94
- # Test Ray ADAG for a subset of the tests
95
- pp_env = {
96
- "VLLM_USE_RAY_COMPILED_DAG": "1",
97
- "VLLM_USE_RAY_SPMD_WORKER": "1",
98
- "VLLM_USE_RAY_COMPILED_DAG_NCCL_CHANNEL": "1",
99
- }
100
- # Temporary. Currently when zeromq + SPMD is used, it does not properly
101
- # terminate because of aDAG issue.
102
- pp_args.append("--disable-frontend-multiprocessing")
103
- tp_args.append("--disable-frontend-multiprocessing")
287
288
try:
106
- compare_two_settings(MODEL_NAME, pp_args, tp_args, pp_env)
289
+ compare_two_settings(model_name, pp_args, tp_args, pp_env)
290
except Exception:
291
if pp_env is None:
292
raise
tests/models/test_registry.py
+49
-3
@@ -1,9 +1,55 @@
1
+import warnings
2
+
3
import pytest
4
+import torch.cuda
5
6
from vllm.model_executor.models import _MODELS, ModelRegistry
7
+from vllm.platforms import current_platform
8
+
9
+from ..utils import fork_new_process_for_each_test
10
11
6
-@pytest.mark.parametrize("model_cls", _MODELS)
7
-def test_registry_imports(model_cls):
12
+@pytest.mark.parametrize("model_arch", _MODELS)
13
+def test_registry_imports(model_arch):
14
# Ensure all model classes can be imported successfully
9
- ModelRegistry.resolve_model_cls([model_cls])
15
+ ModelRegistry.resolve_model_cls(model_arch)
16
+
17
+
18
+@fork_new_process_for_each_test
19
+@pytest.mark.parametrize("model_arch,is_mm,init_cuda", [
20
+ ("LlamaForCausalLM", False, False),
21
+ ("MllamaForConditionalGeneration", True, False),
22
+ ("LlavaForConditionalGeneration", True, True),
23
+])
24
+def test_registry_is_multimodal(model_arch, is_mm, init_cuda):
25
+ assert ModelRegistry.is_multimodal_model(model_arch) is is_mm
26
+
27
+ if init_cuda and current_platform.is_cuda_alike():
28
+ assert not torch.cuda.is_initialized()
29
+
30
+ ModelRegistry.resolve_model_cls(model_arch)
31
+ if not torch.cuda.is_initialized():
32
+ warnings.warn(
33
+ "This model no longer initializes CUDA on import. "
34
+ "Please test using a different one.",
35
+ stacklevel=2)
36
+
37
+
38
+@fork_new_process_for_each_test
39
+@pytest.mark.parametrize("model_arch,is_pp,init_cuda", [
40
+ ("MLPSpeculatorPreTrainedModel", False, False),
41
+ ("DeepseekV2ForCausalLM", True, False),
42
+ ("Qwen2VLForConditionalGeneration", True, True),
43
+])
44
+def test_registry_is_pp(model_arch, is_pp, init_cuda):
45
+ assert ModelRegistry.is_pp_supported_model(model_arch) is is_pp
46
+
47
+ if init_cuda and current_platform.is_cuda_alike():
48
+ assert not torch.cuda.is_initialized()
49
+
50
+ ModelRegistry.resolve_model_cls(model_arch)
51
+ if not torch.cuda.is_initialized():
52
+ warnings.warn(
53
+ "This model no longer initializes CUDA on import. "
54
+ "Please test using a different one.",
55
+ stacklevel=2)
tests/utils.py
+19
-8
@@ -14,7 +14,6 @@ import openai
14
import pytest
15
import requests
16
from openai.types.completion import Completion
17
-from transformers import AutoTokenizer
17
from typing_extensions import ParamSpec
18
19
from tests.models.utils import TextTextLogprobs
@@ -24,6 +23,7 @@ from vllm.engine.arg_utils import AsyncEngineArgs
23
from vllm.entrypoints.openai.cli_args import make_arg_parser
24
from vllm.model_executor.model_loader.loader import get_model_loader
25
from vllm.platforms import current_platform
26
+from vllm.transformers_utils.tokenizer import get_tokenizer
27
from vllm.utils import (FlexibleArgumentParser, GB_bytes,
28
cuda_device_count_stateless, get_open_port, is_hip)
29
@@ -181,15 +181,26 @@ def compare_two_settings(model: str,
181
env2: The second set of environment variables to pass to the API server.
182
"""
183
184
- trust_remote_code = "--trust-remote-code"
185
- if trust_remote_code in arg1 or trust_remote_code in arg2:
186
- tokenizer = AutoTokenizer.from_pretrained(model,
187
- trust_remote_code=True)
188
- else:
189
- tokenizer = AutoTokenizer.from_pretrained(model)
184
+ trust_remote_code = False
185
+ for args in (arg1, arg2):
186
+ if "--trust-remote-code" in args:
187
+ trust_remote_code = True
188
+ break
189
+
190
+ tokenizer_mode = "auto"
191
+ for args in (arg1, arg2):
192
+ if "--tokenizer-mode" in args:
193
+ tokenizer_mode = args[args.index("--tokenizer-mode") + 1]
194
+ break
195
+
196
+ tokenizer = get_tokenizer(
197
+ model,
198
+ trust_remote_code=trust_remote_code,
199
+ tokenizer_mode=tokenizer_mode,
200
+ )
201
202
prompt = "Hello, my name is"
192
- token_ids = tokenizer(prompt)["input_ids"]
203
+ token_ids = tokenizer(prompt).input_ids
204
results = []
205
for args, env in ((arg1, env1), (arg2, env2)):
206
with RemoteOpenAIServer(model,
vllm/config.py
+19
-43
@@ -31,28 +31,7 @@ if TYPE_CHECKING:
31
logger = init_logger(__name__)
32
33
_EMBEDDING_MODEL_MAX_NUM_BATCHED_TOKENS = 32768
34
-_MULTIMODAL_MODEL_MAX_NUM_BATCHED_TOKENS = 4096
35
-
36
-_PP_SUPPORTED_MODELS = [
37
- "AquilaForCausalLM",
38
- "AquilaModel",
39
- "DeepseekV2ForCausalLM",
40
- "GPT2LMHeadModel",
41
- "InternLM2ForCausalLM",
42
- "InternLMForCausalLM",
43
- "InternVLChatModel",
44
- "JAISLMHeadModel",
45
- "LlamaForCausalLM",
46
- "LLaMAForCausalLM",
47
- "MistralForCausalLM",
48
- "MixtralForCausalLM",
49
- "NemotronForCausalLM",
50
- "Phi3ForCausalLM",
51
- "Qwen2ForCausalLM",
52
- "Qwen2MoeForCausalLM",
53
- "QWenLMHeadModel",
54
- "Qwen2VLForConditionalGeneration",
55
-]
34
+_MULTIMODAL_MODEL_MAX_NUM_BATCHED_TOKENS = 5120
35
36
37
class ModelConfig:
@@ -228,16 +207,14 @@ class ModelConfig:
207
self, limit_mm_per_prompt: Optional[Mapping[str, int]]
208
) -> Optional["MultiModalConfig"]:
209
architectures = getattr(self.hf_config, "architectures", [])
231
- if any(
232
- ModelRegistry.is_multimodal_model(arch)
233
- for arch in architectures):
210
+ if ModelRegistry.is_multimodal_model(architectures):
211
return MultiModalConfig(limit_per_prompt=limit_mm_per_prompt or {})
235
- else:
236
- if limit_mm_per_prompt:
237
- raise ValueError(
238
- "limit_mm_per_prompt is only supported for multimodal "
239
- "models.")
240
- return None
212
+
213
+ if limit_mm_per_prompt:
214
+ raise ValueError("`limit_mm_per_prompt` is only supported for "
215
+ "multimodal models.")
216
+
217
+ return None
218
219
def _verify_tokenizer_mode(self) -> None:
220
tokenizer_mode = self.tokenizer_mode.lower()
@@ -249,8 +226,7 @@ class ModelConfig:
226
227
def _verify_embedding_mode(self) -> None:
228
architectures = getattr(self.hf_config, "architectures", [])
252
- self.embedding_mode = any(
253
- ModelRegistry.is_embedding_model(arch) for arch in architectures)
229
+ self.embedding_mode = ModelRegistry.is_embedding_model(architectures)
230
231
def _parse_quant_hf_config(self):
232
quant_cfg = getattr(self.hf_config, "quantization_config", None)
@@ -417,17 +393,17 @@ class ModelConfig:
393
f"({tensor_parallel_size}).")
394
395
pipeline_parallel_size = parallel_config.pipeline_parallel_size
420
- architectures = getattr(self.hf_config, "architectures", [])
421
- if not all(arch in _PP_SUPPORTED_MODELS
422
- for arch in architectures) and pipeline_parallel_size > 1:
423
- raise NotImplementedError(
424
- "Pipeline parallelism is only supported for the following "
425
- f" architectures: {_PP_SUPPORTED_MODELS}.")
396
+ if pipeline_parallel_size > 1:
397
+ architectures = getattr(self.hf_config, "architectures", [])
398
+ if not ModelRegistry.is_pp_supported_model(architectures):
399
+ raise NotImplementedError(
400
+ "Pipeline parallelism is not supported for this model. "
401
+ "Supported models implement the `SupportsPP` interface.")
402
427
- if pipeline_parallel_size > 1 and self.use_async_output_proc:
428
- logger.warning("Async output processor is not supported with "
429
- "pipeline parallelism currently. Disabling it.")
430
- self.use_async_output_proc = False
403
+ if self.use_async_output_proc:
404
+ logger.warning("Async output processor is not supported with "
405
+ "pipeline parallelism currently. Disabling it.")
406
+ self.use_async_output_proc = False
407
408
def get_hf_config_sliding_window(self) -> Optional[int]:
409
"""Get the sliding window size, or None if disabled."""
vllm/model_executor/models/__init__.py
+125
-22
@@ -1,12 +1,18 @@
1
-import functools
1
import importlib
3
-from typing import Dict, List, Optional, Tuple, Type
2
+import string
3
+import subprocess
4
+import sys
5
+import uuid
6
+from functools import lru_cache, partial
7
+from typing import Callable, Dict, List, Optional, Tuple, Type, Union
8
9
import torch.nn as nn
10
11
from vllm.logger import init_logger
12
from vllm.utils import is_hip
13
14
+from .interfaces import supports_multimodal, supports_pp
15
+
16
logger = init_logger(__name__)
17
18
_GENERATION_MODELS = {
@@ -152,19 +158,25 @@ _ROCM_PARTIALLY_SUPPORTED_MODELS: Dict[str, str] = {
158
class ModelRegistry:
159
160
@staticmethod
155
- @functools.lru_cache(maxsize=128)
156
- def _get_model(model_arch: str):
157
- module_name, model_cls_name = _MODELS[model_arch]
158
- module = importlib.import_module(
159
- f"vllm.model_executor.models.{module_name}")
160
- return getattr(module, model_cls_name, None)
161
+ def _get_module_cls_name(model_arch: str) -> Tuple[str, str]:
162
+ module_relname, cls_name = _MODELS[model_arch]
163
+ return f"vllm.model_executor.models.{module_relname}", cls_name
164
165
@staticmethod
163
- def _try_load_model_cls(model_arch: str) -> Optional[Type[nn.Module]]:
164
- if model_arch in _OOT_MODELS:
165
- return _OOT_MODELS[model_arch]
166
+ @lru_cache(maxsize=128)
167
+ def _try_get_model_stateful(model_arch: str) -> Optional[Type[nn.Module]]:
168
if model_arch not in _MODELS:
169
return None
170
+
171
+ module_name, cls_name = ModelRegistry._get_module_cls_name(model_arch)
172
+ module = importlib.import_module(module_name)
173
+ return getattr(module, cls_name, None)
174
+
175
+ @staticmethod
176
+ def _try_get_model_stateless(model_arch: str) -> Optional[Type[nn.Module]]:
177
+ if model_arch in _OOT_MODELS:
178
+ return _OOT_MODELS[model_arch]
179
+
180
if is_hip():
181
if model_arch in _ROCM_UNSUPPORTED_MODELS:
182
raise ValueError(
@@ -175,11 +187,24 @@ class ModelRegistry:
187
"Model architecture %s is partially supported by ROCm: %s",
188
model_arch, _ROCM_PARTIALLY_SUPPORTED_MODELS[model_arch])
189
178
- return ModelRegistry._get_model(model_arch)
190
+ return None
191
+
192
+ @staticmethod
193
+ def _try_load_model_cls(model_arch: str) -> Optional[Type[nn.Module]]:
194
+ model = ModelRegistry._try_get_model_stateless(model_arch)
195
+ if model is not None:
196
+ return model
197
+
198
+ return ModelRegistry._try_get_model_stateful(model_arch)
199
200
@staticmethod
201
def resolve_model_cls(
182
- architectures: List[str]) -> Tuple[Type[nn.Module], str]:
202
+ architectures: Union[str, List[str]], ) -> Tuple[Type[nn.Module], str]:
203
+ if isinstance(architectures, str):
204
+ architectures = [architectures]
205
+ if not architectures:
206
+ logger.warning("No model architectures are specified")
207
+
208
for arch in architectures:
209
model_cls = ModelRegistry._try_load_model_cls(arch)
210
if model_cls is not None:
@@ -200,21 +225,99 @@ class ModelRegistry:
225
"Model architecture %s is already registered, and will be "
226
"overwritten by the new model class %s.", model_arch,
227
model_cls.__name__)
203
- global _OOT_MODELS
228
+
229
_OOT_MODELS[model_arch] = model_cls
230
231
@staticmethod
207
- def is_embedding_model(model_arch: str) -> bool:
208
- return model_arch in _EMBEDDING_MODELS
232
+ @lru_cache(maxsize=128)
233
+ def _check_stateless(
234
+ func: Callable[[Type[nn.Module]], bool],
235
+ model_arch: str,
236
+ *,
237
+ default: Optional[bool] = None,
238
+ ) -> bool:
239
+ """
240
+ Run a boolean function against a model and return the result.
241
+
242
+ If the model is not found, returns the provided default value.
243
+
244
+ If the model is not already imported, the function is run inside a
245
+ subprocess to avoid initializing CUDA for the main program.
246
+ """
247
+ model = ModelRegistry._try_get_model_stateless(model_arch)
248
+ if model is not None:
249
+ return func(model)
250
+
251
+ if model_arch not in _MODELS and default is not None:
252
+ return default
253
+
254
+ module_name, cls_name = ModelRegistry._get_module_cls_name(model_arch)
255
+
256
+ valid_name_characters = string.ascii_letters + string.digits + "._"
257
+ if any(s not in valid_name_characters for s in module_name):
258
+ raise ValueError(f"Unsafe module name detected for {model_arch}")
259
+ if any(s not in valid_name_characters for s in cls_name):
260
+ raise ValueError(f"Unsafe class name detected for {model_arch}")
261
+ if any(s not in valid_name_characters for s in func.__module__):
262
+ raise ValueError(f"Unsafe module name detected for {func}")
263
+ if any(s not in valid_name_characters for s in func.__name__):
264
+ raise ValueError(f"Unsafe class name detected for {func}")
265
+
266
+ err_id = uuid.uuid4()
267
+
268
+ stmts = ";".join([
269
+ f"from {module_name} import {cls_name}",
270
+ f"from {func.__module__} import {func.__name__}",
271
+ f"assert {func.__name__}({cls_name}), '{err_id}'",
272
+ ])
273
+
274
+ result = subprocess.run([sys.executable, "-c", stmts],
275
+ capture_output=True)
276
+
277
+ if result.returncode != 0:
278
+ err_lines = [line.decode() for line in result.stderr.splitlines()]
279
+ if err_lines and err_lines[-1] != f"AssertionError: {err_id}":
280
+ err_str = "\n".join(err_lines)
281
+ raise RuntimeError(
282
+ "An unexpected error occurred while importing the model in "
283
+ f"another process. Error log:\n{err_str}")
284
+
285
+ return result.returncode == 0
286
+
287
+ @staticmethod
288
+ def is_embedding_model(architectures: Union[str, List[str]]) -> bool:
289
+ if isinstance(architectures, str):
290
+ architectures = [architectures]
291
+ if not architectures:
292
+ logger.warning("No model architectures are specified")
293
+
294
+ return any(arch in _EMBEDDING_MODELS for arch in architectures)
295
+
296
+ @staticmethod
297
+ def is_multimodal_model(architectures: Union[str, List[str]]) -> bool:
298
+ if isinstance(architectures, str):
299
+ architectures = [architectures]
300
+ if not architectures:
301
+ logger.warning("No model architectures are specified")
302
+
303
+ is_mm = partial(ModelRegistry._check_stateless,
304
+ supports_multimodal,
305
+ default=False)
306
+
307
+ return any(is_mm(arch) for arch in architectures)
308
309
@staticmethod
211
- def is_multimodal_model(model_arch: str) -> bool:
310
+ def is_pp_supported_model(architectures: Union[str, List[str]]) -> bool:
311
+ if isinstance(architectures, str):
312
+ architectures = [architectures]
313
+ if not architectures:
314
+ logger.warning("No model architectures are specified")
315
+
316
+ is_pp = partial(ModelRegistry._check_stateless,
317
+ supports_pp,
318
+ default=False)
319
213
- # TODO: find a way to avoid initializing CUDA prematurely to
214
- # use `supports_multimodal` to determine if a model is multimodal
215
- # model_cls = ModelRegistry._try_load_model_cls(model_arch)
216
- # from vllm.model_executor.models.interfaces import supports_multimodal
217
- return model_arch in _MULTIMODAL_MODELS
320
+ return any(is_pp(arch) for arch in architectures)
321
322
323
__all__ = [
vllm/model_executor/models/arctic.py
+41
-18
@@ -1,12 +1,12 @@
1
"""Inference-only Snowflake Arctic model."""
2
-from typing import Iterable, List, Optional, Tuple
2
+from typing import Iterable, List, Optional, Tuple, Union
3
4
import torch
5
from torch import nn
6
7
from vllm.attention import Attention, AttentionMetadata
8
from vllm.config import CacheConfig
9
-from vllm.distributed import (get_tensor_model_parallel_rank,
9
+from vllm.distributed import (get_pp_group, get_tensor_model_parallel_rank,
10
get_tensor_model_parallel_world_size,
11
tensor_model_parallel_all_reduce)
12
from vllm.logger import init_logger
@@ -18,8 +18,7 @@ from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
18
ReplicatedLinear,
19
RowParallelLinear)
20
from vllm.model_executor.layers.logits_processor import LogitsProcessor
21
-from vllm.model_executor.layers.quantization.base_config import (
22
- QuantizationConfig)
21
+from vllm.model_executor.layers.quantization import QuantizationConfig
22
from vllm.model_executor.layers.quantization.deepspeedfp import (
23
DeepSpeedFPConfig, DeepSpeedFPParameter)
24
from vllm.model_executor.layers.rotary_embedding import get_rope
@@ -32,6 +31,10 @@ from vllm.model_executor.utils import set_weight_attrs
31
from vllm.sequence import IntermediateTensors
32
from vllm.transformers_utils.configs.arctic import ArcticConfig
33
34
+from .interfaces import SupportsPP
35
+from .utils import (is_pp_missing_parameter,
36
+ make_empty_intermediate_tensors_factory, make_layers)
37
+
38
logger = init_logger(__name__)
39
40
@@ -364,6 +367,7 @@ class ArcticModel(nn.Module):
367
config: ArcticConfig,
368
cache_config: Optional[CacheConfig] = None,
369
quant_config: Optional[QuantizationConfig] = None,
370
+ prefix: str = "",
371
) -> None:
372
super().__init__()
373
self.padding_idx = config.pad_token_id
@@ -372,15 +376,16 @@ class ArcticModel(nn.Module):
376
self.vocab_size,
377
config.hidden_size,
378
org_num_embeddings=self.vocab_size)
375
- self.layers = nn.ModuleList([
376
- ArcticDecoderLayer(config,
377
- layer_idx,
378
- cache_config,
379
- quant_config=quant_config)
380
- for layer_idx in range(config.num_hidden_layers)
381
- ])
379
+ self.start_layer, self.end_layer, self.layers = make_layers(
380
+ config.num_hidden_layers,
381
+ lambda prefix: ArcticDecoderLayer(config, int(
382
+ prefix.split(".")[-1]), cache_config, quant_config),
383
+ prefix=f"{prefix}.layers")
384
self._attn_implementation = config._attn_implementation
385
self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
386
+ self.make_empty_intermediate_tensors = (
387
+ make_empty_intermediate_tensors_factory(["hidden_states"],
388
+ config.hidden_size))
389
390
def forward(
391
self,
@@ -388,17 +393,25 @@ class ArcticModel(nn.Module):
393
positions: torch.Tensor,
394
kv_caches: List[torch.Tensor],
395
attn_metadata: AttentionMetadata,
391
- ) -> torch.Tensor:
392
- hidden_states = self.embed_tokens(input_ids)
393
- for i in range(len(self.layers)):
396
+ intermediate_tensors: Optional[IntermediateTensors],
397
+ ) -> Union[torch.Tensor, IntermediateTensors]:
398
+ if get_pp_group().is_first_rank:
399
+ hidden_states = self.embed_tokens(input_ids)
400
+ else:
401
+ assert intermediate_tensors is not None
402
+ hidden_states = intermediate_tensors["hidden_states"]
403
+ for i in range(self.start_layer, self.end_layer):
404
layer = self.layers[i]
395
- hidden_states = layer(positions, hidden_states, kv_caches[i],
405
+ hidden_states = layer(positions, hidden_states,
406
+ kv_caches[i - self.start_layer],
407
attn_metadata)
408
+ if not get_pp_group().is_last_rank:
409
+ return IntermediateTensors({"hidden_states": hidden_states})
410
hidden_states = self.norm(hidden_states)
411
return hidden_states
412
413
401
-class ArcticForCausalLM(nn.Module):
414
+class ArcticForCausalLM(nn.Module, SupportsPP):
415
416
def __init__(self,
417
config: ArcticConfig,
@@ -422,6 +435,8 @@ class ArcticForCausalLM(nn.Module):
435
self.logits_processor = LogitsProcessor(self.unpadded_vocab_size,
436
config.vocab_size)
437
self.sampler = Sampler()
438
+ self.make_empty_intermediate_tensors = (
439
+ self.model.make_empty_intermediate_tensors)
440
441
def forward(
442
self,
@@ -430,9 +445,9 @@ class ArcticForCausalLM(nn.Module):
445
kv_caches: List[torch.Tensor],
446
attn_metadata: AttentionMetadata,
447
intermediate_tensors: Optional[IntermediateTensors] = None,
433
- ) -> torch.Tensor:
448
+ ) -> Union[torch.Tensor, IntermediateTensors]:
449
hidden_states = self.model(input_ids, positions, kv_caches,
435
- attn_metadata)
450
+ attn_metadata, intermediate_tensors)
451
return hidden_states
452
453
def compute_logits(
@@ -503,6 +518,8 @@ class ArcticForCausalLM(nn.Module):
518
# Skip loading extra bias for GPTQ models.
519
if name.endswith(".bias") and name not in params_dict:
520
continue
521
+ if is_pp_missing_parameter(name, self):
522
+ continue
523
param = params_dict[name]
524
weight_loader = param.weight_loader
525
weight_loader(param, loaded_weight, shard_id)
@@ -512,6 +529,8 @@ class ArcticForCausalLM(nn.Module):
529
if weight_name not in name:
530
continue
531
name = name.replace(weight_name, param_name)
532
+ if is_pp_missing_parameter(name, self):
533
+ continue
534
param = params_dict[name]
535
weight_loader = param.weight_loader
536
weight_loader(param, loaded_weight, shard_id)
@@ -522,6 +541,8 @@ class ArcticForCausalLM(nn.Module):
541
if weight_name not in name:
542
continue
543
name = name.replace(weight_name, param_name)
544
+ if is_pp_missing_parameter(name, self):
545
+ continue
546
param = params_dict[name]
547
weight_loader = param.weight_loader
548
weight_loader(param,
@@ -532,6 +553,8 @@ class ArcticForCausalLM(nn.Module):
553
else:
554
if name.endswith(".bias") and name not in params_dict:
555
continue
556
+ if is_pp_missing_parameter(name, self):
557
+ continue
558
param = params_dict[name]
559
560
weight_loader = getattr(param, "weight_loader",
vllm/model_executor/models/baichuan.py
+44
-21
@@ -19,7 +19,7 @@
19
# limitations under the License.
20
"""Inference-only BaiChuan model compatible with HuggingFace weights."""
21
import math
22
-from typing import Iterable, List, Optional, Tuple
22
+from typing import Iterable, List, Optional, Tuple, Union
23
24
import torch
25
from torch import nn
@@ -27,7 +27,7 @@ from transformers import PretrainedConfig
27
28
from vllm.attention import Attention, AttentionMetadata
29
from vllm.config import CacheConfig, LoRAConfig
30
-from vllm.distributed import (get_tensor_model_parallel_rank,
30
+from vllm.distributed import (get_pp_group, get_tensor_model_parallel_rank,
31
get_tensor_model_parallel_world_size)
32
from vllm.model_executor.layers.activation import SiluAndMul
33
from vllm.model_executor.layers.layernorm import RMSNorm
@@ -35,8 +35,7 @@ from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
35
QKVParallelLinear,
36
RowParallelLinear)
37
from vllm.model_executor.layers.logits_processor import LogitsProcessor
38
-from vllm.model_executor.layers.quantization.base_config import (
39
- QuantizationConfig)
38
+from vllm.model_executor.layers.quantization import QuantizationConfig
39
from vllm.model_executor.layers.rotary_embedding import get_rope
40
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
41
from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -45,7 +44,9 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
44
from vllm.model_executor.sampling_metadata import SamplingMetadata
45
from vllm.sequence import IntermediateTensors
46
48
-from .interfaces import SupportsLoRA
47
+from .interfaces import SupportsLoRA, SupportsPP
48
+from .utils import (is_pp_missing_parameter,
49
+ make_empty_intermediate_tensors_factory, make_layers)
50
51
52
def _get_alibi_slopes(total_num_heads: int) -> torch.Tensor:
@@ -255,7 +256,8 @@ class BaiChuanModel(nn.Module):
256
config: PretrainedConfig,
257
position_embedding: str,
258
cache_config: Optional[CacheConfig] = None,
258
- quant_config: Optional[QuantizationConfig] = None):
259
+ quant_config: Optional[QuantizationConfig] = None,
260
+ prefix: str = ""):
261
super().__init__()
262
self.config = config
263
self.padding_idx = config.pad_token_id
@@ -265,12 +267,16 @@ class BaiChuanModel(nn.Module):
267
config.vocab_size,
268
config.hidden_size,
269
)
268
- self.layers = nn.ModuleList([
269
- BaiChuanDecoderLayer(config, position_embedding, cache_config,
270
- quant_config)
271
- for _ in range(config.num_hidden_layers)
272
- ])
270
+ self.start_layer, self.end_layer, self.layers = make_layers(
271
+ config.num_hidden_layers,
272
+ lambda prefix: BaiChuanDecoderLayer(config, position_embedding,
273
+ cache_config, quant_config),
274
+ prefix=f"{prefix}.layers",
275
+ )
276
self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
277
+ self.make_empty_intermediate_tensors = (
278
+ make_empty_intermediate_tensors_factory(
279
+ ["hidden_states", "residual"], config.hidden_size))
280
281
def forward(
282
self,
@@ -278,23 +284,34 @@ class BaiChuanModel(nn.Module):
284
positions: torch.Tensor,
285
kv_caches: List[torch.Tensor],
286
attn_metadata: AttentionMetadata,
281
- ) -> torch.Tensor:
282
- hidden_states = self.embed_tokens(input_ids)
283
- residual = None
284
- for i in range(len(self.layers)):
287
+ intermediate_tensors: Optional[IntermediateTensors],
288
+ ) -> Union[torch.Tensor, IntermediateTensors]:
289
+ if get_pp_group().is_first_rank:
290
+ hidden_states = self.embed_tokens(input_ids)
291
+ residual = None
292
+ else:
293
+ assert intermediate_tensors is not None
294
+ hidden_states = intermediate_tensors["hidden_states"]
295
+ residual = intermediate_tensors["residual"]
296
+ for i in range(self.start_layer, self.end_layer):
297
layer = self.layers[i]
298
hidden_states, residual = layer(
299
positions,
300
hidden_states,
289
- kv_caches[i],
301
+ kv_caches[i - self.start_layer],
302
attn_metadata,
303
residual,
304
)
305
+ if not get_pp_group().is_last_rank:
306
+ return IntermediateTensors({
307
+ "hidden_states": hidden_states,
308
+ "residual": residual,
309
+ })
310
hidden_states, _ = self.norm(hidden_states, residual)
311
return hidden_states
312
313
297
-class BaiChuanBaseForCausalLM(nn.Module, SupportsLoRA):
314
+class BaiChuanBaseForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
315
packed_modules_mapping = {
316
"W_pack": ["W_pack"],
317
"gate_up_proj": [
@@ -335,6 +352,8 @@ class BaiChuanBaseForCausalLM(nn.Module, SupportsLoRA):
352
self.lm_head.weight = self.model.embed_tokens.weight
353
self.logits_processor = LogitsProcessor(config.vocab_size)
354
self.sampler = Sampler()
355
+ self.make_empty_intermediate_tensors = (
356
+ self.model.make_empty_intermediate_tensors)
357
358
def forward(
359
self,
@@ -343,9 +362,9 @@ class BaiChuanBaseForCausalLM(nn.Module, SupportsLoRA):
362
kv_caches: List[torch.Tensor],
363
attn_metadata: AttentionMetadata,
364
intermediate_tensors: Optional[IntermediateTensors] = None,
346
- ) -> torch.Tensor:
365
+ ) -> Union[torch.Tensor, IntermediateTensors]:
366
hidden_states = self.model(input_ids, positions, kv_caches,
348
- attn_metadata)
367
+ attn_metadata, intermediate_tensors)
368
return hidden_states
369
370
def compute_logits(
@@ -394,6 +413,8 @@ class BaiChuanBaseForCausalLM(nn.Module, SupportsLoRA):
413
# Skip loading extra bias for GPTQ models.
414
if name.endswith(".bias") and name not in params_dict:
415
continue
416
+ if is_pp_missing_parameter(name, self):
417
+ continue
418
param = params_dict[name]
419
weight_loader = param.weight_loader
420
weight_loader(param, loaded_weight, shard_id)
@@ -402,6 +423,8 @@ class BaiChuanBaseForCausalLM(nn.Module, SupportsLoRA):
423
# Skip loading extra bias for GPTQ models.
424
if name.endswith(".bias") and name not in params_dict:
425
continue
426
+ if is_pp_missing_parameter(name, self):
427
+ continue
428
param = params_dict[name]
429
weight_loader = getattr(param, "weight_loader",
430
default_weight_loader)
@@ -413,7 +436,7 @@ class BaichuanForCausalLM(BaiChuanBaseForCausalLM):
436
437
def __init__(
438
self,
416
- config,
439
+ config: PretrainedConfig,
440
cache_config: Optional[CacheConfig] = None,
441
quant_config: Optional[QuantizationConfig] = None,
442
lora_config: Optional[LoRAConfig] = None,
@@ -431,7 +454,7 @@ class BaiChuanForCausalLM(BaiChuanBaseForCausalLM):
454
455
def __init__(
456
self,
434
- config,
457
+ config: PretrainedConfig,
458
cache_config: Optional[CacheConfig] = None,
459
quant_config: Optional[QuantizationConfig] = None,
460
lora_config: Optional[LoRAConfig] = None,
vllm/model_executor/models/blip2.py
+39
-22
@@ -1,3 +1,4 @@
1
+from functools import cached_property
2
from typing import (Iterable, List, Literal, Mapping, Optional, Tuple,
3
TypedDict, Union)
4
@@ -11,7 +12,7 @@ from vllm.config import CacheConfig, MultiModalConfig
12
from vllm.inputs import INPUT_REGISTRY, InputContext, LLMInputs
13
from vllm.model_executor.layers.activation import get_act_fn
14
from vllm.model_executor.layers.quantization import QuantizationConfig
14
-from vllm.model_executor.layers.sampler import SamplerOutput
15
+from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
16
from vllm.model_executor.model_loader.weight_utils import default_weight_loader
17
from vllm.model_executor.sampling_metadata import SamplingMetadata
18
from vllm.multimodal import MULTIMODAL_REGISTRY
@@ -19,7 +20,7 @@ from vllm.sequence import IntermediateTensors, SequenceData
20
21
from .blip import (BlipVisionModel, dummy_image_for_blip,
22
get_max_blip_image_tokens)
22
-from .interfaces import SupportsMultiModal
23
+from .interfaces import SupportsMultiModal, SupportsPP
24
from .utils import (group_weights_with_prefix, init_vllm_registered_model,
25
merge_multimodal_embeddings)
26
@@ -475,7 +476,7 @@ def input_processor_for_blip2(ctx: InputContext, llm_inputs: LLMInputs):
476
@MULTIMODAL_REGISTRY.register_max_image_tokens(get_max_blip2_image_tokens)
477
@INPUT_REGISTRY.register_dummy_data(dummy_data_for_blip2)
478
@INPUT_REGISTRY.register_input_processor(input_processor_for_blip2)
478
-class Blip2ForConditionalGeneration(nn.Module, SupportsMultiModal):
479
+class Blip2ForConditionalGeneration(nn.Module, SupportsMultiModal, SupportsPP):
480
481
def __init__(self,
482
config: Blip2Config,
@@ -508,6 +509,16 @@ class Blip2ForConditionalGeneration(nn.Module, SupportsMultiModal):
509
self.language_model = init_vllm_registered_model(
510
config.text_config, cache_config, quant_config)
511
512
+ self.make_empty_intermediate_tensors = (
513
+ self.language_model.make_empty_intermediate_tensors)
514
+
515
+ @cached_property
516
+ def sampler(self):
517
+ if hasattr(self.language_model, "sampler"):
518
+ return self.language_model.sampler
519
+
520
+ return Sampler()
521
+
522
def _validate_pixel_values(self, data: torch.Tensor) -> torch.Tensor:
523
h = w = self.config.vision_config.image_size
524
expected_dims = (3, h, w)
@@ -600,7 +611,7 @@ class Blip2ForConditionalGeneration(nn.Module, SupportsMultiModal):
611
attn_metadata: AttentionMetadata,
612
intermediate_tensors: Optional[IntermediateTensors] = None,
613
**kwargs: object,
603
- ) -> SamplerOutput:
614
+ ) -> Union[SamplerOutput, IntermediateTensors]:
615
"""Run forward pass for BLIP-2.
616
617
One key thing to understand is the `input_ids` already accounts for the
@@ -631,26 +642,32 @@ class Blip2ForConditionalGeneration(nn.Module, SupportsMultiModal):
642
See also:
643
:class:`Blip2ImageInputs`
644
"""
634
- image_input = self._parse_and_validate_image_input(**kwargs)
635
-
636
- if image_input is not None:
637
- vision_embeddings = self._process_image_input(image_input)
638
- inputs_embeds = self.language_model.model.get_input_embeddings(
639
- input_ids)
640
-
641
- inputs_embeds = merge_multimodal_embeddings(
642
- input_ids, inputs_embeds, vision_embeddings,
643
- BLIP2_IMAGE_TOKEN_ID)
644
-
645
+ if intermediate_tensors is not None:
646
input_ids = None
646
- else:
647
inputs_embeds = None
648
-
649
- hidden_states = self.language_model.model(input_ids,
650
- positions,
651
- kv_caches,
652
- attn_metadata,
653
- inputs_embeds=inputs_embeds)
648
+ else:
649
+ image_input = self._parse_and_validate_image_input(**kwargs)
650
+
651
+ if image_input is not None:
652
+ vision_embeddings = self._process_image_input(image_input)
653
+ inputs_embeds = self.language_model.model.get_input_embeddings(
654
+ input_ids)
655
+
656
+ inputs_embeds = merge_multimodal_embeddings(
657
+ input_ids, inputs_embeds, vision_embeddings,
658
+ BLIP2_IMAGE_TOKEN_ID)
659
+
660
+ input_ids = None
661
+ else:
662
+ inputs_embeds = None
663
+
664
+ hidden_states = self.language_model.model(
665
+ input_ids,
666
+ positions,
667
+ kv_caches,
668
+ attn_metadata,
669
+ intermediate_tensors=intermediate_tensors,
670
+ inputs_embeds=inputs_embeds)
671
672
return hidden_states
673
vllm/model_executor/models/bloom.py
+34
-16
@@ -17,7 +17,7 @@
17
# limitations under the License.
18
"""Inference-only BLOOM model compatible with HuggingFace weights."""
19
import math
20
-from typing import Iterable, List, Optional, Tuple
20
+from typing import Iterable, List, Optional, Tuple, Union
21
22
import torch
23
from torch import nn
@@ -25,15 +25,14 @@ from transformers import BloomConfig
25
26
from vllm.attention import Attention, AttentionMetadata
27
from vllm.config import CacheConfig
28
-from vllm.distributed import (get_tensor_model_parallel_rank,
28
+from vllm.distributed import (get_pp_group, get_tensor_model_parallel_rank,
29
get_tensor_model_parallel_world_size)
30
from vllm.model_executor.layers.activation import get_act_fn
31
from vllm.model_executor.layers.linear import (ColumnParallelLinear,
32
QKVParallelLinear,
33
RowParallelLinear)
34
from vllm.model_executor.layers.logits_processor import LogitsProcessor
35
-from vllm.model_executor.layers.quantization.base_config import (
36
- QuantizationConfig)
35
+from vllm.model_executor.layers.quantization import QuantizationConfig
36
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
37
from vllm.model_executor.layers.vocab_parallel_embedding import (
38
ParallelLMHead, VocabParallelEmbedding)
@@ -41,6 +40,10 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
40
from vllm.model_executor.sampling_metadata import SamplingMetadata
41
from vllm.sequence import IntermediateTensors
42
43
+from .interfaces import SupportsPP
44
+from .utils import (is_pp_missing_parameter,
45
+ make_empty_intermediate_tensors_factory, make_layers)
46
+
47
48
def _get_alibi_slopes(total_num_heads: int) -> torch.Tensor:
49
closest_power_of_2 = 2**math.floor(math.log2(total_num_heads))
@@ -222,6 +225,7 @@ class BloomModel(nn.Module):
225
config: BloomConfig,
226
cache_config: Optional[CacheConfig] = None,
227
quant_config: Optional[QuantizationConfig] = None,
228
+ prefix: str = "",
229
):
230
super().__init__()
231
self.embed_dim = config.hidden_size
@@ -235,13 +239,16 @@ class BloomModel(nn.Module):
239
self.embed_dim, eps=config.layer_norm_epsilon)
240
241
# Transformer blocks
238
- self.h = nn.ModuleList([
239
- BloomBlock(config, cache_config, quant_config)
240
- for _ in range(config.num_hidden_layers)
241
- ])
242
+ self.start_layer, self.end_layer, self.h = make_layers(
243
+ config.num_hidden_layers,
244
+ lambda prefix: BloomBlock(config, cache_config, quant_config),
245
+ prefix=f"{prefix}.h")
246
247
# Final Layer Norm
248
self.ln_f = nn.LayerNorm(self.embed_dim, eps=config.layer_norm_epsilon)
249
+ self.make_empty_intermediate_tensors = (
250
+ make_empty_intermediate_tensors_factory(["hidden_states"],
251
+ config.hidden_size))
252
253
def forward(
254
self,
@@ -249,22 +256,29 @@ class BloomModel(nn.Module):
256
position_ids: torch.Tensor,
257
kv_caches: List[torch.Tensor],
258
attn_metadata: AttentionMetadata,
252
- ) -> torch.Tensor:
253
- hidden_states = self.word_embeddings(input_ids)
254
- hidden_states = self.word_embeddings_layernorm(hidden_states)
255
- for i in range(len(self.h)):
259
+ intermediate_tensors: Optional[IntermediateTensors],
260
+ ) -> Union[torch.Tensor, IntermediateTensors]:
261
+ if get_pp_group().is_first_rank:
262
+ hidden_states = self.word_embeddings(input_ids)
263
+ hidden_states = self.word_embeddings_layernorm(hidden_states)
264
+ else:
265
+ assert intermediate_tensors is not None
266
+ hidden_states = intermediate_tensors["hidden_states"]
267
+ for i in range(self.start_layer, self.end_layer):
268
layer = self.h[i]
269
hidden_states = layer(
270
position_ids,
271
hidden_states,
260
- kv_caches[i],
272
+ kv_caches[i - self.start_layer],
273
attn_metadata,
274
)
275
+ if not get_pp_group().is_last_rank:
276
+ return IntermediateTensors({"hidden_states": hidden_states})
277
hidden_states = self.ln_f(hidden_states)
278
return hidden_states
279
280
267
-class BloomForCausalLM(nn.Module):
281
+class BloomForCausalLM(nn.Module, SupportsPP):
282
283
def __init__(
284
self,
@@ -284,6 +298,8 @@ class BloomForCausalLM(nn.Module):
298
299
self.logits_processor = LogitsProcessor(config.vocab_size)
300
self.sampler = Sampler()
301
+ self.make_empty_intermediate_tensors = (
302
+ self.transformer.make_empty_intermediate_tensors)
303
304
def forward(
305
self,
@@ -292,9 +308,9 @@ class BloomForCausalLM(nn.Module):
308
kv_caches: List[torch.Tensor],
309
attn_metadata: AttentionMetadata,
310
intermediate_tensors: Optional[IntermediateTensors] = None,
295
- ) -> torch.Tensor:
311
+ ) -> Union[torch.Tensor, IntermediateTensors]:
312
hidden_states = self.transformer(input_ids, positions, kv_caches,
297
- attn_metadata)
313
+ attn_metadata, intermediate_tensors)
314
return hidden_states
315
316
def compute_logits(
@@ -321,6 +337,8 @@ class BloomForCausalLM(nn.Module):
337
continue
338
if not name.startswith("transformer."):
339
name = "transformer." + name
340
+ if is_pp_missing_parameter(name, self):
341
+ continue
342
param = params_dict[name]
343
344
if "query_key_value" in name:
vllm/model_executor/models/chameleon.py
+63
-30
@@ -1,6 +1,6 @@
1
from functools import cached_property
2
from typing import (Any, Dict, Iterable, List, Literal, Mapping, Optional,
3
- Tuple, TypedDict)
3
+ Tuple, TypedDict, Union)
4
5
import torch
6
import torch.nn.functional as F
@@ -10,7 +10,7 @@ from transformers import ChameleonConfig, ChameleonVQVAEConfig
10
11
from vllm.attention import Attention, AttentionMetadata
12
from vllm.config import CacheConfig, MultiModalConfig
13
-from vllm.distributed import get_tensor_model_parallel_world_size
13
+from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
14
from vllm.inputs import INPUT_REGISTRY, InputContext, LLMInputs
15
from vllm.model_executor.layers.activation import SiluAndMul
16
from vllm.model_executor.layers.layernorm import RMSNorm
@@ -33,7 +33,9 @@ from vllm.multimodal.utils import (cached_get_tokenizer,
33
from vllm.sequence import IntermediateTensors, SequenceData
34
from vllm.utils import print_warning_once
35
36
-from .interfaces import SupportsMultiModal
36
+from .interfaces import SupportsMultiModal, SupportsPP
37
+from .utils import (is_pp_missing_parameter,
38
+ make_empty_intermediate_tensors_factory, make_layers)
39
40
# These configs are not part of the model config but the preprocessor
41
# and processor files, so we hardcode them in the model file for now.
@@ -822,6 +824,7 @@ class ChameleonModel(nn.Module):
824
config: ChameleonConfig,
825
cache_config: Optional[CacheConfig] = None,
826
quant_config: Optional[QuantizationConfig] = None,
827
+ prefix: str = "",
828
) -> None:
829
super().__init__()
830
self.config = config
@@ -835,14 +838,20 @@ class ChameleonModel(nn.Module):
838
config.vocabulary_map)
839
decoder_layer = ChameleonDecoderLayer if not self.config.swin_norm \
840
else ChameleonSwinDecoderLayer
838
- self.layers = nn.ModuleList([
839
- decoder_layer(config=config,
840
- cache_config=cache_config,
841
- quant_config=quant_config)
842
- for _ in range(config.num_hidden_layers)
843
- ])
841
+
842
+ self.start_layer, self.end_layer, self.layers = make_layers(
843
+ config.num_hidden_layers,
844
+ lambda prefix: decoder_layer(config=config,
845
+ cache_config=cache_config,
846
+ quant_config=quant_config),
847
+ prefix=f"{prefix}.layers",
848
+ )
849
+
850
self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
851
self.vqmodel = ChameleonVQVAE(config.vq_config)
852
+ self.make_empty_intermediate_tensors = (
853
+ make_empty_intermediate_tensors_factory(
854
+ ["hidden_states", "residual"], config.hidden_size))
855
856
def get_input_embeddings(self, input_ids: torch.Tensor) -> torch.Tensor:
857
return self.embed_tokens(input_ids)
@@ -865,22 +874,33 @@ class ChameleonModel(nn.Module):
874
positions: torch.Tensor,
875
kv_caches: List[torch.Tensor],
876
attn_metadata: AttentionMetadata,
877
+ intermediate_tensors: Optional[IntermediateTensors],
878
inputs_embeds: Optional[torch.Tensor] = None,
869
- ) -> torch.Tensor:
870
- if inputs_embeds is not None:
871
- hidden_states = inputs_embeds
879
+ ) -> Union[torch.Tensor, IntermediateTensors]:
880
+ if get_pp_group().is_first_rank:
881
+ if inputs_embeds is not None:
882
+ hidden_states = inputs_embeds
883
+ else:
884
+ hidden_states = self.get_input_embeddings(input_ids)
885
+ residual = None
886
else:
873
- hidden_states = self.get_input_embeddings(input_ids)
874
- residual = None
875
- for i in range(len(self.layers)):
887
+ assert intermediate_tensors is not None
888
+ hidden_states = intermediate_tensors["hidden_states"]
889
+ residual = intermediate_tensors["residual"]
890
+ for i in range(self.start_layer, self.end_layer):
891
layer = self.layers[i]
892
hidden_states, residual = layer(
893
positions,
894
hidden_states,
880
- kv_caches[i],
895
+ kv_caches[i - self.start_layer],
896
attn_metadata,
897
residual,
898
)
899
+ if not get_pp_group().is_last_rank:
900
+ return IntermediateTensors({
901
+ "hidden_states": hidden_states,
902
+ "residual": residual
903
+ })
904
hidden_states, _ = self.norm(hidden_states, residual)
905
return hidden_states
906
@@ -889,7 +909,8 @@ class ChameleonModel(nn.Module):
909
@MULTIMODAL_REGISTRY.register_max_image_tokens(get_max_chameleon_image_tokens)
910
@INPUT_REGISTRY.register_dummy_data(dummy_data_for_chameleon)
911
@INPUT_REGISTRY.register_input_processor(input_processor_for_chameleon)
892
-class ChameleonForConditionalGeneration(nn.Module, SupportsMultiModal):
912
+class ChameleonForConditionalGeneration(nn.Module, SupportsMultiModal,
913
+ SupportsPP):
914
915
def __init__(
916
self,
@@ -914,6 +935,8 @@ class ChameleonForConditionalGeneration(nn.Module, SupportsMultiModal):
935
self.logits_processor = LogitsProcessor(self.unpadded_vocab_size,
936
config.vocab_size, logit_scale)
937
self.sampler = Sampler()
938
+ self.make_empty_intermediate_tensors = (
939
+ self.model.make_empty_intermediate_tensors)
940
941
def _validate_pixel_values(self, data: torch.Tensor) -> torch.Tensor:
942
@@ -956,22 +979,26 @@ class ChameleonForConditionalGeneration(nn.Module, SupportsMultiModal):
979
attn_metadata: AttentionMetadata,
980
intermediate_tensors: Optional[IntermediateTensors] = None,
981
**kwargs,
959
- ) -> torch.Tensor:
960
-
961
- image_input = self._parse_and_validate_image_input(**kwargs)
982
+ ) -> Union[torch.Tensor, IntermediateTensors]:
983
963
- if image_input is not None:
964
- assert self.model.vqmodel is not None
965
- image_tokens = self.model.get_image_tokens(image_input["data"].to(
966
- self.config.torch_dtype))
967
- image_token_id = self.model.vocabulary_mapping.image_token_id
968
- special_image_mask = input_ids == image_token_id
969
- image_tokens = image_tokens.to(input_ids.device, input_ids.dtype)
970
- input_ids = input_ids.masked_scatter(special_image_mask,
971
- image_tokens)
984
+ if intermediate_tensors is not None:
985
+ input_ids = None
986
+ else:
987
+ image_input = self._parse_and_validate_image_input(**kwargs)
988
+
989
+ if image_input is not None:
990
+ assert self.model.vqmodel is not None
991
+ image_tokens = self.model.get_image_tokens(
992
+ image_input["data"].to(self.config.torch_dtype))
993
+ image_token_id = self.model.vocabulary_mapping.image_token_id
994
+ special_image_mask = input_ids == image_token_id
995
+ image_tokens = image_tokens.to(input_ids.device,
996
+ input_ids.dtype)
997
+ input_ids = input_ids.masked_scatter(special_image_mask,
998
+ image_tokens)
999
1000
hidden_states = self.model(input_ids, positions, kv_caches,
974
- attn_metadata)
1001
+ attn_metadata, intermediate_tensors)
1002
return hidden_states
1003
1004
def compute_logits(
@@ -1039,6 +1066,8 @@ class ChameleonForConditionalGeneration(nn.Module, SupportsMultiModal):
1066
# Skip loading extra bias for GPTQ models.
1067
if name.endswith(".bias") and name not in params_dict:
1068
continue
1069
+ if is_pp_missing_parameter(name, self):
1070
+ continue
1071
param = params_dict[name]
1072
weight_loader = param.weight_loader
1073
weight_loader(param, loaded_weight, shard_id)
@@ -1060,11 +1089,15 @@ class ChameleonForConditionalGeneration(nn.Module, SupportsMultiModal):
1089
continue
1090
else:
1091
name = remapped_kv_scale_name
1092
+ if is_pp_missing_parameter(name, self):
1093
+ continue
1094
param = params_dict[name]
1095
weight_loader = getattr(param, "weight_loader",
1096
default_weight_loader)
1097
weight_loader(param, loaded_weight)
1098
if use_default_weight_loading and name in params_dict:
1099
+ if is_pp_missing_parameter(name, self):
1100
+ continue
1101
param = params_dict[name]
1102
weight_loader = getattr(param, "weight_loader",
1103
default_weight_loader)
vllm/model_executor/models/chatglm.py
+39
-22
@@ -2,7 +2,7 @@
2
# Adapted from
3
# https://github.com/THUDM/ChatGLM2-6B
4
"""Inference-only ChatGLM model compatible with THUDM weights."""
5
-from typing import Iterable, List, Optional, Tuple
5
+from typing import Iterable, List, Optional, Tuple, Union
6
7
import torch
8
from torch import nn
@@ -10,15 +10,14 @@ from torch.nn import LayerNorm
10
11
from vllm.attention import Attention, AttentionMetadata
12
from vllm.config import CacheConfig, LoRAConfig
13
-from vllm.distributed import get_tensor_model_parallel_world_size
13
+from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
14
from vllm.model_executor.layers.activation import SiluAndMul
15
from vllm.model_executor.layers.layernorm import RMSNorm
16
from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
17
QKVParallelLinear,
18
RowParallelLinear)
19
from vllm.model_executor.layers.logits_processor import LogitsProcessor
20
-from vllm.model_executor.layers.quantization.base_config import (
21
- QuantizationConfig)
20
+from vllm.model_executor.layers.quantization import QuantizationConfig
21
from vllm.model_executor.layers.rotary_embedding import get_rope
22
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
23
from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -28,14 +27,16 @@ from vllm.model_executor.sampling_metadata import SamplingMetadata
27
from vllm.sequence import IntermediateTensors
28
from vllm.transformers_utils.configs import ChatGLMConfig
29
31
-from .interfaces import SupportsLoRA
30
+from .interfaces import SupportsLoRA, SupportsPP
31
+from .utils import (is_pp_missing_parameter,
32
+ make_empty_intermediate_tensors_factory, make_layers)
33
34
35
class GLMAttention(nn.Module):
36
37
def __init__(
38
self,
38
- config,
39
+ config: ChatGLMConfig,
40
cache_config: Optional[CacheConfig] = None,
41
quant_config: Optional[QuantizationConfig] = None,
42
):
@@ -126,7 +127,7 @@ class GLMMLP(nn.Module):
127
128
def __init__(
129
self,
129
- config,
130
+ config: ChatGLMConfig,
131
quant_config: Optional[QuantizationConfig] = None,
132
):
133
super().__init__()
@@ -169,7 +170,7 @@ class GLMBlock(nn.Module):
170
171
def __init__(
172
self,
172
- config,
173
+ config: ChatGLMConfig,
174
cache_config: Optional[CacheConfig] = None,
175
quant_config: Optional[QuantizationConfig] = None,
176
):
@@ -240,9 +241,10 @@ class GLMTransformer(nn.Module):
241
242
def __init__(
243
self,
243
- config,
244
+ config: ChatGLMConfig,
245
cache_config: Optional[CacheConfig] = None,
246
quant_config: Optional[QuantizationConfig] = None,
247
+ prefix: str = "",
248
):
249
super().__init__()
250
self.post_layer_norm = config.post_layer_norm
@@ -251,10 +253,11 @@ class GLMTransformer(nn.Module):
253
self.num_layers = config.num_layers
254
255
# Transformer layers.
254
- self.layers = nn.ModuleList([
255
- GLMBlock(config, cache_config, quant_config)
256
- for i in range(self.num_layers)
257
- ])
256
+ self.start_layer, self.end_layer, self.layers = make_layers(
257
+ self.num_layers,
258
+ lambda prefix: GLMBlock(config, cache_config, quant_config),
259
+ prefix=f"{prefix}.layers",
260
+ )
261
262
if self.post_layer_norm:
263
layer_norm_func = RMSNorm if config.rmsnorm else LayerNorm
@@ -269,16 +272,16 @@ class GLMTransformer(nn.Module):
272
kv_caches: List[torch.Tensor],
273
attn_metadata: AttentionMetadata,
274
) -> torch.Tensor:
272
- for i in range(self.num_layers):
275
+ for i in range(self.start_layer, self.end_layer):
276
layer = self.layers[i]
277
hidden_states = layer(
278
hidden_states=hidden_states,
279
position_ids=position_ids,
277
- kv_cache=kv_caches[i],
280
+ kv_cache=kv_caches[i - self.start_layer],
281
attn_metadata=attn_metadata,
282
)
283
# Final layer norm.
281
- if self.post_layer_norm:
284
+ if get_pp_group().is_last_rank and self.post_layer_norm:
285
hidden_states = self.final_layernorm(hidden_states)
286
287
return hidden_states
@@ -288,7 +291,7 @@ class ChatGLMModel(nn.Module):
291
292
def __init__(
293
self,
291
- config,
294
+ config: ChatGLMConfig,
295
cache_config: Optional[CacheConfig] = None,
296
quant_config: Optional[QuantizationConfig] = None,
297
):
@@ -305,6 +308,9 @@ class ChatGLMModel(nn.Module):
308
self.output_layer = ParallelLMHead(config.padded_vocab_size,
309
config.hidden_size,
310
quant_config=quant_config)
311
+ self.make_empty_intermediate_tensors = (
312
+ make_empty_intermediate_tensors_factory(["hidden_states"],
313
+ config.hidden_size))
314
315
def forward(
316
self,
@@ -312,8 +318,12 @@ class ChatGLMModel(nn.Module):
318
position_ids: torch.Tensor,
319
kv_caches: List[torch.Tensor],
320
attn_metadata: AttentionMetadata,
315
- ) -> torch.Tensor:
316
- inputs_embeds = self.embedding(input_ids)
321
+ intermediate_tensors: Optional[IntermediateTensors],
322
+ ) -> Union[torch.Tensor, IntermediateTensors]:
323
+ if get_pp_group().is_first_rank:
324
+ inputs_embeds = self.embedding(input_ids)
325
+ else:
326
+ inputs_embeds = intermediate_tensors["hidden_states"]
327
328
# Run encoder.
329
hidden_states = self.encoder(
@@ -322,10 +332,13 @@ class ChatGLMModel(nn.Module):
332
kv_caches=kv_caches,
333
attn_metadata=attn_metadata,
334
)
335
+
336
+ if not get_pp_group().is_last_rank:
337
+ return IntermediateTensors({"hidden_states": hidden_states})
338
return hidden_states
339
340
328
-class ChatGLMForCausalLM(nn.Module, SupportsLoRA):
341
+class ChatGLMForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
342
packed_modules_mapping = {
343
"query_key_value": ["query_key_value"],
344
"dense_h_to_4h": ["dense_h_to_4h"]
@@ -362,6 +375,8 @@ class ChatGLMForCausalLM(nn.Module, SupportsLoRA):
375
self.lm_head = self.transformer.output_layer
376
self.logits_processor = LogitsProcessor(config.padded_vocab_size)
377
self.sampler = Sampler()
378
+ self.make_empty_intermediate_tensors = (
379
+ self.transformer.make_empty_intermediate_tensors)
380
381
def forward(
382
self,
@@ -370,9 +385,9 @@ class ChatGLMForCausalLM(nn.Module, SupportsLoRA):
385
kv_caches: List[torch.Tensor],
386
attn_metadata: AttentionMetadata,
387
intermediate_tensors: Optional[IntermediateTensors] = None,
373
- ) -> torch.Tensor:
388
+ ) -> Union[torch.Tensor, IntermediateTensors]:
389
hidden_states = self.transformer(input_ids, positions, kv_caches,
375
- attn_metadata)
390
+ attn_metadata, intermediate_tensors)
391
return hidden_states
392
393
def compute_logits(
@@ -402,6 +417,8 @@ class ChatGLMForCausalLM(nn.Module, SupportsLoRA):
417
# Skip loading extra bias for GPTQ models.
418
if name.endswith(".bias") and name not in params_dict:
419
continue
420
+ if is_pp_missing_parameter(name, self):
421
+ continue
422
param = params_dict[name]
423
weight_loader = getattr(param, "weight_loader",
424
default_weight_loader)
vllm/model_executor/models/commandr.py
+41
-18
@@ -20,7 +20,7 @@
20
21
# This file is based on the LLama model definition file in transformers
22
"""PyTorch Cohere model."""
23
-from typing import Iterable, List, Optional, Set, Tuple
23
+from typing import Iterable, List, Optional, Set, Tuple, Union
24
25
import torch
26
import torch.utils.checkpoint
@@ -29,14 +29,13 @@ from transformers import CohereConfig
29
30
from vllm.attention import Attention, AttentionMetadata
31
from vllm.config import CacheConfig, LoRAConfig
32
-from vllm.distributed import get_tensor_model_parallel_world_size
32
+from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
33
from vllm.model_executor.layers.activation import SiluAndMul
34
from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
35
QKVParallelLinear,
36
RowParallelLinear)
37
from vllm.model_executor.layers.logits_processor import LogitsProcessor
38
-from vllm.model_executor.layers.quantization.base_config import (
39
- QuantizationConfig)
38
+from vllm.model_executor.layers.quantization import QuantizationConfig
39
from vllm.model_executor.layers.rotary_embedding import get_rope
40
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
41
from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -47,7 +46,9 @@ from vllm.model_executor.sampling_metadata import SamplingMetadata
46
from vllm.model_executor.utils import set_weight_attrs
47
from vllm.sequence import IntermediateTensors
48
50
-from .interfaces import SupportsLoRA
49
+from .interfaces import SupportsLoRA, SupportsPP
50
+from .utils import (is_pp_missing_parameter,
51
+ make_empty_intermediate_tensors_factory, make_layers)
52
53
54
@torch.compile
@@ -82,7 +83,7 @@ class CohereMLP(nn.Module):
83
84
def __init__(
85
self,
85
- config,
86
+ config: CohereConfig,
87
quant_config: Optional[QuantizationConfig] = None,
88
):
89
super().__init__()
@@ -256,6 +257,7 @@ class CohereModel(nn.Module):
257
cache_config: Optional[CacheConfig] = None,
258
quant_config: Optional[QuantizationConfig] = None,
259
lora_config: Optional[LoRAConfig] = None,
260
+ prefix: str = "",
261
):
262
super().__init__()
263
self.config = config
@@ -265,12 +267,16 @@ class CohereModel(nn.Module):
267
self.org_vocab_size = config.vocab_size
268
self.embed_tokens = VocabParallelEmbedding(config.vocab_size,
269
config.hidden_size)
268
- self.layers = nn.ModuleList([
269
- CohereDecoderLayer(config, cache_config, quant_config=quant_config)
270
- for _ in range(config.num_hidden_layers)
271
- ])
270
+ self.start_layer, self.end_layer, self.layers = make_layers(
271
+ config.num_hidden_layers,
272
+ lambda prefix: CohereDecoderLayer(config, cache_config,
273
+ quant_config),
274
+ prefix=f"{prefix}.layers")
275
self.norm = LayerNorm(param_shape=(config.hidden_size),
276
eps=config.layer_norm_eps)
277
+ self.make_empty_intermediate_tensors = (
278
+ make_empty_intermediate_tensors_factory(
279
+ ["hidden_states", "residual"], config.hidden_size))
280
281
def forward(
282
self,
@@ -278,23 +284,34 @@ class CohereModel(nn.Module):
284
positions: torch.Tensor,
285
kv_caches: List[torch.Tensor],
286
attn_metadata: AttentionMetadata,
281
- ) -> torch.Tensor:
282
- hidden_states = self.embed_tokens(input_ids)
283
- residual = None
284
- for i in range(len(self.layers)):
287
+ intermediate_tensors: Optional[IntermediateTensors],
288
+ ) -> Union[torch.Tensor, IntermediateTensors]:
289
+ if get_pp_group().is_first_rank:
290
+ hidden_states = self.embed_tokens(input_ids)
291
+ residual = None
292
+ else:
293
+ assert intermediate_tensors is not None
294
+ hidden_states = intermediate_tensors["hidden_states"]
295
+ residual = intermediate_tensors["residual"]
296
+ for i in range(self.start_layer, self.end_layer):
297
layer = self.layers[i]
298
hidden_states, residual = layer(
299
positions,
300
hidden_states,
289
- kv_caches[i],
301
+ kv_caches[i - self.start_layer],
302
attn_metadata,
303
residual,
304
)
305
+ if not get_pp_group().is_last_rank:
306
+ return IntermediateTensors({
307
+ "hidden_states": hidden_states,
308
+ "residual": residual
309
+ })
310
hidden_states, _ = self.norm(hidden_states, residual)
311
return hidden_states
312
313
297
-class CohereForCausalLM(nn.Module, SupportsLoRA):
314
+class CohereForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
315
packed_modules_mapping = {
316
"qkv_proj": [
317
"q_proj",
@@ -337,6 +354,8 @@ class CohereForCausalLM(nn.Module, SupportsLoRA):
354
quant_config,
355
lora_config=lora_config)
356
self.sampler = Sampler()
357
+ self.make_empty_intermediate_tensors = (
358
+ self.model.make_empty_intermediate_tensors)
359
360
@torch.no_grad()
361
def forward(
@@ -346,9 +365,9 @@ class CohereForCausalLM(nn.Module, SupportsLoRA):
365
kv_caches: List[torch.Tensor],
366
attn_metadata: AttentionMetadata,
367
intermediate_tensors: Optional[IntermediateTensors] = None,
349
- ) -> torch.Tensor:
368
+ ) -> Union[torch.Tensor, IntermediateTensors]:
369
hidden_states = self.model(input_ids, positions, kv_caches,
351
- attn_metadata)
370
+ attn_metadata, intermediate_tensors)
371
return hidden_states
372
373
def compute_logits(
@@ -393,6 +412,8 @@ class CohereForCausalLM(nn.Module, SupportsLoRA):
412
# Skip loading extra bias for GPTQ models.
413
if name.endswith(".bias") and name not in params_dict:
414
continue
415
+ if is_pp_missing_parameter(name, self):
416
+ continue
417
param = params_dict[name]
418
weight_loader = param.weight_loader
419
weight_loader(param, loaded_weight, shard_id)
@@ -405,6 +426,8 @@ class CohereForCausalLM(nn.Module, SupportsLoRA):
426
# Skip loading extra bias for GPTQ models.
427
if name.endswith(".bias") and name not in params_dict:
428
continue
429
+ if is_pp_missing_parameter(name, self):
430
+ continue
431
param = params_dict[name]
432
weight_loader = getattr(param, "weight_loader",
433
default_weight_loader)
vllm/model_executor/models/dbrx.py
+36
-15
@@ -1,20 +1,19 @@
1
# coding=utf-8
2
-from typing import Iterable, List, Optional, Tuple
2
+from typing import Iterable, List, Optional, Tuple, Union
3
4
import torch
5
import torch.nn as nn
6
7
from vllm.attention import Attention, AttentionMetadata
8
from vllm.config import CacheConfig
9
-from vllm.distributed import (get_tensor_model_parallel_rank,
9
+from vllm.distributed import (get_pp_group, get_tensor_model_parallel_rank,
10
get_tensor_model_parallel_world_size)
11
from vllm.model_executor.layers.fused_moe import FusedMoE
12
from vllm.model_executor.layers.linear import (QKVParallelLinear,
13
ReplicatedLinear,
14
RowParallelLinear)
15
from vllm.model_executor.layers.logits_processor import LogitsProcessor
16
-from vllm.model_executor.layers.quantization.base_config import (
17
- QuantizationConfig)
16
+from vllm.model_executor.layers.quantization import QuantizationConfig
17
from vllm.model_executor.layers.rotary_embedding import get_rope
18
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
19
from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -24,6 +23,10 @@ from vllm.model_executor.sampling_metadata import SamplingMetadata
23
from vllm.sequence import IntermediateTensors
24
from vllm.transformers_utils.configs.dbrx import DbrxConfig
25
26
+from .interfaces import SupportsPP
27
+from .utils import (is_pp_missing_parameter,
28
+ make_empty_intermediate_tensors_factory, make_layers)
29
+
30
31
class DbrxRouter(nn.Module):
32
"""A Router implementation for DBRX that returns logits for each expert
@@ -296,22 +299,27 @@ class DbrxModel(nn.Module):
299
config: DbrxConfig,
300
cache_config: Optional[CacheConfig] = None,
301
quant_config: Optional[QuantizationConfig] = None,
302
+ prefix: str = "",
303
):
304
super().__init__()
305
self.wte = VocabParallelEmbedding(
306
config.vocab_size,
307
config.d_model,
308
)
305
- self.blocks = nn.ModuleList([
306
- DbrxBlock(config, cache_config, quant_config)
307
- for _ in range(config.n_layers)
308
- ])
309
+ self.start_layer, self.end_layer, self.blocks = make_layers(
310
+ config.n_layers,
311
+ lambda prefix: DbrxBlock(config, cache_config, quant_config),
312
+ prefix=f"{prefix}.blocks",
313
+ )
314
self.norm_f = nn.LayerNorm(config.d_model, eps=1e-5)
315
for module in self.modules():
316
if hasattr(module, "bias") and isinstance(module.bias,
317
nn.Parameter):
318
# Remove the bias term in Linear and LayerNorm.
319
module.register_parameter("bias", None)
320
+ self.make_empty_intermediate_tensors = (
321
+ make_empty_intermediate_tensors_factory(["hidden_states"],
322
+ config.d_model))
323
324
def forward(
325
self,
@@ -319,21 +327,28 @@ class DbrxModel(nn.Module):
327
position_ids: torch.Tensor,
328
kv_caches: List[torch.Tensor],
329
attn_metadata: AttentionMetadata,
322
- ) -> torch.Tensor:
323
- hidden_states = self.wte(input_ids)
324
- for i in range(len(self.blocks)):
330
+ intermediate_tensors: Optional[IntermediateTensors],
331
+ ) -> Union[torch.Tensor, IntermediateTensors]:
332
+ if get_pp_group().is_first_rank:
333
+ hidden_states = self.wte(input_ids)
334
+ else:
335
+ assert intermediate_tensors
336
+ hidden_states = intermediate_tensors["hidden_states"]
337
+ for i in range(self.start_layer, self.end_layer):
338
block = self.blocks[i]
339
hidden_states = block(
340
position_ids,
341
hidden_states,
329
- kv_caches[i],
342
+ kv_caches[i - self.start_layer],
343
attn_metadata,
344
)
345
+ if not get_pp_group().is_last_rank:
346
+ return IntermediateTensors({"hidden_states": hidden_states})
347
hidden_states = self.norm_f(hidden_states)
348
return hidden_states
349
350
336
-class DbrxForCausalLM(nn.Module):
351
+class DbrxForCausalLM(nn.Module, SupportsPP):
352
353
def __init__(
354
self,
@@ -359,6 +374,8 @@ class DbrxForCausalLM(nn.Module):
374
self.logits_processor = LogitsProcessor(self.unpadded_vocab_size,
375
config.vocab_size)
376
self.sampler = Sampler()
377
+ self.make_empty_intermediate_tensors = (
378
+ self.transformer.make_empty_intermediate_tensors)
379
380
def forward(
381
self,
@@ -367,9 +384,9 @@ class DbrxForCausalLM(nn.Module):
384
kv_caches: List[torch.Tensor],
385
attn_metadata: AttentionMetadata,
386
intermediate_tensors: Optional[IntermediateTensors] = None,
370
- ) -> torch.Tensor:
387
+ ) -> Union[torch.Tensor, IntermediateTensors]:
388
hidden_states = self.transformer(input_ids, positions, kv_caches,
372
- attn_metadata)
389
+ attn_metadata, intermediate_tensors)
390
return hidden_states
391
392
def compute_logits(
@@ -401,11 +418,15 @@ class DbrxForCausalLM(nn.Module):
418
if weight_name not in name:
419
continue
420
name = name.replace(weight_name, param_name)
421
+ if is_pp_missing_parameter(name, self):
422
+ continue
423
param = params_dict[name]
424
weight_loader = param.weight_loader
425
weight_loader(param, loaded_weight, weight_name)
426
break
427
else:
428
+ if is_pp_missing_parameter(name, self):
429
+ continue
430
param = params_dict[name]
431
weight_loader = getattr(param, "weight_loader",
432
default_weight_loader)
vllm/model_executor/models/decilm.py
+7
-2
@@ -29,11 +29,12 @@ import torch
29
from transformers import LlamaConfig
30
31
from vllm.config import CacheConfig, LoRAConfig
32
-from vllm.model_executor.layers.quantization.base_config import (
33
- QuantizationConfig)
32
+from vllm.model_executor.layers.quantization import QuantizationConfig
33
from vllm.model_executor.model_loader.weight_utils import default_weight_loader
34
from vllm.model_executor.models.llama import LlamaForCausalLM
35
36
+from .utils import is_pp_missing_parameter
37
+
38
39
class DeciLMForCausalLM(LlamaForCausalLM):
40
"""
@@ -91,6 +92,8 @@ class DeciLMForCausalLM(LlamaForCausalLM):
92
# Skip loading extra bias for GPTQ models.
93
if name.endswith(".bias") and name not in params_dict:
94
continue
95
+ if is_pp_missing_parameter(name, self):
96
+ continue
97
param = params_dict[name]
98
weight_loader = param.weight_loader
99
weight_loader(param, loaded_weight, shard_id)
@@ -99,6 +102,8 @@ class DeciLMForCausalLM(LlamaForCausalLM):
102
# Skip loading extra bias for GPTQ models.
103
if name.endswith(".bias") and name not in params_dict:
104
continue
105
+ if is_pp_missing_parameter(name, self):
106
+ continue
107
param = params_dict[name]
108
weight_loader = getattr(param, "weight_loader",
109
default_weight_loader)
vllm/model_executor/models/deepseek.py
+43
-20
@@ -21,7 +21,7 @@
21
# See the License for the specific language governing permissions and
22
# limitations under the License.
23
"""Inference-only Deepseek model."""
24
-from typing import Any, Dict, Iterable, List, Optional, Tuple
24
+from typing import Any, Dict, Iterable, List, Optional, Tuple, Union
25
26
import torch
27
from torch import nn
@@ -29,7 +29,7 @@ from transformers import PretrainedConfig
29
30
from vllm.attention import Attention, AttentionMetadata
31
from vllm.config import CacheConfig
32
-from vllm.distributed import (get_tensor_model_parallel_rank,
32
+from vllm.distributed import (get_pp_group, get_tensor_model_parallel_rank,
33
get_tensor_model_parallel_world_size,
34
tensor_model_parallel_all_reduce)
35
from vllm.model_executor.layers.activation import SiluAndMul
@@ -40,8 +40,7 @@ from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
40
ReplicatedLinear,
41
RowParallelLinear)
42
from vllm.model_executor.layers.logits_processor import LogitsProcessor
43
-from vllm.model_executor.layers.quantization.base_config import (
44
- QuantizationConfig)
43
+from vllm.model_executor.layers.quantization import QuantizationConfig
44
from vllm.model_executor.layers.rotary_embedding import get_rope
45
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
46
from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -50,6 +49,10 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
49
from vllm.model_executor.sampling_metadata import SamplingMetadata
50
from vllm.sequence import IntermediateTensors
51
52
+from .interfaces import SupportsPP
53
+from .utils import (is_pp_missing_parameter,
54
+ make_empty_intermediate_tensors_factory, make_layers)
55
+
56
57
class DeepseekMLP(nn.Module):
58
@@ -329,6 +332,7 @@ class DeepseekModel(nn.Module):
332
config: PretrainedConfig,
333
cache_config: Optional[CacheConfig] = None,
334
quant_config: Optional[QuantizationConfig] = None,
335
+ prefix: str = "",
336
) -> None:
337
super().__init__()
338
self.padding_idx = config.pad_token_id
@@ -338,14 +342,17 @@ class DeepseekModel(nn.Module):
342
config.vocab_size,
343
config.hidden_size,
344
)
341
- self.layers = nn.ModuleList([
342
- DeepseekDecoderLayer(config,
343
- layer_idx,
344
- cache_config,
345
- quant_config=quant_config)
346
- for layer_idx in range(config.num_hidden_layers)
347
- ])
345
+ self.start_layer, self.end_layer, self.layers = make_layers(
346
+ config.num_hidden_layers,
347
+ lambda prefix: DeepseekDecoderLayer(config,
348
+ int(prefix.split(".")[-1]),
349
+ cache_config,
350
+ quant_config=quant_config),
351
+ prefix=f"{prefix}.layers")
352
self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
353
+ self.make_empty_intermediate_tensors = (
354
+ make_empty_intermediate_tensors_factory(
355
+ ["hidden_states", "residual"], config.hidden_size))
356
357
def forward(
358
self,
@@ -353,19 +360,29 @@ class DeepseekModel(nn.Module):
360
positions: torch.Tensor,
361
kv_caches: List[torch.Tensor],
362
attn_metadata: AttentionMetadata,
356
- ) -> torch.Tensor:
357
- hidden_states = self.embed_tokens(input_ids)
358
- residual = None
359
- for i in range(len(self.layers)):
363
+ intermediate_tensors: Optional[IntermediateTensors],
364
+ ) -> Union[torch.Tensor, IntermediateTensors]:
365
+ if get_pp_group().is_first_rank:
366
+ hidden_states = self.embed_tokens(input_ids)
367
+ residual = None
368
+ else:
369
+ hidden_states = intermediate_tensors["hidden_states"]
370
+ residual = intermediate_tensors["residual"]
371
+ for i in range(self.start_layer, self.end_layer):
372
layer = self.layers[i]
373
hidden_states, residual = layer(positions, hidden_states,
362
- kv_caches[i], attn_metadata,
363
- residual)
374
+ kv_caches[i - self.start_layer],
375
+ attn_metadata, residual)
376
+ if not get_pp_group().is_last_rank:
377
+ return IntermediateTensors({
378
+ "hidden_states": hidden_states,
379
+ "residual": residual
380
+ })
381
hidden_states, _ = self.norm(hidden_states, residual)
382
return hidden_states
383
384
368
-class DeepseekForCausalLM(nn.Module):
385
+class DeepseekForCausalLM(nn.Module, SupportsPP):
386
387
def __init__(
388
self,
@@ -384,6 +401,8 @@ class DeepseekForCausalLM(nn.Module):
401
self.lm_head.weight = self.model.embed_tokens.weight
402
self.logits_processor = LogitsProcessor(config.vocab_size)
403
self.sampler = Sampler()
404
+ self.make_empty_intermediate_tensors = (
405
+ self.model.make_empty_intermediate_tensors)
406
407
def forward(
408
self,
@@ -392,9 +411,9 @@ class DeepseekForCausalLM(nn.Module):
411
kv_caches: List[torch.Tensor],
412
attn_metadata: AttentionMetadata,
413
intermediate_tensors: Optional[IntermediateTensors] = None,
395
- ) -> torch.Tensor:
414
+ ) -> Union[torch.Tensor, IntermediateTensors]:
415
hidden_states = self.model(input_ids, positions, kv_caches,
397
- attn_metadata)
416
+ attn_metadata, intermediate_tensors)
417
return hidden_states
418
419
def compute_logits(
@@ -439,6 +458,8 @@ class DeepseekForCausalLM(nn.Module):
458
if (("mlp.experts." in name or "mlp.shared_experts." in name)
459
and name not in params_dict):
460
continue
461
+ if is_pp_missing_parameter(name, self):
462
+ continue
463
param = params_dict[name]
464
weight_loader = param.weight_loader
465
weight_loader(param, loaded_weight, shard_id)
@@ -451,6 +472,8 @@ class DeepseekForCausalLM(nn.Module):
472
if (("mlp.experts." in name or "mlp.shared_experts." in name)
473
and name not in params_dict):
474
continue
475
+ if is_pp_missing_parameter(name, self):
476
+ continue
477
param = params_dict[name]
478
weight_loader = getattr(param, "weight_loader",
479
default_weight_loader)
vllm/model_executor/models/deepseek_v2.py
+13
-7
@@ -21,7 +21,7 @@
21
# See the License for the specific language governing permissions and
22
# limitations under the License.
23
"""Inference-only DeepseekV2 model."""
24
-from typing import Any, Dict, Iterable, List, Optional, Tuple
24
+from typing import Any, Dict, Iterable, List, Optional, Tuple, Union
25
26
import torch
27
from torch import nn
@@ -40,8 +40,7 @@ from vllm.model_executor.layers.linear import (ColumnParallelLinear,
40
ReplicatedLinear,
41
RowParallelLinear)
42
from vllm.model_executor.layers.logits_processor import LogitsProcessor
43
-from vllm.model_executor.layers.quantization.base_config import (
44
- QuantizationConfig)
43
+from vllm.model_executor.layers.quantization import QuantizationConfig
44
from vllm.model_executor.layers.rotary_embedding import get_rope
45
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
46
from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -50,7 +49,9 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
49
from vllm.model_executor.sampling_metadata import SamplingMetadata
50
from vllm.sequence import IntermediateTensors
51
53
-from .utils import PPMissingLayer, is_pp_missing_parameter, make_layers
52
+from .interfaces import SupportsPP
53
+from .utils import (PPMissingLayer, is_pp_missing_parameter,
54
+ make_empty_intermediate_tensors_factory, make_layers)
55
56
57
class DeepseekV2MLP(nn.Module):
@@ -439,6 +440,9 @@ class DeepseekV2Model(nn.Module):
440
self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
441
else:
442
self.norm = PPMissingLayer()
443
+ self.make_empty_intermediate_tensors = (
444
+ make_empty_intermediate_tensors_factory(
445
+ ["hidden_states", "residual"], config.hidden_size))
446
447
def forward(
448
self,
@@ -447,7 +451,7 @@ class DeepseekV2Model(nn.Module):
451
kv_caches: List[torch.Tensor],
452
attn_metadata: AttentionMetadata,
453
intermediate_tensors: Optional[IntermediateTensors],
450
- ) -> torch.Tensor:
454
+ ) -> Union[torch.Tensor, IntermediateTensors]:
455
if get_pp_group().is_first_rank:
456
hidden_states = self.embed_tokens(input_ids)
457
residual = None
@@ -472,7 +476,7 @@ class DeepseekV2Model(nn.Module):
476
return hidden_states
477
478
475
-class DeepseekV2ForCausalLM(nn.Module):
479
+class DeepseekV2ForCausalLM(nn.Module, SupportsPP):
480
481
def __init__(
482
self,
@@ -492,6 +496,8 @@ class DeepseekV2ForCausalLM(nn.Module):
496
quant_config=quant_config)
497
self.logits_processor = LogitsProcessor(config.vocab_size)
498
self.sampler = Sampler()
499
+ self.make_empty_intermediate_tensors = (
500
+ self.model.make_empty_intermediate_tensors)
501
502
def forward(
503
self,
@@ -500,7 +506,7 @@ class DeepseekV2ForCausalLM(nn.Module):
506
kv_caches: List[torch.Tensor],
507
attn_metadata: AttentionMetadata,
508
intermediate_tensors: Optional[IntermediateTensors] = None,
503
- ) -> torch.Tensor:
509
+ ) -> Union[torch.Tensor, IntermediateTensors]:
510
hidden_states = self.model(input_ids, positions, kv_caches,
511
attn_metadata, intermediate_tensors)
512
return hidden_states
vllm/model_executor/models/exaone.py
+12
-23
@@ -38,8 +38,7 @@ from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
38
QKVParallelLinear,
39
RowParallelLinear)
40
from vllm.model_executor.layers.logits_processor import LogitsProcessor
41
-from vllm.model_executor.layers.quantization.base_config import (
42
- QuantizationConfig)
41
+from vllm.model_executor.layers.quantization import QuantizationConfig
42
from vllm.model_executor.layers.quantization.compressed_tensors.utils import (
43
get_compressed_tensors_cache_scale)
44
from vllm.model_executor.layers.rotary_embedding import get_rope
@@ -53,8 +52,9 @@ from vllm.sequence import IntermediateTensors
52
from vllm.transformers_utils.configs.exaone import ExaoneConfig
53
from vllm.utils import is_hip
54
56
-from .interfaces import SupportsLoRA
57
-from .utils import PPMissingLayer, is_pp_missing_parameter, make_layers
55
+from .interfaces import SupportsLoRA, SupportsPP
56
+from .utils import (PPMissingLayer, is_pp_missing_parameter,
57
+ make_empty_intermediate_tensors_factory, make_layers)
58
59
60
class ExaoneGatedMLP(nn.Module):
@@ -354,6 +354,10 @@ class ExaoneModel(nn.Module):
354
else:
355
self.ln_f = PPMissingLayer()
356
357
+ self.make_empty_intermediate_tensors = (
358
+ make_empty_intermediate_tensors_factory(
359
+ ["hidden_states", "residual"], config.hidden_size))
360
+
361
def get_input_embeddings(self, input_ids: torch.Tensor) -> torch.Tensor:
362
return self.wte(input_ids)
363
@@ -397,7 +401,7 @@ class ExaoneModel(nn.Module):
401
return hidden_states
402
403
400
-class ExaoneForCausalLM(nn.Module, SupportsLoRA):
404
+class ExaoneForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
405
packed_modules_mapping = {
406
"qkv_proj": [
407
"q_proj",
@@ -477,6 +481,9 @@ class ExaoneForCausalLM(nn.Module, SupportsLoRA):
481
else:
482
self.lm_head = PPMissingLayer()
483
484
+ self.make_empty_intermediate_tensors = (
485
+ self.transformer.make_empty_intermediate_tensors)
486
+
487
def forward(
488
self,
489
input_ids: torch.Tensor,
@@ -506,24 +513,6 @@ class ExaoneForCausalLM(nn.Module, SupportsLoRA):
513
next_tokens = self.sampler(logits, sampling_metadata)
514
return next_tokens
515
509
- def make_empty_intermediate_tensors(
510
- self, batch_size: int, dtype: torch.dtype,
511
- device: torch.device) -> IntermediateTensors:
512
- return IntermediateTensors({
513
- "hidden_states":
514
- torch.zeros(
515
- (batch_size, self.config.hidden_size),
516
- dtype=dtype,
517
- device=device,
518
- ),
519
- "residual":
520
- torch.zeros(
521
- (batch_size, self.config.hidden_size),
522
- dtype=dtype,
523
- device=device,
524
- ),
525
- })
526
-
516
def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
517
stacked_params_mapping = [
518
# (param_name, shard_name, shard_id)
vllm/model_executor/models/falcon.py
+33
-19
@@ -28,7 +28,7 @@ from transformers import FalconConfig as HF_FalconConfig
28
29
from vllm.attention import Attention, AttentionMetadata
30
from vllm.config import CacheConfig
31
-from vllm.distributed import (get_tensor_model_parallel_rank,
31
+from vllm.distributed import (get_pp_group, get_tensor_model_parallel_rank,
32
get_tensor_model_parallel_world_size,
33
tensor_model_parallel_all_reduce)
34
from vllm.model_executor.layers.activation import get_act_fn
@@ -36,8 +36,7 @@ from vllm.model_executor.layers.linear import (ColumnParallelLinear,
36
QKVParallelLinear,
37
RowParallelLinear)
38
from vllm.model_executor.layers.logits_processor import LogitsProcessor
39
-from vllm.model_executor.layers.quantization.base_config import (
40
- QuantizationConfig)
39
+from vllm.model_executor.layers.quantization import QuantizationConfig
40
from vllm.model_executor.layers.rotary_embedding import get_rope
41
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
42
from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -47,6 +46,10 @@ from vllm.model_executor.sampling_metadata import SamplingMetadata
46
from vllm.sequence import IntermediateTensors
47
from vllm.transformers_utils.configs import RWConfig
48
49
+from .interfaces import SupportsPP
50
+from .utils import (is_pp_missing_parameter,
51
+ make_empty_intermediate_tensors_factory, make_layers)
52
+
53
FalconConfig = Union[HF_FalconConfig, RWConfig]
54
55
@@ -333,6 +336,7 @@ class FalconModel(nn.Module):
336
config: FalconConfig,
337
cache_config: Optional[CacheConfig] = None,
338
quant_config: Optional[QuantizationConfig] = None,
339
+ prefix: str = "",
340
):
341
super().__init__()
342
self.config = config
@@ -347,35 +351,45 @@ class FalconModel(nn.Module):
351
)
352
353
# Transformer blocks
350
- self.h = nn.ModuleList([
351
- FalconDecoderLayer(config, cache_config, quant_config)
352
- for _ in range(config.num_hidden_layers)
353
- ])
354
+ self.start_layer, self.end_layer, self.h = make_layers(
355
+ config.num_hidden_layers,
356
+ lambda prefix: FalconDecoderLayer(config, cache_config,
357
+ quant_config),
358
+ prefix=f"{prefix}.h")
359
360
# Final Layer Norm
361
self.ln_f = LayerNorm(self.embed_dim, eps=config.layer_norm_epsilon)
362
+ self.make_empty_intermediate_tensors = (
363
+ make_empty_intermediate_tensors_factory(["hidden_states"],
364
+ config.hidden_size))
365
366
def forward(
367
self,
360
- input_ids: torch.LongTensor,
368
+ input_ids: torch.Tensor,
369
positions: torch.Tensor,
370
kv_caches: List[torch.Tensor],
371
attn_metadata: AttentionMetadata,
364
- ) -> torch.Tensor:
365
- hidden_states = self.word_embeddings(input_ids)
366
- for i in range(len(self.h)):
372
+ intermediate_tensors: Optional[IntermediateTensors],
373
+ ) -> Union[torch.Tensor, IntermediateTensors]:
374
+ if get_pp_group().is_first_rank:
375
+ hidden_states = self.word_embeddings(input_ids)
376
+ else:
377
+ hidden_states = intermediate_tensors["hidden_states"]
378
+ for i in range(self.start_layer, self.end_layer):
379
layer = self.h[i]
380
hidden_states = layer(
381
positions,
382
hidden_states,
371
- kv_caches[i],
383
+ kv_caches[i - self.start_layer],
384
attn_metadata,
385
)
386
+ if not get_pp_group().is_last_rank:
387
+ return IntermediateTensors({"hidden_states": hidden_states})
388
hidden_states = self.ln_f(hidden_states)
389
return hidden_states
390
391
378
-class FalconForCausalLM(nn.Module):
392
+class FalconForCausalLM(nn.Module, SupportsPP):
393
394
def __init__(
395
self,
@@ -403,6 +417,8 @@ class FalconForCausalLM(nn.Module):
417
)
418
self.logits_processor = LogitsProcessor(config.vocab_size)
419
self.sampler = Sampler()
420
+ self.make_empty_intermediate_tensors = (
421
+ self.transformer.make_empty_intermediate_tensors)
422
423
def forward(
424
self,
@@ -412,12 +428,8 @@ class FalconForCausalLM(nn.Module):
428
attn_metadata: AttentionMetadata,
429
intermediate_tensors: Optional[IntermediateTensors] = None,
430
) -> torch.Tensor:
415
- hidden_states = self.transformer(
416
- input_ids,
417
- positions,
418
- kv_caches,
419
- attn_metadata,
420
- )
431
+ hidden_states = self.transformer(input_ids, positions, kv_caches,
432
+ attn_metadata, intermediate_tensors)
433
return hidden_states
434
435
def compute_logits(
@@ -454,6 +466,8 @@ class FalconForCausalLM(nn.Module):
466
# Skip loading extra bias for GPTQ models.
467
if name.endswith(".bias") and name not in params_dict:
468
continue
469
+ if is_pp_missing_parameter(name, self):
470
+ continue
471
param = params_dict[name]
472
if "query_key_value" in name:
473
output_dim = getattr(param, "output_dim", None)
vllm/model_executor/models/fuyu.py
+34
-39
@@ -41,8 +41,9 @@ from vllm.multimodal.utils import cached_get_tokenizer
41
from vllm.sequence import (VLLM_TOKEN_ID_ARRAY_TYPE, IntermediateTensors,
42
SequenceData)
43
44
-from .interfaces import SupportsMultiModal
45
-from .utils import flatten_bn, merge_multimodal_embeddings
44
+from .interfaces import SupportsMultiModal, SupportsPP
45
+from .utils import (flatten_bn, group_weights_with_prefix,
46
+ merge_multimodal_embeddings)
47
48
# Cannot find the following 2 numbers from hf config.
49
_IMAGE_TOKEN_ID = 71011
@@ -217,7 +218,7 @@ def input_mapper_for_fuyu(ctx: InputContext, data: object):
218
@MULTIMODAL_REGISTRY.register_max_image_tokens(get_max_fuyu_image_tokens)
219
@INPUT_REGISTRY.register_dummy_data(dummy_data_for_fuyu)
220
@INPUT_REGISTRY.register_input_processor(input_processor_for_fuyu)
220
-class FuyuForCausalLM(nn.Module, SupportsMultiModal):
221
+class FuyuForCausalLM(nn.Module, SupportsMultiModal, SupportsPP):
222
223
def __init__(self,
224
config: FuyuConfig,
@@ -242,6 +243,12 @@ class FuyuForCausalLM(nn.Module, SupportsMultiModal):
243
self.language_model = PersimmonForCausalLM(config.text_config,
244
cache_config=cache_config,
245
quant_config=quant_config)
246
+ self.make_empty_intermediate_tensors = (
247
+ self.language_model.make_empty_intermediate_tensors)
248
+
249
+ @property
250
+ def sampler(self):
251
+ return self.language_model.sampler
252
253
def _validate_pixel_values(self, data: torch.Tensor) -> torch.Tensor:
254
@@ -297,23 +304,29 @@ class FuyuForCausalLM(nn.Module, SupportsMultiModal):
304
intermediate_tensors: Optional[IntermediateTensors] = None,
305
**kwargs: object,
306
):
300
- image_input = self._parse_and_validate_image_input(**kwargs)
307
+ if intermediate_tensors is not None:
308
+ input_ids = None
309
+ inputs_embeds = None
310
+ else:
311
+ image_input = self._parse_and_validate_image_input(**kwargs)
312
302
- if image_input is not None:
303
- vision_embeddings = self._process_image_input(image_input)
304
- inputs_embeds = self.language_model.model.embed_tokens(input_ids)
305
- inputs_embeds = merge_multimodal_embeddings(
306
- input_ids, inputs_embeds, vision_embeddings,
307
- self.image_token_id)
313
+ if image_input is not None:
314
+ vision_embeddings = self._process_image_input(image_input)
315
+ inputs_embeds = self.language_model.model.embed_tokens(
316
+ input_ids)
317
+ inputs_embeds = merge_multimodal_embeddings(
318
+ input_ids, inputs_embeds, vision_embeddings,
319
+ self.image_token_id)
320
309
- else:
310
- inputs_embeds = None
321
+ else:
322
+ inputs_embeds = None
323
324
hidden_states = self.language_model(
325
input_ids=input_ids,
326
positions=positions,
327
kv_caches=kv_caches,
328
attn_metadata=attn_metadata,
329
+ intermediate_tensors=intermediate_tensors,
330
inputs_embeds=inputs_embeds,
331
)
332
return hidden_states
@@ -336,34 +349,16 @@ class FuyuForCausalLM(nn.Module, SupportsMultiModal):
349
return next_tokens
350
351
def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
339
- params_dict = dict(self.named_parameters(remove_duplicate=False))
340
- for name, loaded_weight in weights:
341
- if "rotary_emb.inv_freq" in name:
342
- continue
343
- if ("rotary_emb.cos_cached" in name
344
- or "rotary_emb.sin_cached" in name):
345
- # Models trained using ColossalAI may include these tensors in
346
- # the checkpoint. Skip them.
347
- continue
348
- param = params_dict[name]
349
-
350
- if "query_key_value" in name:
351
- # copy from vllm/model_executor/models/bloom.py
352
- # NOTE: Fuyu's fused QKV's output_dim has the shape of
353
- # (num_heads * 3 * head_size), while the
354
- # required shape is (3 * num_heads * head_size).
355
- # Thus, we need weight conversion.
356
- output_dim = getattr(param, "output_dim", None)
357
- num_heads = self.config.num_attention_heads
358
- if output_dim is not None:
359
- loaded_weight_shape = loaded_weight.shape
360
- loaded_weight = loaded_weight.view(
361
- loaded_weight_shape[:output_dim] + (num_heads, 3, -1) +
362
- loaded_weight_shape[output_dim + 1:])
363
- loaded_weight = loaded_weight.transpose(
364
- output_dim, output_dim + 1)
365
- loaded_weight = loaded_weight.reshape(loaded_weight_shape)
352
+ # prepare weight iterators for components
353
+ weights_group = group_weights_with_prefix(weights)
354
355
+ # load vision embeddings
356
+ vision_params_dict = dict(self.vision_embed_tokens.named_parameters())
357
+ for name, loaded_weight in weights_group["vision_embed_tokens"]:
358
+ param = vision_params_dict[name]
359
weight_loader = getattr(param, "weight_loader",
360
default_weight_loader)
361
weight_loader(param, loaded_weight)
362
+
363
+ # load llm backbone
364
+ self.language_model.load_weights(weights_group["language_model"])
vllm/model_executor/models/gemma.py
+42
-21
@@ -15,7 +15,7 @@
15
# limitations under the License.
16
"""Inference-only Gemma model compatible with HuggingFace weights."""
17
from functools import lru_cache
18
-from typing import Iterable, List, Optional, Set, Tuple
18
+from typing import Iterable, List, Optional, Set, Tuple, Union
19
20
import torch
21
from torch import nn
@@ -23,7 +23,7 @@ from transformers import GemmaConfig
23
24
from vllm.attention import Attention, AttentionMetadata
25
from vllm.config import CacheConfig, LoRAConfig
26
-from vllm.distributed import get_tensor_model_parallel_world_size
26
+from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
27
from vllm.logger import init_logger
28
from vllm.model_executor.layers.activation import GeluAndMul
29
from vllm.model_executor.layers.layernorm import GemmaRMSNorm
@@ -31,8 +31,7 @@ from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
31
QKVParallelLinear,
32
RowParallelLinear)
33
from vllm.model_executor.layers.logits_processor import LogitsProcessor
34
-from vllm.model_executor.layers.quantization.base_config import (
35
- QuantizationConfig)
34
+from vllm.model_executor.layers.quantization import QuantizationConfig
35
from vllm.model_executor.layers.rotary_embedding import get_rope
36
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
37
from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -41,7 +40,9 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
40
from vllm.model_executor.sampling_metadata import SamplingMetadata
41
from vllm.sequence import IntermediateTensors
42
44
-from .interfaces import SupportsLoRA
43
+from .interfaces import SupportsLoRA, SupportsPP
44
+from .utils import (is_pp_missing_parameter,
45
+ make_empty_intermediate_tensors_factory, make_layers)
46
47
logger = init_logger(__name__)
48
@@ -245,6 +246,7 @@ class GemmaModel(nn.Module):
246
config: GemmaConfig,
247
cache_config: Optional[CacheConfig] = None,
248
quant_config: Optional[QuantizationConfig] = None,
249
+ prefix: str = "",
250
) -> None:
251
super().__init__()
252
self.config = config
@@ -253,10 +255,11 @@ class GemmaModel(nn.Module):
255
config.vocab_size,
256
config.hidden_size,
257
)
256
- self.layers = nn.ModuleList([
257
- GemmaDecoderLayer(config, cache_config, quant_config)
258
- for _ in range(config.num_hidden_layers)
259
- ])
258
+ self.start_layer, self.end_layer, self.layers = make_layers(
259
+ config.num_hidden_layers,
260
+ lambda prefix: GemmaDecoderLayer(config, cache_config, quant_config
261
+ ),
262
+ prefix=f"{prefix}.layers")
263
self.norm = GemmaRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
264
265
# Normalize the embedding by sqrt(hidden_size)
@@ -265,6 +268,9 @@ class GemmaModel(nn.Module):
268
# See https://github.com/huggingface/transformers/pull/29402
269
normalizer = self.config.hidden_size**0.5
270
self.register_buffer("normalizer", torch.tensor(normalizer))
271
+ self.make_empty_intermediate_tensors = (
272
+ make_empty_intermediate_tensors_factory(
273
+ ["hidden_states", "residual"], config.hidden_size))
274
275
def get_input_embeddings(self, input_ids: torch.Tensor) -> torch.Tensor:
276
return self.embed_tokens(input_ids)
@@ -275,29 +281,38 @@ class GemmaModel(nn.Module):
281
positions: torch.Tensor,
282
kv_caches: List[torch.Tensor],
283
attn_metadata: AttentionMetadata,
278
- intermediate_tensors: Optional[IntermediateTensors] = None,
284
+ intermediate_tensors: Optional[IntermediateTensors],
285
inputs_embeds: Optional[torch.Tensor] = None,
280
- ) -> torch.Tensor:
281
- if inputs_embeds is not None:
282
- hidden_states = inputs_embeds
286
+ ) -> Union[torch.Tensor, IntermediateTensors]:
287
+ if get_pp_group().is_first_rank:
288
+ if inputs_embeds is not None:
289
+ hidden_states = inputs_embeds
290
+ else:
291
+ hidden_states = self.get_input_embeddings(input_ids)
292
+ hidden_states *= self.normalizer
293
+ residual = None
294
else:
284
- hidden_states = self.get_input_embeddings(input_ids)
285
- hidden_states *= self.normalizer
286
- residual = None
287
- for i in range(len(self.layers)):
295
+ hidden_states = intermediate_tensors["hidden_states"]
296
+ residual = intermediate_tensors["residual"]
297
+ for i in range(self.start_layer, self.end_layer):
298
layer = self.layers[i]
299
hidden_states, residual = layer(
300
positions,
301
hidden_states,
292
- kv_caches[i],
302
+ kv_caches[i - self.start_layer],
303
attn_metadata,
304
residual,
305
)
306
+ if not get_pp_group().is_last_rank:
307
+ return IntermediateTensors({
308
+ "hidden_states": hidden_states,
309
+ "residual": residual
310
+ })
311
hidden_states, _ = self.norm(hidden_states, residual)
312
return hidden_states
313
314
300
-class GemmaForCausalLM(nn.Module, SupportsLoRA):
315
+class GemmaForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
316
packed_modules_mapping = {
317
"qkv_proj": [
318
"q_proj",
@@ -339,6 +354,8 @@ class GemmaForCausalLM(nn.Module, SupportsLoRA):
354
self.model = GemmaModel(config, cache_config, quant_config)
355
self.logits_processor = LogitsProcessor(config.vocab_size)
356
self.sampler = Sampler()
357
+ self.make_empty_intermediate_tensors = (
358
+ self.model.make_empty_intermediate_tensors)
359
360
def forward(
361
self,
@@ -347,9 +364,9 @@ class GemmaForCausalLM(nn.Module, SupportsLoRA):
364
kv_caches: List[torch.Tensor],
365
attn_metadata: AttentionMetadata,
366
intermediate_tensors: Optional[IntermediateTensors] = None,
350
- ) -> torch.Tensor:
367
+ ) -> Union[torch.Tensor, IntermediateTensors]:
368
hidden_states = self.model(input_ids, positions, kv_caches,
352
- attn_metadata)
369
+ attn_metadata, intermediate_tensors)
370
return hidden_states
371
372
def compute_logits(
@@ -388,6 +405,8 @@ class GemmaForCausalLM(nn.Module, SupportsLoRA):
405
# Skip loading extra bias for GPTQ models.
406
if name.endswith(".bias") and name not in params_dict:
407
continue
408
+ if is_pp_missing_parameter(name, self):
409
+ continue
410
param = params_dict[name]
411
weight_loader = param.weight_loader
412
weight_loader(param, loaded_weight, shard_id)
@@ -400,6 +419,8 @@ class GemmaForCausalLM(nn.Module, SupportsLoRA):
419
# Skip loading extra bias for GPTQ models.
420
if name.endswith(".bias") and name not in params_dict:
421
continue
422
+ if is_pp_missing_parameter(name, self):
423
+ continue
424
param = params_dict[name]
425
weight_loader = getattr(param, "weight_loader",
426
default_weight_loader)
vllm/model_executor/models/gemma2.py
+41
-18
@@ -14,7 +14,7 @@
14
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15
# See the License for the specific language governing permissions and
16
# limitations under the License.
17
-from typing import Iterable, List, Optional, Set, Tuple
17
+from typing import Iterable, List, Optional, Set, Tuple, Union
18
19
import torch
20
from torch import nn
@@ -22,7 +22,7 @@ from transformers import Gemma2Config
22
23
from vllm.attention import Attention, AttentionMetadata
24
from vllm.config import CacheConfig, LoRAConfig
25
-from vllm.distributed import get_tensor_model_parallel_world_size
25
+from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
26
from vllm.logger import init_logger
27
from vllm.model_executor.layers.activation import GeluAndMul
28
from vllm.model_executor.layers.layernorm import GemmaRMSNorm
@@ -30,8 +30,7 @@ from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
30
QKVParallelLinear,
31
RowParallelLinear)
32
from vllm.model_executor.layers.logits_processor import LogitsProcessor
33
-from vllm.model_executor.layers.quantization.base_config import (
34
- QuantizationConfig)
33
+from vllm.model_executor.layers.quantization import QuantizationConfig
34
from vllm.model_executor.layers.rotary_embedding import get_rope
35
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
36
from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -40,7 +39,9 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
39
from vllm.model_executor.sampling_metadata import SamplingMetadata
40
from vllm.sequence import IntermediateTensors
41
43
-from .interfaces import SupportsLoRA
42
+from .interfaces import SupportsLoRA, SupportsPP
43
+from .utils import (is_pp_missing_parameter,
44
+ make_empty_intermediate_tensors_factory, make_layers)
45
46
logger = init_logger(__name__)
47
@@ -244,6 +245,7 @@ class Gemma2Model(nn.Module):
245
config: Gemma2Config,
246
cache_config: Optional[CacheConfig] = None,
247
quant_config: Optional[QuantizationConfig] = None,
248
+ prefix: str = "",
249
) -> None:
250
super().__init__()
251
self.config = config
@@ -252,10 +254,11 @@ class Gemma2Model(nn.Module):
254
config.vocab_size,
255
config.hidden_size,
256
)
255
- self.layers = nn.ModuleList([
256
- Gemma2DecoderLayer(layer_idx, config, cache_config, quant_config)
257
- for layer_idx in range(config.num_hidden_layers)
258
- ])
257
+ self.start_layer, self.end_layer, self.layers = make_layers(
258
+ config.num_hidden_layers,
259
+ lambda prefix: Gemma2DecoderLayer(int(prefix.split(".")[
260
+ -1]), config, cache_config, quant_config),
261
+ prefix=f"{prefix}.layers")
262
self.norm = GemmaRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
263
264
# Normalize the embedding by sqrt(hidden_size)
@@ -264,6 +267,9 @@ class Gemma2Model(nn.Module):
267
# See https://github.com/huggingface/transformers/pull/29402
268
normalizer = self.config.hidden_size**0.5
269
self.register_buffer("normalizer", torch.tensor(normalizer))
270
+ self.make_empty_intermediate_tensors = (
271
+ make_empty_intermediate_tensors_factory(
272
+ ["hidden_states", "residual"], config.hidden_size))
273
274
def forward(
275
self,
@@ -271,25 +277,36 @@ class Gemma2Model(nn.Module):
277
positions: torch.Tensor,
278
kv_caches: List[torch.Tensor],
279
attn_metadata: AttentionMetadata,
274
- ) -> torch.Tensor:
275
- hidden_states = self.embed_tokens(input_ids)
276
- hidden_states *= self.normalizer
280
+ intermediate_tensors: Optional[IntermediateTensors],
281
+ ) -> Union[torch.Tensor, IntermediateTensors]:
282
+ if get_pp_group().is_first_rank:
283
+ hidden_states = self.embed_tokens(input_ids)
284
+ hidden_states *= self.normalizer
285
278
- residual = None
279
- for i in range(len(self.layers)):
286
+ residual = None
287
+ else:
288
+ assert intermediate_tensors is not None
289
+ hidden_states = intermediate_tensors["hidden_states"]
290
+ residual = intermediate_tensors["residual"]
291
+ for i in range(self.start_layer, self.end_layer):
292
layer = self.layers[i]
293
hidden_states, residual = layer(
294
positions,
295
hidden_states,
284
- kv_caches[i],
296
+ kv_caches[i - self.start_layer],
297
attn_metadata,
298
residual,
299
)
300
+ if not get_pp_group().is_last_rank:
301
+ return IntermediateTensors({
302
+ "hidden_states": hidden_states,
303
+ "residual": residual
304
+ })
305
hidden_states, _ = self.norm(hidden_states, residual)
306
return hidden_states
307
308
292
-class Gemma2ForCausalLM(nn.Module, SupportsLoRA):
309
+class Gemma2ForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
310
packed_modules_mapping = {
311
"qkv_proj": [
312
"q_proj",
@@ -338,6 +355,8 @@ class Gemma2ForCausalLM(nn.Module, SupportsLoRA):
355
self.logits_processor = LogitsProcessor(
356
config.vocab_size, soft_cap=config.final_logit_softcapping)
357
self.sampler = Sampler()
358
+ self.make_empty_intermediate_tensors = (
359
+ self.model.make_empty_intermediate_tensors)
360
361
def forward(
362
self,
@@ -346,9 +365,9 @@ class Gemma2ForCausalLM(nn.Module, SupportsLoRA):
365
kv_caches: List[torch.Tensor],
366
attn_metadata: AttentionMetadata,
367
intermediate_tensors: Optional[IntermediateTensors] = None,
349
- ) -> torch.Tensor:
368
+ ) -> Union[torch.Tensor, IntermediateTensors]:
369
hidden_states = self.model(input_ids, positions, kv_caches,
351
- attn_metadata)
370
+ attn_metadata, intermediate_tensors)
371
return hidden_states
372
373
def compute_logits(
@@ -387,6 +406,8 @@ class Gemma2ForCausalLM(nn.Module, SupportsLoRA):
406
# Skip loading extra bias for GPTQ models.
407
if name.endswith(".bias") and name not in params_dict:
408
continue
409
+ if is_pp_missing_parameter(name, self):
410
+ continue
411
param = params_dict[name]
412
weight_loader = param.weight_loader
413
weight_loader(param, loaded_weight, shard_id)
@@ -399,6 +420,8 @@ class Gemma2ForCausalLM(nn.Module, SupportsLoRA):
420
# Skip loading extra bias for GPTQ models.
421
if name.endswith(".bias") and name not in params_dict:
422
continue
423
+ if is_pp_missing_parameter(name, self):
424
+ continue
425
param = params_dict[name]
426
weight_loader = getattr(param, "weight_loader",
427
default_weight_loader)
vllm/model_executor/models/gpt2.py
+11
-15
@@ -32,8 +32,7 @@ from vllm.model_executor.layers.linear import (ColumnParallelLinear,
32
QKVParallelLinear,
33
RowParallelLinear)
34
from vllm.model_executor.layers.logits_processor import LogitsProcessor
35
-from vllm.model_executor.layers.quantization.base_config import (
36
- QuantizationConfig)
35
+from vllm.model_executor.layers.quantization import QuantizationConfig
36
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
37
from vllm.model_executor.layers.vocab_parallel_embedding import (
38
ParallelLMHead, VocabParallelEmbedding)
@@ -41,7 +40,9 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
40
from vllm.model_executor.sampling_metadata import SamplingMetadata
41
from vllm.sequence import IntermediateTensors
42
44
-from .utils import is_pp_missing_parameter, make_layers
43
+from .interfaces import SupportsPP
44
+from .utils import (is_pp_missing_parameter,
45
+ make_empty_intermediate_tensors_factory, make_layers)
46
47
48
class GPT2Attention(nn.Module):
@@ -204,6 +205,9 @@ class GPT2Model(nn.Module):
205
config, cache_config, quant_config, prefix=prefix),
206
prefix=f"{prefix}.h")
207
self.ln_f = nn.LayerNorm(self.embed_dim, eps=config.layer_norm_epsilon)
208
+ self.make_empty_intermediate_tensors = (
209
+ make_empty_intermediate_tensors_factory(["hidden_states"],
210
+ config.n_embd))
211
212
def forward(
213
self,
@@ -234,7 +238,7 @@ class GPT2Model(nn.Module):
238
return hidden_states
239
240
237
-class GPT2LMHeadModel(nn.Module):
241
+class GPT2LMHeadModel(nn.Module, SupportsPP):
242
243
def __init__(
244
self,
@@ -256,6 +260,8 @@ class GPT2LMHeadModel(nn.Module):
260
self.config.hidden_size)
261
self.logits_processor = LogitsProcessor(config.vocab_size)
262
self.sampler = Sampler()
263
+ self.make_empty_intermediate_tensors = (
264
+ self.transformer.make_empty_intermediate_tensors)
265
266
def forward(
267
self,
@@ -264,7 +270,7 @@ class GPT2LMHeadModel(nn.Module):
270
kv_caches: List[torch.Tensor],
271
attn_metadata: AttentionMetadata,
272
intermediate_tensors: Optional[IntermediateTensors] = None,
267
- ) -> torch.Tensor:
273
+ ) -> Union[torch.Tensor, IntermediateTensors]:
274
hidden_states = self.transformer(input_ids, positions, kv_caches,
275
attn_metadata, intermediate_tensors)
276
return hidden_states
@@ -286,16 +292,6 @@ class GPT2LMHeadModel(nn.Module):
292
next_tokens = self.sampler(logits, sampling_metadata)
293
return next_tokens
294
289
- def make_empty_intermediate_tensors(
290
- self, batch_size: int, dtype: torch.dtype,
291
- device: torch.device) -> IntermediateTensors:
292
- return IntermediateTensors({
293
- "hidden_states":
294
- torch.zeros((batch_size, self.config.hidden_size),
295
- dtype=dtype,
296
- device=device),
297
- })
298
-
295
def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
296
params_dict = dict(self.named_parameters(remove_duplicate=False))
297
for name, loaded_weight in weights:
vllm/model_executor/models/gpt_bigcode.py
+36
-18
@@ -18,7 +18,7 @@
18
# See the License for the specific language governing permissions and
19
# limitations under the License.
20
"""Inference-only GPTBigCode model compatible with HuggingFace weights."""
21
-from typing import Iterable, List, Optional, Tuple
21
+from typing import Iterable, List, Optional, Tuple, Union
22
23
import torch
24
from torch import nn
@@ -26,14 +26,13 @@ from transformers import GPTBigCodeConfig
26
27
from vllm.attention import Attention, AttentionMetadata
28
from vllm.config import CacheConfig, LoRAConfig
29
-from vllm.distributed import get_tensor_model_parallel_world_size
29
+from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
30
from vllm.model_executor.layers.activation import get_act_fn
31
from vllm.model_executor.layers.linear import (ColumnParallelLinear,
32
QKVParallelLinear,
33
RowParallelLinear)
34
from vllm.model_executor.layers.logits_processor import LogitsProcessor
35
-from vllm.model_executor.layers.quantization.base_config import (
36
- QuantizationConfig)
35
+from vllm.model_executor.layers.quantization import QuantizationConfig
36
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
37
from vllm.model_executor.layers.vocab_parallel_embedding import (
38
ParallelLMHead, VocabParallelEmbedding)
@@ -41,7 +40,9 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
40
from vllm.model_executor.sampling_metadata import SamplingMetadata
41
from vllm.sequence import IntermediateTensors
42
44
-from .interfaces import SupportsLoRA
43
+from .interfaces import SupportsLoRA, SupportsPP
44
+from .utils import (is_pp_missing_parameter,
45
+ make_empty_intermediate_tensors_factory, make_layers)
46
47
48
class GPTBigCodeAttention(nn.Module):
@@ -194,6 +195,7 @@ class GPTBigCodeModel(nn.Module):
195
cache_config: Optional[CacheConfig] = None,
196
quant_config: Optional[QuantizationConfig] = None,
197
lora_config: Optional[LoRAConfig] = None,
198
+ prefix: str = "",
199
):
200
super().__init__()
201
self.config = config
@@ -207,11 +209,15 @@ class GPTBigCodeModel(nn.Module):
209
self.embed_dim,
210
org_num_embeddings=config.vocab_size)
211
self.wpe = nn.Embedding(config.max_position_embeddings, self.embed_dim)
210
- self.h = nn.ModuleList([
211
- GPTBigCodeBlock(config, cache_config, quant_config)
212
- for _ in range(config.num_hidden_layers)
213
- ])
212
+ self.start_layer, self.end_layer, self.h = make_layers(
213
+ config.num_hidden_layers,
214
+ lambda prefix: GPTBigCodeBlock(config, cache_config, quant_config),
215
+ prefix=f"{prefix}.h",
216
+ )
217
self.ln_f = nn.LayerNorm(self.embed_dim, eps=config.layer_norm_epsilon)
218
+ self.make_empty_intermediate_tensors = (
219
+ make_empty_intermediate_tensors_factory(["hidden_states"],
220
+ config.n_embd))
221
222
def forward(
223
self,
@@ -219,20 +225,28 @@ class GPTBigCodeModel(nn.Module):
225
position_ids: torch.Tensor,
226
kv_caches: List[torch.Tensor],
227
attn_metadata: AttentionMetadata,
222
- ) -> torch.Tensor:
223
- inputs_embeds = self.wte(input_ids)
224
- position_embeds = self.wpe(position_ids)
225
- hidden_states = inputs_embeds + position_embeds
228
+ intermediate_tensors: Optional[IntermediateTensors],
229
+ ) -> Union[torch.Tensor, IntermediateTensors]:
230
+ if get_pp_group().is_first_rank:
231
+ inputs_embeds = self.wte(input_ids)
232
+ position_embeds = self.wpe(position_ids)
233
+ hidden_states = inputs_embeds + position_embeds
234
+ else:
235
+ hidden_states = intermediate_tensors["hidden_states"]
236
227
- for i in range(len(self.h)):
237
+ for i in range(self.start_layer, self.end_layer):
238
layer = self.h[i]
229
- hidden_states = layer(hidden_states, kv_caches[i], attn_metadata)
239
+ hidden_states = layer(hidden_states,
240
+ kv_caches[i - self.start_layer],
241
+ attn_metadata)
242
243
+ if not get_pp_group().is_last_rank:
244
+ return IntermediateTensors({"hidden_states": hidden_states})
245
hidden_states = self.ln_f(hidden_states)
246
return hidden_states
247
248
235
-class GPTBigCodeForCausalLM(nn.Module, SupportsLoRA):
249
+class GPTBigCodeForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
250
packed_modules_mapping = {"c_attn": ["c_attn"]}
251
252
supported_lora_modules = ["c_fc", "c_proj", "wte", "c_attn"]
@@ -272,6 +286,8 @@ class GPTBigCodeForCausalLM(nn.Module, SupportsLoRA):
286
self.logits_processor = LogitsProcessor(self.unpadded_vocab_size,
287
config.vocab_size)
288
self.sampler = Sampler()
289
+ self.make_empty_intermediate_tensors = (
290
+ self.transformer.make_empty_intermediate_tensors)
291
292
def forward(
293
self,
@@ -280,9 +296,9 @@ class GPTBigCodeForCausalLM(nn.Module, SupportsLoRA):
296
kv_caches: List[torch.Tensor],
297
attn_metadata: AttentionMetadata,
298
intermediate_tensors: Optional[IntermediateTensors] = None,
283
- ) -> torch.Tensor:
299
+ ) -> Union[torch.Tensor, IntermediateTensors]:
300
hidden_states = self.transformer(input_ids, positions, kv_caches,
285
- attn_metadata)
301
+ attn_metadata, intermediate_tensors)
302
return hidden_states
303
304
def compute_logits(
@@ -311,6 +327,8 @@ class GPTBigCodeForCausalLM(nn.Module, SupportsLoRA):
327
# Skip attention mask.
328
# NOTE: "c_attn.bias" should not be skipped.
329
continue
330
+ if is_pp_missing_parameter(name, self):
331
+ continue
332
param = params_dict[name]
333
weight_loader = getattr(param, "weight_loader",
334
default_weight_loader)
vllm/model_executor/models/gpt_j.py
+35
-15
@@ -16,7 +16,7 @@
16
# See the License for the specific language governing permissions and
17
# limitations under the License.
18
"""Inference-only GPT-J model compatible with HuggingFace weights."""
19
-from typing import Iterable, List, Optional, Tuple
19
+from typing import Iterable, List, Optional, Tuple, Union
20
21
import torch
22
from torch import nn
@@ -24,14 +24,13 @@ from transformers import GPTJConfig
24
25
from vllm.attention import Attention, AttentionMetadata
26
from vllm.config import CacheConfig
27
-from vllm.distributed import get_tensor_model_parallel_world_size
27
+from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
28
from vllm.model_executor.layers.activation import get_act_fn
29
from vllm.model_executor.layers.linear import (ColumnParallelLinear,
30
QKVParallelLinear,
31
RowParallelLinear)
32
from vllm.model_executor.layers.logits_processor import LogitsProcessor
33
-from vllm.model_executor.layers.quantization.base_config import (
34
- QuantizationConfig)
33
+from vllm.model_executor.layers.quantization import QuantizationConfig
34
from vllm.model_executor.layers.rotary_embedding import get_rope
35
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
36
from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -40,6 +39,10 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
39
from vllm.model_executor.sampling_metadata import SamplingMetadata
40
from vllm.sequence import IntermediateTensors
41
42
+from .interfaces import SupportsPP
43
+from .utils import (is_pp_missing_parameter,
44
+ make_empty_intermediate_tensors_factory, make_layers)
45
+
46
47
class GPTJAttention(nn.Module):
48
@@ -178,6 +181,7 @@ class GPTJModel(nn.Module):
181
config: GPTJConfig,
182
cache_config: Optional[CacheConfig] = None,
183
quant_config: Optional[QuantizationConfig] = None,
184
+ prefix: str = "",
185
):
186
super().__init__()
187
self.config = config
@@ -186,11 +190,15 @@ class GPTJModel(nn.Module):
190
config.vocab_size,
191
self.embed_dim,
192
)
189
- self.h = nn.ModuleList([
190
- GPTJBlock(config, cache_config, quant_config)
191
- for _ in range(config.n_layer)
192
- ])
193
+ self.start_layer, self.end_layer, self.h = make_layers(
194
+ config.n_layer,
195
+ lambda prefix: GPTJBlock(config, cache_config, quant_config),
196
+ prefix=f"{prefix}.h",
197
+ )
198
self.ln_f = nn.LayerNorm(self.embed_dim, eps=config.layer_norm_epsilon)
199
+ self.make_empty_intermediate_tensors = (
200
+ make_empty_intermediate_tensors_factory(["hidden_states"],
201
+ config.n_embd))
202
203
def forward(
204
self,
@@ -198,21 +206,27 @@ class GPTJModel(nn.Module):
206
position_ids: torch.Tensor,
207
kv_caches: List[torch.Tensor],
208
attn_metadata: AttentionMetadata,
201
- ) -> torch.Tensor:
202
- hidden_states = self.wte(input_ids)
203
- for i in range(len(self.h)):
209
+ intermediate_tensors: Optional[IntermediateTensors],
210
+ ) -> Union[torch.Tensor, IntermediateTensors]:
211
+ if get_pp_group().is_first_rank:
212
+ hidden_states = self.wte(input_ids)
213
+ else:
214
+ hidden_states = intermediate_tensors["hidden_states"]
215
+ for i in range(self.start_layer, self.end_layer):
216
layer = self.h[i]
217
hidden_states = layer(
218
position_ids,
219
hidden_states,
208
- kv_caches[i],
220
+ kv_caches[i - self.start_layer],
221
attn_metadata,
222
)
223
+ if not get_pp_group().is_last_rank:
224
+ return IntermediateTensors({"hidden_states": hidden_states})
225
hidden_states = self.ln_f(hidden_states)
226
return hidden_states
227
228
215
-class GPTJForCausalLM(nn.Module):
229
+class GPTJForCausalLM(nn.Module, SupportsPP):
230
231
def __init__(
232
self,
@@ -233,6 +247,8 @@ class GPTJForCausalLM(nn.Module):
247
)
248
self.logits_processor = LogitsProcessor(config.vocab_size)
249
self.sampler = Sampler()
250
+ self.make_empty_intermediate_tensors = (
251
+ self.transformer.make_empty_intermediate_tensors)
252
253
def forward(
254
self,
@@ -241,9 +257,9 @@ class GPTJForCausalLM(nn.Module):
257
kv_caches: List[torch.Tensor],
258
attn_metadata: AttentionMetadata,
259
intermediate_tensors: Optional[IntermediateTensors] = None,
244
- ) -> torch.Tensor:
260
+ ) -> Union[torch.Tensor, IntermediateTensors]:
261
hidden_states = self.transformer(input_ids, positions, kv_caches,
246
- attn_metadata)
262
+ attn_metadata, intermediate_tensors)
263
return hidden_states
264
265
def compute_logits(
@@ -283,6 +299,8 @@ class GPTJForCausalLM(nn.Module):
299
# Skip loading extra bias for GPTQ models.
300
if name.endswith(".bias") and name not in params_dict:
301
continue
302
+ if is_pp_missing_parameter(name, self):
303
+ continue
304
param = params_dict[name]
305
weight_loader = param.weight_loader
306
weight_loader(param, loaded_weight, shard_id)
@@ -291,6 +309,8 @@ class GPTJForCausalLM(nn.Module):
309
# Skip loading extra bias for GPTQ models.
310
if name.endswith(".bias") and name not in params_dict:
311
continue
312
+ if is_pp_missing_parameter(name, self):
313
+ continue
314
param = params_dict[name]
315
weight_loader = getattr(param, "weight_loader",
316
default_weight_loader)
vllm/model_executor/models/gpt_neox.py
+33
-15
@@ -16,7 +16,7 @@
16
# See the License for the specific language governing permissions and
17
# limitations under the License.
18
"""Inference-only GPT-NeoX model compatible with HuggingFace weights."""
19
-from typing import Iterable, List, Optional, Tuple
19
+from typing import Iterable, List, Optional, Tuple, Union
20
21
import torch
22
from torch import nn
@@ -24,14 +24,13 @@ from transformers import GPTNeoXConfig
24
25
from vllm.attention import Attention, AttentionMetadata
26
from vllm.config import CacheConfig
27
-from vllm.distributed import get_tensor_model_parallel_world_size
27
+from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
28
from vllm.model_executor.layers.activation import get_act_fn
29
from vllm.model_executor.layers.linear import (ColumnParallelLinear,
30
QKVParallelLinear,
31
RowParallelLinear)
32
from vllm.model_executor.layers.logits_processor import LogitsProcessor
33
-from vllm.model_executor.layers.quantization.base_config import (
34
- QuantizationConfig)
33
+from vllm.model_executor.layers.quantization import QuantizationConfig
34
from vllm.model_executor.layers.rotary_embedding import get_rope
35
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
36
from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -40,6 +39,10 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
39
from vllm.model_executor.sampling_metadata import SamplingMetadata
40
from vllm.sequence import IntermediateTensors
41
42
+from .interfaces import SupportsPP
43
+from .utils import (is_pp_missing_parameter,
44
+ make_empty_intermediate_tensors_factory, make_layers)
45
+
46
47
class GPTNeoXAttention(nn.Module):
48
@@ -191,6 +194,7 @@ class GPTNeoXModel(nn.Module):
194
config: GPTNeoXConfig,
195
cache_config: Optional[CacheConfig] = None,
196
quant_config: Optional[QuantizationConfig] = None,
197
+ prefix: str = "",
198
):
199
super().__init__()
200
self.config = config
@@ -199,12 +203,16 @@ class GPTNeoXModel(nn.Module):
203
config.vocab_size,
204
config.hidden_size,
205
)
202
- self.layers = nn.ModuleList([
203
- GPTNeoXLayer(config, cache_config, quant_config)
204
- for _ in range(config.num_hidden_layers)
205
- ])
206
+ self.start_layer, self.end_layer, self.layers = make_layers(
207
+ config.num_hidden_layers,
208
+ lambda prefix: GPTNeoXLayer(config, cache_config, quant_config),
209
+ prefix=f"{prefix}.layers",
210
+ )
211
self.final_layer_norm = nn.LayerNorm(config.hidden_size,
212
eps=config.layer_norm_eps)
213
+ self.make_empty_intermediate_tensors = (
214
+ make_empty_intermediate_tensors_factory(["hidden_states"],
215
+ config.hidden_size))
216
217
def forward(
218
self,
@@ -212,21 +220,27 @@ class GPTNeoXModel(nn.Module):
220
position_ids: torch.Tensor,
221
kv_caches: List[torch.Tensor],
222
attn_metadata: AttentionMetadata,
215
- ) -> torch.Tensor:
216
- hidden_states = self.embed_in(input_ids)
217
- for i in range(len(self.layers)):
223
+ intermediate_tensors: Optional[IntermediateTensors],
224
+ ) -> Union[torch.Tensor, IntermediateTensors]:
225
+ if get_pp_group().is_first_rank:
226
+ hidden_states = self.embed_in(input_ids)
227
+ else:
228
+ hidden_states = intermediate_tensors["hidden_states"]
229
+ for i in range(self.start_layer, self.end_layer):
230
layer = self.layers[i]
231
hidden_states = layer(
232
position_ids,
233
hidden_states,
222
- kv_caches[i],
234
+ kv_caches[i - self.start_layer],
235
attn_metadata,
236
)
237
+ if not get_pp_group().is_last_rank:
238
+ return IntermediateTensors({"hidden_states": hidden_states})
239
hidden_states = self.final_layer_norm(hidden_states)
240
return hidden_states
241
242
229
-class GPTNeoXForCausalLM(nn.Module):
243
+class GPTNeoXForCausalLM(nn.Module, SupportsPP):
244
245
def __init__(
246
self,
@@ -247,6 +261,8 @@ class GPTNeoXForCausalLM(nn.Module):
261
self.embed_out.weight = self.gpt_neox.embed_in.weight
262
self.logits_processor = LogitsProcessor(config.vocab_size)
263
self.sampler = Sampler()
264
+ self.make_empty_intermediate_tensors = (
265
+ self.gpt_neox.make_empty_intermediate_tensors)
266
267
def forward(
268
self,
@@ -255,9 +271,9 @@ class GPTNeoXForCausalLM(nn.Module):
271
kv_caches: List[torch.Tensor],
272
attn_metadata: AttentionMetadata,
273
intermediate_tensors: Optional[IntermediateTensors] = None,
258
- ) -> torch.Tensor:
274
+ ) -> Union[torch.Tensor, IntermediateTensors]:
275
hidden_states = self.gpt_neox(input_ids, positions, kv_caches,
260
- attn_metadata)
276
+ attn_metadata, intermediate_tensors)
277
return hidden_states
278
279
def compute_logits(
@@ -288,6 +304,8 @@ class GPTNeoXForCausalLM(nn.Module):
304
# Models trained using OpenRLHF may include
305
# these tensors in the checkpoint. Skip them.
306
continue
307
+ if is_pp_missing_parameter(name, self):
308
+ continue
309
param = params_dict[name]
310
311
if "query_key_value" in name:
vllm/model_executor/models/granite.py
+4
-4
@@ -51,7 +51,7 @@ from vllm.model_executor.sampling_metadata import SamplingMetadata
51
from vllm.sequence import IntermediateTensors
52
from vllm.utils import is_hip
53
54
-from .interfaces import SupportsLoRA
54
+from .interfaces import SupportsLoRA, SupportsPP
55
from .utils import PPMissingLayer, is_pp_missing_parameter, make_layers
56
57
@@ -311,13 +311,13 @@ class GraniteModel(nn.Module):
311
else:
312
hidden_states = self.get_input_embeddings(input_ids)
313
residual = None
314
+
315
+ hidden_states *= self.config.embedding_multiplier
316
else:
317
assert intermediate_tensors is not None
318
hidden_states = intermediate_tensors["hidden_states"]
319
residual = intermediate_tensors["residual"]
320
319
- hidden_states *= self.config.embedding_multiplier
320
-
321
for i in range(self.start_layer, self.end_layer):
322
layer = self.layers[i]
323
hidden_states = layer(
@@ -337,7 +337,7 @@ class GraniteModel(nn.Module):
337
return hidden_states
338
339
340
-class GraniteForCausalLM(nn.Module, SupportsLoRA):
340
+class GraniteForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
341
packed_modules_mapping = {
342
"qkv_proj": [
343
"q_proj",
vllm/model_executor/models/granitemoe.py
+2
-2
@@ -46,7 +46,7 @@ from vllm.model_executor.sampling_metadata import SamplingMetadata
46
from vllm.sequence import IntermediateTensors
47
48
from . import mixtral
49
-from .interfaces import SupportsLoRA
49
+from .interfaces import SupportsLoRA, SupportsPP
50
from .utils import make_layers
51
52
@@ -307,7 +307,7 @@ class GraniteMoeModel(nn.Module):
307
return hidden_states
308
309
310
-class GraniteMoeForCausalLM(nn.Module, SupportsLoRA):
310
+class GraniteMoeForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
311
fall_back_to_pt_during_load = False
312
313
packed_modules_mapping = {
vllm/model_executor/models/interfaces.py
+141
-9
@@ -1,11 +1,17 @@
1
-from typing import (ClassVar, Dict, List, Literal, Optional, Protocol, Type,
2
- Union, overload, runtime_checkable)
1
+import inspect
2
+from typing import (TYPE_CHECKING, ClassVar, Dict, List, Literal, Optional,
3
+ Protocol, Type, Union, overload, runtime_checkable)
4
5
+import torch
6
from typing_extensions import TypeIs
7
6
-from vllm.config import LoRAConfig, MultiModalConfig, SchedulerConfig
8
from vllm.logger import init_logger
9
10
+if TYPE_CHECKING:
11
+ from vllm.attention import AttentionMetadata
12
+ from vllm.config import LoRAConfig, MultiModalConfig, SchedulerConfig
13
+ from vllm.sequence import IntermediateTensors
14
+
15
logger = init_logger(__name__)
16
17
@@ -22,7 +28,7 @@ class SupportsMultiModal(Protocol):
28
MRO of your model class.
29
"""
30
25
- def __init__(self, *, multimodal_config: MultiModalConfig) -> None:
31
+ def __init__(self, *, multimodal_config: "MultiModalConfig") -> None:
32
...
33
34
@@ -32,7 +38,7 @@ class SupportsMultiModal(Protocol):
38
class _SupportsMultiModalType(Protocol):
39
supports_multimodal: Literal[True]
40
35
- def __call__(self, *, multimodal_config: MultiModalConfig) -> None:
41
+ def __call__(self, *, multimodal_config: "MultiModalConfig") -> None:
42
...
43
44
@@ -75,7 +81,7 @@ class SupportsLoRA(Protocol):
81
embedding_padding_modules: ClassVar[List[str]]
82
83
# lora_config is None when LoRA is not enabled
78
- def __init__(self, *, lora_config: Optional[LoRAConfig] = None) -> None:
84
+ def __init__(self, *, lora_config: Optional["LoRAConfig"] = None) -> None:
85
...
86
87
@@ -90,7 +96,7 @@ class _SupportsLoRAType(Protocol):
96
embedding_modules: Dict[str, str]
97
embedding_padding_modules: List[str]
98
93
- def __call__(self, *, lora_config: Optional[LoRAConfig] = None) -> None:
99
+ def __call__(self, *, lora_config: Optional["LoRAConfig"] = None) -> None:
100
...
101
102
@@ -145,6 +151,132 @@ def _supports_lora(
151
return isinstance(model, SupportsLoRA)
152
153
154
+@runtime_checkable
155
+class SupportsPP(Protocol):
156
+ """The interface required for all models that support pipeline parallel."""
157
+
158
+ supports_pp: ClassVar[Literal[True]] = True
159
+ """
160
+ A flag that indicates this model supports pipeline parallel.
161
+
162
+ Note:
163
+ There is no need to redefine this flag if this class is in the
164
+ MRO of your model class.
165
+ """
166
+
167
+ def make_empty_intermediate_tensors(
168
+ self,
169
+ batch_size: int,
170
+ dtype: torch.dtype,
171
+ device: torch.device,
172
+ ) -> "IntermediateTensors":
173
+ """Called when PP rank > 0 for profiling purposes."""
174
+ ...
175
+
176
+ def forward(
177
+ self,
178
+ input_ids: torch.Tensor,
179
+ position_ids: torch.Tensor,
180
+ kv_caches: List[torch.Tensor],
181
+ attn_metadata: "AttentionMetadata",
182
+ intermediate_tensors: Optional["IntermediateTensors"],
183
+ ) -> Union[torch.Tensor, "IntermediateTensors"]:
184
+ """
185
+ Accept :class:`IntermediateTensors` when PP rank > 0.
186
+
187
+ Return :class:`IntermediateTensors` only for the last PP rank.
188
+ """
189
+ ...
190
+
191
+
192
+# We can't use runtime_checkable with ClassVar for issubclass checks
193
+# so we need to treat the class as an instance and use isinstance instead
194
+@runtime_checkable
195
+class _SupportsPPType(Protocol):
196
+ supports_pp: Literal[True]
197
+
198
+ def make_empty_intermediate_tensors(
199
+ self,
200
+ batch_size: int,
201
+ dtype: torch.dtype,
202
+ device: torch.device,
203
+ ) -> "IntermediateTensors":
204
+ ...
205
+
206
+ def forward(
207
+ self,
208
+ input_ids: torch.Tensor,
209
+ position_ids: torch.Tensor,
210
+ kv_caches: List[torch.Tensor],
211
+ attn_metadata: "AttentionMetadata",
212
+ intermediate_tensors: Optional["IntermediateTensors"],
213
+ ) -> Union[torch.Tensor, "IntermediateTensors"]:
214
+ ...
215
+
216
+
217
+@overload
218
+def supports_pp(model: Type[object]) -> TypeIs[Type[SupportsPP]]:
219
+ ...
220
+
221
+
222
+@overload
223
+def supports_pp(model: object) -> TypeIs[SupportsPP]:
224
+ ...
225
+
226
+
227
+def supports_pp(
228
+ model: Union[Type[object], object],
229
+) -> Union[bool, TypeIs[Type[SupportsPP]], TypeIs[SupportsPP]]:
230
+ supports_attributes = _supports_pp_attributes(model)
231
+ supports_inspect = _supports_pp_inspect(model)
232
+
233
+ if supports_attributes and not supports_inspect:
234
+ logger.warning(
235
+ "The model (%s) sets `supports_pp=True`, but does not accept "
236
+ "`intermediate_tensors` in its `forward` method", model)
237
+
238
+ if not supports_attributes:
239
+ pp_attrs = ("make_empty_intermediate_tensors", )
240
+ missing_attrs = tuple(attr for attr in pp_attrs
241
+ if not hasattr(model, attr))
242
+
243
+ if getattr(model, "supports_pp", False):
244
+ if missing_attrs:
245
+ logger.warning(
246
+ "The model (%s) sets `supports_pp=True`, "
247
+ "but is missing PP-specific attributes: %s",
248
+ model,
249
+ missing_attrs,
250
+ )
251
+ else:
252
+ if not missing_attrs:
253
+ logger.warning(
254
+ "The model (%s) contains all PP-specific attributes, "
255
+ "but does not set `supports_pp=True`.", model)
256
+
257
+ return supports_attributes and supports_inspect
258
+
259
+
260
+def _supports_pp_attributes(
261
+ model: Union[Type[object], object],
262
+) -> Union[bool, TypeIs[Type[SupportsPP]], TypeIs[SupportsPP]]:
263
+ if isinstance(model, type):
264
+ return isinstance(model, _SupportsPPType)
265
+
266
+ return isinstance(model, SupportsPP)
267
+
268
+
269
+def _supports_pp_inspect(
270
+ model: Union[Type[object], object],
271
+) -> Union[bool, TypeIs[Type[SupportsPP]], TypeIs[SupportsPP]]:
272
+ model_forward = getattr(model, "forward", None)
273
+ if not callable(model_forward):
274
+ return False
275
+
276
+ forward_params = inspect.signature(model_forward).parameters
277
+ return "intermediate_tensors" in forward_params
278
+
279
+
280
@runtime_checkable
281
class HasInnerState(Protocol):
282
"""The interface required for all models that has inner state."""
@@ -158,7 +290,7 @@ class HasInnerState(Protocol):
290
291
def __init__(self,
292
*,
161
- scheduler_config: Optional[SchedulerConfig] = None) -> None:
293
+ scheduler_config: Optional["SchedulerConfig"] = None) -> None:
294
...
295
296
@@ -168,7 +300,7 @@ class _HasInnerStateType(Protocol):
300
301
def __init__(self,
302
*,
171
- scheduler_config: Optional[SchedulerConfig] = None) -> None:
303
+ scheduler_config: Optional["SchedulerConfig"] = None) -> None:
304
...
305
306
vllm/model_executor/models/internlm2.py
+5
-5
@@ -18,8 +18,7 @@ from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
18
QKVParallelLinear,
19
RowParallelLinear)
20
from vllm.model_executor.layers.logits_processor import LogitsProcessor
21
-from vllm.model_executor.layers.quantization.base_config import (
22
- QuantizationConfig)
21
+from vllm.model_executor.layers.quantization import QuantizationConfig
22
from vllm.model_executor.layers.rotary_embedding import get_rope
23
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
24
from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -28,6 +27,7 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
27
from vllm.model_executor.sampling_metadata import SamplingMetadata
28
from vllm.sequence import IntermediateTensors
29
30
+from .interfaces import SupportsPP
31
from .utils import (is_pp_missing_parameter,
32
make_empty_intermediate_tensors_factory, make_layers)
33
@@ -266,7 +266,7 @@ class InternLM2Model(nn.Module):
266
positions: torch.Tensor,
267
kv_caches: List[torch.Tensor],
268
attn_metadata: AttentionMetadata,
269
- intermediate_tensors: IntermediateTensors = None,
269
+ intermediate_tensors: Optional[IntermediateTensors] = None,
270
inputs_embeds: Optional[torch.Tensor] = None,
271
) -> Union[torch.Tensor, IntermediateTensors]:
272
if get_pp_group().is_first_rank:
@@ -297,7 +297,7 @@ class InternLM2Model(nn.Module):
297
return hidden_states
298
299
300
-class InternLM2ForCausalLM(nn.Module):
300
+class InternLM2ForCausalLM(nn.Module, SupportsPP):
301
302
def __init__(
303
self,
@@ -325,7 +325,7 @@ class InternLM2ForCausalLM(nn.Module):
325
positions: torch.Tensor,
326
kv_caches: List[torch.Tensor],
327
attn_metadata: AttentionMetadata,
328
- intermediate_tensors: IntermediateTensors,
328
+ intermediate_tensors: Optional[IntermediateTensors],
329
) -> torch.Tensor:
330
hidden_states = self.model(input_ids, positions, kv_caches,
331
attn_metadata, intermediate_tensors)
vllm/model_executor/models/internvl.py
+27
-22
@@ -5,9 +5,9 @@
5
# Licensed under The MIT License [see LICENSE for details]
6
# --------------------------------------------------------
7
import re
8
-from functools import partial
9
-from typing import (Any, Dict, Iterable, List, Literal, Mapping, Optional,
10
- Tuple, TypedDict, Union)
8
+from functools import cached_property, partial
9
+from typing import (Iterable, List, Literal, Mapping, Optional, Tuple,
10
+ TypedDict, Union)
11
12
import torch
13
import torch.nn as nn
@@ -17,7 +17,6 @@ from transformers import PretrainedConfig
17
18
from vllm.attention import AttentionMetadata
19
from vllm.config import CacheConfig, MultiModalConfig
20
-from vllm.distributed import get_pp_group
20
from vllm.inputs import INPUT_REGISTRY, InputContext, LLMInputs
21
from vllm.model_executor.layers.quantization import QuantizationConfig
22
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
@@ -32,7 +31,7 @@ from vllm.utils import is_list_of
31
32
from .clip import (dummy_image_for_clip, dummy_seq_data_for_clip,
33
get_clip_num_patches)
35
-from .interfaces import SupportsMultiModal
34
+from .interfaces import SupportsMultiModal, SupportsPP
35
from .utils import (flatten_bn, group_weights_with_prefix,
36
init_vllm_registered_model, merge_multimodal_embeddings)
37
@@ -123,7 +122,7 @@ def calculate_num_blocks(orig_width: int, orig_height: int, min_num: int,
122
return blocks, target_width, target_height
123
124
126
-def calculate_num_blocks_wrapper(hf_config: Dict[str, Any],
125
+def calculate_num_blocks_wrapper(hf_config: PretrainedConfig,
126
max_dynamic_patch: Optional[int] = None):
127
if max_dynamic_patch is None:
128
max_dynamic_patch = hf_config.max_dynamic_patch
@@ -183,7 +182,7 @@ def image_to_pixel_values(image: Image.Image, input_size: int, min_num: int,
182
return pixel_values
183
184
186
-def image_to_pixel_values_wrapper(hf_config: Dict[str, Any],
185
+def image_to_pixel_values_wrapper(hf_config: PretrainedConfig,
186
max_dynamic_patch: Optional[int] = None):
187
image_size = hf_config.vision_config.image_size
188
min_num = hf_config.min_dynamic_patch
@@ -197,7 +196,7 @@ def image_to_pixel_values_wrapper(hf_config: Dict[str, Any],
196
use_thumbnail=use_thumbnail)
197
198
200
-def get_internvl_num_patches(hf_config: Dict[str, Any]):
199
+def get_internvl_num_patches(hf_config: PretrainedConfig):
200
vision_config = hf_config.vision_config
201
downsample_ratio = hf_config.downsample_ratio
202
image_size = vision_config.image_size
@@ -362,7 +361,7 @@ def dummy_data_for_internvl(ctx: InputContext,
361
@MULTIMODAL_REGISTRY.register_max_image_tokens(get_max_internvl_image_tokens)
362
@INPUT_REGISTRY.register_dummy_data(dummy_data_for_internvl)
363
@INPUT_REGISTRY.register_input_processor(input_processor_for_internvl)
365
-class InternVLChatModel(nn.Module, SupportsMultiModal):
364
+class InternVLChatModel(nn.Module, SupportsMultiModal, SupportsPP):
365
366
def __init__(self,
367
config: PretrainedConfig,
@@ -408,10 +407,12 @@ class InternVLChatModel(nn.Module, SupportsMultiModal):
407
self.make_empty_intermediate_tensors = (
408
self.language_model.make_empty_intermediate_tensors)
409
410
+ @cached_property
411
+ def sampler(self):
412
if hasattr(self.language_model, "sampler"):
412
- self.sampler = self.language_model.sampler
413
- else:
414
- self.sampler = Sampler()
413
+ return self.language_model.sampler
414
+
415
+ return Sampler()
416
417
def pixel_shuffle(self, x, scale_factor=0.5):
418
n, w, h, c = x.size()
@@ -515,18 +516,22 @@ class InternVLChatModel(nn.Module, SupportsMultiModal):
516
attn_metadata: AttentionMetadata,
517
intermediate_tensors: Optional[IntermediateTensors] = None,
518
**kwargs: object,
518
- ) -> SamplerOutput:
519
- image_input = self._parse_and_validate_image_input(**kwargs)
520
- if image_input is not None and get_pp_group().is_first_rank:
521
- inputs_embeds = self.language_model.model.get_input_embeddings(
522
- input_ids)
523
- vision_embeddings = self._process_image_input(image_input)
524
- inputs_embeds = merge_multimodal_embeddings(
525
- input_ids, inputs_embeds, vision_embeddings,
526
- self.img_context_token_id)
519
+ ) -> Union[SamplerOutput, IntermediateTensors]:
520
+ if intermediate_tensors is not None:
521
input_ids = None
528
- else:
522
inputs_embeds = None
523
+ else:
524
+ image_input = self._parse_and_validate_image_input(**kwargs)
525
+ if image_input is not None:
526
+ inputs_embeds = self.language_model.model.get_input_embeddings(
527
+ input_ids)
528
+ vision_embeddings = self._process_image_input(image_input)
529
+ inputs_embeds = merge_multimodal_embeddings(
530
+ input_ids, inputs_embeds, vision_embeddings,
531
+ self.img_context_token_id)
532
+ input_ids = None
533
+ else:
534
+ inputs_embeds = None
535
536
hidden_states = self.language_model.model(input_ids,
537
positions,
vllm/model_executor/models/jais.py
+10
-14
@@ -33,8 +33,7 @@ from vllm.model_executor.layers.linear import (ColumnParallelLinear,
33
QKVParallelLinear,
34
RowParallelLinear)
35
from vllm.model_executor.layers.logits_processor import LogitsProcessor
36
-from vllm.model_executor.layers.quantization.base_config import (
37
- QuantizationConfig)
36
+from vllm.model_executor.layers.quantization import QuantizationConfig
37
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
38
from vllm.model_executor.layers.vocab_parallel_embedding import (
39
ParallelLMHead, VocabParallelEmbedding)
@@ -43,7 +42,9 @@ from vllm.model_executor.sampling_metadata import SamplingMetadata
42
from vllm.sequence import IntermediateTensors
43
from vllm.transformers_utils.configs import JAISConfig
44
46
-from .utils import is_pp_missing_parameter, make_layers
45
+from .interfaces import SupportsPP
46
+from .utils import (is_pp_missing_parameter,
47
+ make_empty_intermediate_tensors_factory, make_layers)
48
49
50
class SwiGLUActivation(nn.Module):
@@ -244,6 +245,9 @@ class JAISModel(nn.Module):
245
)
246
247
self.ln_f = nn.LayerNorm(self.embed_dim, eps=config.layer_norm_epsilon)
248
+ self.make_empty_intermediate_tensors = (
249
+ make_empty_intermediate_tensors_factory(["hidden_states"],
250
+ config.n_embd))
251
252
def forward(
253
self,
@@ -279,7 +283,7 @@ class JAISModel(nn.Module):
283
return hidden_states
284
285
282
-class JAISLMHeadModel(nn.Module):
286
+class JAISLMHeadModel(nn.Module, SupportsPP):
287
288
def __init__(
289
self,
@@ -304,6 +308,8 @@ class JAISLMHeadModel(nn.Module):
308
self.logits_processor = LogitsProcessor(vocab_size=config.vocab_size,
309
scale=self.output_logits_scale)
310
self.sampler = Sampler()
311
+ self.make_empty_intermediate_tensors = (
312
+ self.transformer.make_empty_intermediate_tensors)
313
314
def forward(
315
self,
@@ -326,16 +332,6 @@ class JAISLMHeadModel(nn.Module):
332
sampling_metadata)
333
return logits
334
329
- def make_empty_intermediate_tensors(
330
- self, batch_size: int, dtype: torch.dtype,
331
- device: torch.device) -> IntermediateTensors:
332
- return IntermediateTensors({
333
- "hidden_states":
334
- torch.zeros((batch_size, self.config.hidden_size),
335
- dtype=dtype,
336
- device=device),
337
- })
338
-
335
def sample(
336
self,
337
logits: torch.Tensor,
vllm/model_executor/models/llama.py
+49
-65
@@ -37,8 +37,7 @@ from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
37
QKVParallelLinear,
38
RowParallelLinear)
39
from vllm.model_executor.layers.logits_processor import LogitsProcessor
40
-from vllm.model_executor.layers.quantization.base_config import (
41
- QuantizationConfig)
40
+from vllm.model_executor.layers.quantization import QuantizationConfig
41
from vllm.model_executor.layers.quantization.compressed_tensors.utils import (
42
get_compressed_tensors_cache_scale)
43
from vllm.model_executor.layers.rotary_embedding import get_rope
@@ -51,8 +50,9 @@ from vllm.model_executor.sampling_metadata import SamplingMetadata
50
from vllm.sequence import IntermediateTensors
51
from vllm.utils import is_hip
52
54
-from .interfaces import SupportsLoRA
55
-from .utils import PPMissingLayer, is_pp_missing_parameter, make_layers
53
+from .interfaces import SupportsLoRA, SupportsPP
54
+from .utils import (PPMissingLayer, is_pp_missing_parameter,
55
+ make_empty_intermediate_tensors_factory, make_layers)
56
57
58
class LlamaMLP(nn.Module):
@@ -72,12 +72,15 @@ class LlamaMLP(nn.Module):
72
output_sizes=[intermediate_size] * 2,
73
bias=bias,
74
quant_config=quant_config,
75
- prefix=f"{prefix}.gate_up_proj")
76
- self.down_proj = RowParallelLinear(input_size=intermediate_size,
77
- output_size=hidden_size,
78
- bias=bias,
79
- quant_config=quant_config,
80
- prefix=f"{prefix}.down_proj")
75
+ prefix=f"{prefix}.gate_up_proj",
76
+ )
77
+ self.down_proj = RowParallelLinear(
78
+ input_size=intermediate_size,
79
+ output_size=hidden_size,
80
+ bias=bias,
81
+ quant_config=quant_config,
82
+ prefix=f"{prefix}.down_proj",
83
+ )
84
if hidden_act != "silu":
85
raise ValueError(f"Unsupported activation: {hidden_act}. "
86
"Only silu is supported for now.")
@@ -161,12 +164,14 @@ class LlamaAttention(nn.Module):
164
rope_scaling=rope_scaling,
165
is_neox_style=is_neox_style,
166
)
164
- self.attn = Attention(self.num_heads,
165
- self.head_dim,
166
- self.scaling,
167
- num_kv_heads=self.num_kv_heads,
168
- cache_config=cache_config,
169
- quant_config=quant_config)
167
+ self.attn = Attention(
168
+ self.num_heads,
169
+ self.head_dim,
170
+ self.scaling,
171
+ num_kv_heads=self.num_kv_heads,
172
+ cache_config=cache_config,
173
+ quant_config=quant_config,
174
+ )
175
176
def forward(
177
self,
@@ -248,12 +253,10 @@ class LlamaDecoderLayer(nn.Module):
253
else:
254
hidden_states, residual = self.input_layernorm(
255
hidden_states, residual)
251
- hidden_states = self.self_attn(
252
- positions=positions,
253
- hidden_states=hidden_states,
254
- kv_cache=kv_cache,
255
- attn_metadata=attn_metadata,
256
- )
256
+ hidden_states = self.self_attn(positions=positions,
257
+ hidden_states=hidden_states,
258
+ kv_cache=kv_cache,
259
+ attn_metadata=attn_metadata)
260
261
# Fully Connected
262
hidden_states, residual = self.post_attention_layernorm(
@@ -295,12 +298,17 @@ class LlamaModel(nn.Module):
298
cache_config=cache_config,
299
quant_config=quant_config,
300
prefix=prefix),
298
- prefix=f"{prefix}.layers")
301
+ prefix=f"{prefix}.layers",
302
+ )
303
if get_pp_group().is_last_rank:
304
self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
305
else:
306
self.norm = PPMissingLayer()
307
308
+ self.make_empty_intermediate_tensors = (
309
+ make_empty_intermediate_tensors_factory(
310
+ ["hidden_states", "residual"], config.hidden_size))
311
+
312
def get_input_embeddings(self, input_ids: torch.Tensor) -> torch.Tensor:
313
return self.embed_tokens(input_ids)
314
@@ -326,13 +334,9 @@ class LlamaModel(nn.Module):
334
335
for i in range(self.start_layer, self.end_layer):
336
layer = self.layers[i]
329
- hidden_states, residual = layer(
330
- positions,
331
- hidden_states,
332
- kv_caches[i - self.start_layer],
333
- attn_metadata,
334
- residual,
335
- )
337
+ hidden_states, residual = layer(positions, hidden_states,
338
+ kv_caches[i - self.start_layer],
339
+ attn_metadata, residual)
340
341
if not get_pp_group().is_last_rank:
342
return IntermediateTensors({
@@ -344,17 +348,10 @@ class LlamaModel(nn.Module):
348
return hidden_states
349
350
347
-class LlamaForCausalLM(nn.Module, SupportsLoRA):
351
+class LlamaForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
352
packed_modules_mapping = {
349
- "qkv_proj": [
350
- "q_proj",
351
- "k_proj",
352
- "v_proj",
353
- ],
354
- "gate_up_proj": [
355
- "gate_proj",
356
- "up_proj",
357
- ],
353
+ "qkv_proj": ["q_proj", "k_proj", "v_proj"],
354
+ "gate_up_proj": ["gate_proj", "up_proj"]
355
}
356
357
# LoRA specific attributes
@@ -364,7 +361,7 @@ class LlamaForCausalLM(nn.Module, SupportsLoRA):
361
]
362
embedding_modules = {
363
"embed_tokens": "input_embeddings",
367
- "lm_head": "output_embeddings",
364
+ "lm_head": "output_embeddings"
365
}
366
embedding_padding_modules = ["lm_head"]
367
bitsandbytes_stacked_params_mapping = {
@@ -420,10 +417,12 @@ class LlamaForCausalLM(nn.Module, SupportsLoRA):
417
self.unpadded_vocab_size,
418
config.hidden_size,
419
org_num_embeddings=config.vocab_size,
423
- padding_size=DEFAULT_VOCAB_PADDING_SIZE
424
- # We need bigger padding if using lora for kernel
425
- # compatibility
426
- if not lora_config else lora_config.lora_vocab_padding_size,
420
+ padding_size=(
421
+ DEFAULT_VOCAB_PADDING_SIZE
422
+ # We need bigger padding if using lora for kernel
423
+ # compatibility
424
+ if not lora_config else
425
+ lora_config.lora_vocab_padding_size),
426
quant_config=quant_config,
427
)
428
if config.tie_word_embeddings:
@@ -436,6 +435,8 @@ class LlamaForCausalLM(nn.Module, SupportsLoRA):
435
self.sampler = Sampler()
436
else:
437
self.lm_head = PPMissingLayer()
438
+ self.make_empty_intermediate_tensors = (
439
+ self.model.make_empty_intermediate_tensors)
440
441
def forward(
442
self,
@@ -458,28 +459,11 @@ class LlamaForCausalLM(nn.Module, SupportsLoRA):
459
sampling_metadata)
460
return logits
461
461
- def sample(
462
- self,
463
- logits: torch.Tensor,
464
- sampling_metadata: SamplingMetadata,
465
- ) -> Optional[SamplerOutput]:
462
+ def sample(self, logits: torch.Tensor,
463
+ sampling_metadata: SamplingMetadata) -> Optional[SamplerOutput]:
464
next_tokens = self.sampler(logits, sampling_metadata)
465
return next_tokens
466
469
- def make_empty_intermediate_tensors(
470
- self, batch_size: int, dtype: torch.dtype,
471
- device: torch.device) -> IntermediateTensors:
472
- return IntermediateTensors({
473
- "hidden_states":
474
- torch.zeros((batch_size, self.config.hidden_size),
475
- dtype=dtype,
476
- device=device),
477
- "residual":
478
- torch.zeros((batch_size, self.config.hidden_size),
479
- dtype=dtype,
480
- device=device),
481
- })
482
-
467
def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
468
stacked_params_mapping = [
469
# (param_name, shard_name, shard_id)
@@ -513,7 +497,7 @@ class LlamaForCausalLM(nn.Module, SupportsLoRA):
497
loaded_weight = loaded_weight[0]
498
weight_loader(param, loaded_weight)
499
continue
516
- for (param_name, weight_name, shard_id) in stacked_params_mapping:
500
+ for param_name, weight_name, shard_id in stacked_params_mapping:
501
if weight_name not in name:
502
continue
503
name = name.replace(weight_name, param_name)
vllm/model_executor/models/llama_embedding.py
+16
-5
@@ -1,4 +1,4 @@
1
-from typing import Iterable, List, Optional, Tuple
1
+from typing import Iterable, List, Optional, Tuple, Union
2
3
import torch
4
from torch import nn
@@ -8,10 +8,13 @@ from vllm.model_executor.layers.pooler import Pooler, PoolingType
8
from vllm.model_executor.model_loader.weight_utils import default_weight_loader
9
from vllm.model_executor.models.llama import LlamaModel
10
from vllm.model_executor.pooling_metadata import PoolingMetadata
11
-from vllm.sequence import PoolerOutput
11
+from vllm.sequence import IntermediateTensors, PoolerOutput
12
13
+from .interfaces import SupportsPP
14
+from .utils import is_pp_missing_parameter
15
14
-class LlamaEmbeddingModel(nn.Module):
16
+
17
+class LlamaEmbeddingModel(nn.Module, SupportsPP):
18
"""A model that uses Llama with additional embedding functionalities.
19
20
This class encapsulates the LlamaModel and provides an interface for
@@ -29,6 +32,8 @@ class LlamaEmbeddingModel(nn.Module):
32
super().__init__()
33
self.model = LlamaModel(**kwargs)
34
self._pooler = Pooler(pooling_type=PoolingType.LAST, normalize=True)
35
+ self.make_empty_intermediate_tensors = (
36
+ self.model.make_empty_intermediate_tensors)
37
38
def forward(
39
self,
@@ -36,10 +41,12 @@ class LlamaEmbeddingModel(nn.Module):
41
positions: torch.Tensor,
42
kv_caches: List[torch.Tensor],
43
attn_metadata: AttentionMetadata,
44
+ intermediate_tensors: Optional[IntermediateTensors] = None,
45
inputs_embeds: Optional[torch.Tensor] = None,
40
- ) -> torch.Tensor:
46
+ ) -> Union[torch.Tensor, IntermediateTensors]:
47
return self.model.forward(input_ids, positions, kv_caches,
42
- attn_metadata, inputs_embeds)
48
+ attn_metadata, intermediate_tensors,
49
+ inputs_embeds)
50
51
def pooler(
52
self,
@@ -73,6 +80,8 @@ class LlamaEmbeddingModel(nn.Module):
80
# Skip loading extra bias for GPTQ models.
81
if name.endswith(".bias") and name not in params_dict:
82
continue
83
+ if is_pp_missing_parameter(name, self):
84
+ continue
85
param = params_dict[name]
86
weight_loader = param.weight_loader
87
weight_loader(param, loaded_weight, shard_id)
@@ -81,6 +90,8 @@ class LlamaEmbeddingModel(nn.Module):
90
# Skip loading extra bias for GPTQ models.
91
if name.endswith(".bias") and name not in params_dict:
92
continue
93
+ if is_pp_missing_parameter(name, self):
94
+ continue
95
param = params_dict[name]
96
weight_loader = getattr(param, "weight_loader",
97
default_weight_loader)
vllm/model_executor/models/llava.py
+31
-16
@@ -1,3 +1,4 @@
1
+from functools import cached_property
2
from typing import (Iterable, List, Literal, Mapping, Optional, Tuple,
3
TypedDict, Union)
4
@@ -11,7 +12,7 @@ from vllm.config import CacheConfig, MultiModalConfig
12
from vllm.inputs import INPUT_REGISTRY, InputContext, LLMInputs
13
from vllm.model_executor.layers.activation import get_act_fn
14
from vllm.model_executor.layers.quantization import QuantizationConfig
14
-from vllm.model_executor.layers.sampler import SamplerOutput
15
+from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
16
from vllm.model_executor.model_loader.weight_utils import default_weight_loader
17
from vllm.model_executor.sampling_metadata import SamplingMetadata
18
from vllm.multimodal import MULTIMODAL_REGISTRY
@@ -21,7 +22,7 @@ from vllm.utils import is_list_of
22
from .clip import (CLIPVisionModel, dummy_image_for_clip,
23
dummy_seq_data_for_clip, get_max_clip_image_tokens,
24
input_processor_for_clip)
24
-from .interfaces import SupportsMultiModal
25
+from .interfaces import SupportsMultiModal, SupportsPP
26
from .siglip import (SiglipVisionModel, dummy_image_for_siglip,
27
dummy_seq_data_for_siglip, get_max_siglip_image_tokens,
28
input_processor_for_siglip)
@@ -198,7 +199,7 @@ def _init_vision_tower(hf_config: LlavaConfig):
199
@MULTIMODAL_REGISTRY.register_max_image_tokens(get_max_llava_image_tokens)
200
@INPUT_REGISTRY.register_dummy_data(dummy_data_for_llava)
201
@INPUT_REGISTRY.register_input_processor(input_processor_for_llava)
201
-class LlavaForConditionalGeneration(nn.Module, SupportsMultiModal):
202
+class LlavaForConditionalGeneration(nn.Module, SupportsMultiModal, SupportsPP):
203
204
def __init__(self,
205
config: LlavaConfig,
@@ -220,6 +221,16 @@ class LlavaForConditionalGeneration(nn.Module, SupportsMultiModal):
221
self.language_model = init_vllm_registered_model(
222
config.text_config, cache_config, quant_config)
223
224
+ self.make_empty_intermediate_tensors = (
225
+ self.language_model.make_empty_intermediate_tensors)
226
+
227
+ @cached_property
228
+ def sampler(self):
229
+ if hasattr(self.language_model, "sampler"):
230
+ return self.language_model.sampler
231
+
232
+ return Sampler()
233
+
234
def _validate_pixel_values(self, data: torch.Tensor) -> torch.Tensor:
235
h = w = self.config.vision_config.image_size
236
expected_dims = (3, h, w)
@@ -315,7 +326,7 @@ class LlavaForConditionalGeneration(nn.Module, SupportsMultiModal):
326
attn_metadata: AttentionMetadata,
327
intermediate_tensors: Optional[IntermediateTensors] = None,
328
**kwargs: object,
318
- ) -> SamplerOutput:
329
+ ) -> Union[torch.Tensor, IntermediateTensors]:
330
"""Run forward pass for LLaVA-1.5.
331
332
One key thing to understand is the `input_ids` already accounts for the
@@ -351,26 +362,30 @@ class LlavaForConditionalGeneration(nn.Module, SupportsMultiModal):
362
See also:
363
:class:`LlavaImageInputs`
364
"""
354
- image_input = self._parse_and_validate_image_input(**kwargs)
365
+ if intermediate_tensors is not None:
366
+ input_ids = None
367
+ inputs_embeds = None
368
+ else:
369
+ image_input = self._parse_and_validate_image_input(**kwargs)
370
356
- if image_input is not None:
357
- vision_embeddings = self._process_image_input(image_input)
358
- inputs_embeds = self.language_model.model.get_input_embeddings(
359
- input_ids)
371
+ if image_input is not None:
372
+ vision_embeddings = self._process_image_input(image_input)
373
+ inputs_embeds = self.language_model.model.get_input_embeddings(
374
+ input_ids)
375
361
- inputs_embeds = merge_multimodal_embeddings(
362
- input_ids, inputs_embeds, vision_embeddings,
363
- self.config.image_token_index)
376
+ inputs_embeds = merge_multimodal_embeddings(
377
+ input_ids, inputs_embeds, vision_embeddings,
378
+ self.config.image_token_index)
379
365
- input_ids = None
366
- else:
367
- inputs_embeds = None
380
+ input_ids = None
381
+ else:
382
+ inputs_embeds = None
383
384
hidden_states = self.language_model.model(input_ids,
385
positions,
386
kv_caches,
387
attn_metadata,
373
- None,
388
+ intermediate_tensors,
389
inputs_embeds=inputs_embeds)
390
391
return hidden_states
vllm/model_executor/models/llava_next.py
+33
-18
@@ -1,3 +1,4 @@
1
+from functools import cached_property
2
from typing import (Iterable, List, Literal, Mapping, Optional, Tuple,
3
TypedDict, Union)
4
@@ -13,7 +14,7 @@ from vllm.attention import AttentionMetadata
14
from vllm.config import CacheConfig, MultiModalConfig
15
from vllm.inputs import INPUT_REGISTRY, InputContext, LLMInputs
16
from vllm.model_executor.layers.quantization import QuantizationConfig
16
-from vllm.model_executor.layers.sampler import SamplerOutput
17
+from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
18
from vllm.model_executor.model_loader.weight_utils import default_weight_loader
19
from vllm.model_executor.sampling_metadata import SamplingMetadata
20
from vllm.multimodal import MULTIMODAL_REGISTRY
@@ -23,7 +24,7 @@ from vllm.utils import is_list_of
24
from .clip import (CLIPVisionModel, dummy_image_for_clip,
25
dummy_seq_data_for_clip, get_clip_image_feature_size,
26
get_clip_patch_grid_length, input_processor_for_clip)
26
-from .interfaces import SupportsMultiModal
27
+from .interfaces import SupportsMultiModal, SupportsPP
28
from .llava import LlavaMultiModalProjector
29
from .siglip import (SiglipVisionModel, dummy_image_for_siglip,
30
dummy_seq_data_for_siglip, get_siglip_image_feature_size,
@@ -286,7 +287,8 @@ def _init_vision_tower(hf_config: LlavaNextConfig):
287
@MULTIMODAL_REGISTRY.register_max_image_tokens(get_max_llava_next_image_tokens)
288
@INPUT_REGISTRY.register_dummy_data(dummy_data_for_llava_next)
289
@INPUT_REGISTRY.register_input_processor(input_processor_for_llava_next)
289
-class LlavaNextForConditionalGeneration(nn.Module, SupportsMultiModal):
290
+class LlavaNextForConditionalGeneration(nn.Module, SupportsMultiModal,
291
+ SupportsPP):
292
293
def __init__(self,
294
config: LlavaNextConfig,
@@ -300,6 +302,8 @@ class LlavaNextForConditionalGeneration(nn.Module, SupportsMultiModal):
302
303
# TODO: Optionally initializes this for supporting embeddings.
304
self.vision_tower = _init_vision_tower(config)
305
+ self.image_newline = nn.Parameter(
306
+ torch.empty(config.text_config.hidden_size))
307
self.multi_modal_projector = LlavaMultiModalProjector(
308
vision_hidden_size=config.vision_config.hidden_size,
309
text_hidden_size=config.text_config.hidden_size,
@@ -308,8 +312,15 @@ class LlavaNextForConditionalGeneration(nn.Module, SupportsMultiModal):
312
self.language_model = init_vllm_registered_model(
313
config.text_config, cache_config, quant_config)
314
311
- self.image_newline = nn.Parameter(
312
- torch.empty(config.text_config.hidden_size))
315
+ self.make_empty_intermediate_tensors = (
316
+ self.language_model.make_empty_intermediate_tensors)
317
+
318
+ @cached_property
319
+ def sampler(self):
320
+ if hasattr(self.language_model, "sampler"):
321
+ return self.language_model.sampler
322
+
323
+ return Sampler()
324
325
def _validate_image_sizes(self, data: torch.Tensor) -> torch.Tensor:
326
expected_dims = (2, )
@@ -542,7 +553,7 @@ class LlavaNextForConditionalGeneration(nn.Module, SupportsMultiModal):
553
attn_metadata: AttentionMetadata,
554
intermediate_tensors: Optional[IntermediateTensors] = None,
555
**kwargs: object,
545
- ) -> SamplerOutput:
556
+ ) -> Union[torch.Tensor, IntermediateTensors]:
557
"""Run forward pass for LlaVA-NeXT.
558
559
One key thing to understand is the `input_ids` already accounts for the
@@ -587,26 +598,30 @@ class LlavaNextForConditionalGeneration(nn.Module, SupportsMultiModal):
598
See also:
599
:class:`LlavaNextImageInputs`
600
"""
590
- image_input = self._parse_and_validate_image_input(**kwargs)
601
+ if intermediate_tensors is not None:
602
+ input_ids = None
603
+ inputs_embeds = None
604
+ else:
605
+ image_input = self._parse_and_validate_image_input(**kwargs)
606
592
- if image_input is not None:
593
- vision_embeddings = self._process_image_input(image_input)
594
- inputs_embeds = self.language_model.model.get_input_embeddings(
595
- input_ids)
607
+ if image_input is not None:
608
+ vision_embeddings = self._process_image_input(image_input)
609
+ inputs_embeds = self.language_model.model.get_input_embeddings(
610
+ input_ids)
611
597
- inputs_embeds = merge_multimodal_embeddings(
598
- input_ids, inputs_embeds, vision_embeddings,
599
- self.config.image_token_index)
612
+ inputs_embeds = merge_multimodal_embeddings(
613
+ input_ids, inputs_embeds, vision_embeddings,
614
+ self.config.image_token_index)
615
601
- input_ids = None
602
- else:
603
- inputs_embeds = None
616
+ input_ids = None
617
+ else:
618
+ inputs_embeds = None
619
620
hidden_states = self.language_model.model(input_ids,
621
positions,
622
kv_caches,
623
attn_metadata,
609
- None,
624
+ intermediate_tensors,
625
inputs_embeds=inputs_embeds)
626
627
return hidden_states
vllm/model_executor/models/llava_next_video.py
+34
-21
@@ -1,4 +1,5 @@
1
import math
2
+from functools import cached_property
3
from typing import (Iterable, List, Literal, Mapping, Optional, Tuple,
4
TypedDict, Union)
5
@@ -12,9 +13,8 @@ from vllm.attention import AttentionMetadata
13
from vllm.config import CacheConfig, MultiModalConfig
14
from vllm.inputs import INPUT_REGISTRY, InputContext, LLMInputs
15
from vllm.model_executor.layers.activation import get_act_fn
15
-from vllm.model_executor.layers.quantization.base_config import (
16
- QuantizationConfig)
17
-from vllm.model_executor.layers.sampler import SamplerOutput
16
+from vllm.model_executor.layers.quantization import QuantizationConfig
17
+from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
18
from vllm.model_executor.model_loader.weight_utils import default_weight_loader
19
from vllm.model_executor.models.clip import CLIPVisionModel
20
from vllm.model_executor.sampling_metadata import SamplingMetadata
@@ -25,7 +25,7 @@ from vllm.sequence import IntermediateTensors
25
from vllm.utils import is_list_of
26
27
from .clip import dummy_image_for_clip, dummy_seq_data_for_clip
28
-from .interfaces import SupportsMultiModal
28
+from .interfaces import SupportsMultiModal, SupportsPP
29
from .siglip import (SiglipVisionModel, dummy_image_for_siglip,
30
dummy_seq_data_for_siglip)
31
from .utils import (group_weights_with_prefix, init_vllm_registered_model,
@@ -267,7 +267,8 @@ class LlavaNextMultiModalProjector(nn.Module):
267
"video", get_max_llava_next_video_tokens)
268
@INPUT_REGISTRY.register_dummy_data(dummy_data_for_llava_next_video)
269
@INPUT_REGISTRY.register_input_processor(input_processor_for_llava_next_video)
270
-class LlavaNextVideoForConditionalGeneration(nn.Module, SupportsMultiModal):
270
+class LlavaNextVideoForConditionalGeneration(nn.Module, SupportsMultiModal,
271
+ SupportsPP):
272
273
def __init__(self,
274
config: LlavaNextVideoConfig,
@@ -281,13 +282,23 @@ class LlavaNextVideoForConditionalGeneration(nn.Module, SupportsMultiModal):
282
283
# Initialize the vision tower only up to the required feature layer
284
self.vision_tower = _init_vision_tower(config)
285
+ self.vision_resampler = LlavaNextVideoPooler(config)
286
self.multi_modal_projector = LlavaNextMultiModalProjector(
287
vision_hidden_size=config.vision_config.hidden_size,
288
text_hidden_size=config.text_config.hidden_size,
289
projector_hidden_act=config.projector_hidden_act)
290
self.language_model = init_vllm_registered_model(
291
config.text_config, cache_config, quant_config)
290
- self.vision_resampler = LlavaNextVideoPooler(config)
292
+
293
+ self.make_empty_intermediate_tensors = (
294
+ self.language_model.model.make_empty_intermediate_tensors)
295
+
296
+ @cached_property
297
+ def sampler(self):
298
+ if hasattr(self.language_model, "sampler"):
299
+ return self.language_model.sampler
300
+
301
+ return Sampler()
302
303
def _validate_video_pixel_values(
304
self, data: Union[torch.Tensor, List[torch.Tensor]]
@@ -397,34 +408,36 @@ class LlavaNextVideoForConditionalGeneration(nn.Module, SupportsMultiModal):
408
attn_metadata: AttentionMetadata,
409
intermediate_tensors: Optional[IntermediateTensors] = None,
410
**kwargs: object,
400
- ) -> SamplerOutput:
411
+ ) -> Union[torch.Tensor, IntermediateTensors]:
412
"""Run forward pass for LlaVA-NeXT-Video.
413
Args:
414
input_ids: Flattened (concatenated) input_ids corresponding to a
415
batch.
416
pixel_values_videos: Pixels in each frames for each input videos.
417
"""
407
- video_input = self._parse_and_validate_video_input(**kwargs)
408
-
409
- # merge video embeddings into input embeddings
410
- if video_input is not None:
411
- video_embeddings = self._process_video_pixels(video_input)
412
- inputs_embeds = self.language_model \
413
- .model.get_input_embeddings(input_ids)
414
-
415
- inputs_embeds = merge_multimodal_embeddings(
416
- input_ids, inputs_embeds, video_embeddings,
417
- self.config.video_token_index)
418
-
418
+ if intermediate_tensors is not None:
419
input_ids = None
420
- else:
420
inputs_embeds = None
421
+ else:
422
+ video_input = self._parse_and_validate_video_input(**kwargs)
423
+ if video_input is not None:
424
+ video_embeddings = self._process_video_pixels(video_input)
425
+ inputs_embeds = self.language_model \
426
+ .model.get_input_embeddings(input_ids)
427
+
428
+ inputs_embeds = merge_multimodal_embeddings(
429
+ input_ids, inputs_embeds, video_embeddings,
430
+ self.config.video_token_index)
431
+
432
+ input_ids = None
433
+ else:
434
+ inputs_embeds = None
435
436
hidden_states = self.language_model.model(input_ids,
437
positions,
438
kv_caches,
439
attn_metadata,
427
- None,
440
+ intermediate_tensors,
441
inputs_embeds=inputs_embeds)
442
443
return hidden_states
vllm/model_executor/models/llava_onevision.py
+39
-25
@@ -1,4 +1,5 @@
1
import math
2
+from functools import cached_property
3
from typing import (Iterable, List, Literal, Mapping, Optional, Tuple,
4
TypedDict, Union)
5
@@ -17,9 +18,8 @@ from vllm.config import CacheConfig, MultiModalConfig
18
from vllm.inputs import INPUT_REGISTRY, InputContext, LLMInputs
19
from vllm.logger import init_logger
20
from vllm.model_executor.layers.activation import get_act_fn
20
-from vllm.model_executor.layers.quantization.base_config import (
21
- QuantizationConfig)
22
-from vllm.model_executor.layers.sampler import SamplerOutput
21
+from vllm.model_executor.layers.quantization import QuantizationConfig
22
+from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
23
from vllm.model_executor.model_loader.weight_utils import default_weight_loader
24
from vllm.model_executor.sampling_metadata import SamplingMetadata
25
from vllm.multimodal import MULTIMODAL_REGISTRY
@@ -31,7 +31,7 @@ from vllm.utils import is_list_of
31
from .clip import (CLIPVisionModel, dummy_seq_data_for_clip,
32
dummy_video_for_clip, get_clip_image_feature_size,
33
get_clip_patch_grid_length, input_processor_for_clip)
34
-from .interfaces import SupportsMultiModal
34
+from .interfaces import SupportsMultiModal, SupportsPP
35
from .siglip import (SiglipVisionModel, dummy_seq_data_for_siglip,
36
dummy_video_for_siglip, get_siglip_image_feature_size,
37
get_siglip_patch_grid_length, input_processor_for_siglip)
@@ -414,7 +414,8 @@ class LlavaOnevisionMultiModalProjector(nn.Module):
414
"video", get_max_llava_onevision_video_tokens)
415
@INPUT_REGISTRY.register_dummy_data(dummy_data_for_llava_onevision)
416
@INPUT_REGISTRY.register_input_processor(input_processor_for_llava_onevision)
417
-class LlavaOnevisionForConditionalGeneration(nn.Module, SupportsMultiModal):
417
+class LlavaOnevisionForConditionalGeneration(nn.Module, SupportsMultiModal,
418
+ SupportsPP):
419
420
def __init__(self,
421
config: LlavaOnevisionConfig,
@@ -434,6 +435,16 @@ class LlavaOnevisionForConditionalGeneration(nn.Module, SupportsMultiModal):
435
self.image_newline = nn.Parameter(
436
torch.empty(config.text_config.hidden_size))
437
438
+ self.make_empty_intermediate_tensors = (
439
+ self.language_model.model.make_empty_intermediate_tensors)
440
+
441
+ @cached_property
442
+ def sampler(self):
443
+ if hasattr(self.language_model, "sampler"):
444
+ return self.language_model.sampler
445
+
446
+ return Sampler()
447
+
448
def _validate_image_sizes(self, data: torch.Tensor) -> torch.Tensor:
449
expected_dims = (2, )
450
@@ -805,39 +816,42 @@ class LlavaOnevisionForConditionalGeneration(nn.Module, SupportsMultiModal):
816
attn_metadata: AttentionMetadata,
817
intermediate_tensors: Optional[IntermediateTensors] = None,
818
**kwargs: object,
808
- ) -> SamplerOutput:
819
+ ) -> Union[torch.Tensor, IntermediateTensors]:
820
"""Run forward pass for LlaVA-Onevision.
821
Args:
822
input_ids: Flattened (concatenated) input_ids corresponding to a
823
batch.
824
pixel_values_videos: Pixels in each frames for each input videos.
825
"""
815
- modalities = self._parse_and_validate_multimodal_inputs(**kwargs)
816
- # merge video embeddings into input embeddings
817
- if modalities:
818
- inputs_embeds = self.language_model.model.get_input_embeddings(
819
- input_ids)
820
- if "images" in modalities:
821
- image_input = modalities["images"]
822
- vision_embeddings = self._process_image_input(image_input)
823
- inputs_embeds = merge_multimodal_embeddings(
824
- input_ids, inputs_embeds, vision_embeddings,
825
- self.config.image_token_index)
826
- if "videos" in modalities:
827
- video_input = modalities["videos"]
828
- video_embeddings = self._process_video_pixels(video_input)
829
- inputs_embeds = merge_multimodal_embeddings(
830
- input_ids, inputs_embeds, video_embeddings,
831
- self.config.video_token_index)
826
+ if intermediate_tensors is not None:
827
input_ids = None
833
- else:
828
inputs_embeds = None
829
+ else:
830
+ modalities = self._parse_and_validate_multimodal_inputs(**kwargs)
831
+ if modalities:
832
+ inputs_embeds = self.language_model.model.get_input_embeddings(
833
+ input_ids)
834
+ if "images" in modalities:
835
+ image_input = modalities["images"]
836
+ vision_embeddings = self._process_image_input(image_input)
837
+ inputs_embeds = merge_multimodal_embeddings(
838
+ input_ids, inputs_embeds, vision_embeddings,
839
+ self.config.image_token_index)
840
+ if "videos" in modalities:
841
+ video_input = modalities["videos"]
842
+ video_embeddings = self._process_video_pixels(video_input)
843
+ inputs_embeds = merge_multimodal_embeddings(
844
+ input_ids, inputs_embeds, video_embeddings,
845
+ self.config.video_token_index)
846
+ input_ids = None
847
+ else:
848
+ inputs_embeds = None
849
850
hidden_states = self.language_model.model(input_ids,
851
positions,
852
kv_caches,
853
attn_metadata,
840
- None,
854
+ intermediate_tensors,
855
inputs_embeds=inputs_embeds)
856
857
return hidden_states
vllm/model_executor/models/minicpm.py
+51
-23
@@ -22,7 +22,7 @@
22
# limitations under the License.
23
"""Inference-only MiniCPM model compatible with HuggingFace weights."""
24
import math
25
-from typing import Any, Dict, Iterable, List, Optional, Tuple
25
+from typing import Any, Dict, Iterable, List, Optional, Tuple, Union
26
27
import torch
28
from torch import nn
@@ -30,7 +30,7 @@ from transformers import PretrainedConfig
30
31
from vllm.attention import Attention, AttentionMetadata
32
from vllm.config import CacheConfig, LoRAConfig
33
-from vllm.distributed import (get_tensor_model_parallel_rank,
33
+from vllm.distributed import (get_pp_group, get_tensor_model_parallel_rank,
34
get_tensor_model_parallel_world_size,
35
tensor_model_parallel_all_reduce)
36
from vllm.model_executor.layers.activation import SiluAndMul
@@ -41,8 +41,7 @@ from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
41
ReplicatedLinear,
42
RowParallelLinear)
43
from vllm.model_executor.layers.logits_processor import LogitsProcessor
44
-from vllm.model_executor.layers.quantization.base_config import (
45
- QuantizationConfig)
44
+from vllm.model_executor.layers.quantization import QuantizationConfig
45
from vllm.model_executor.layers.rotary_embedding import get_rope
46
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
47
from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -52,7 +51,9 @@ from vllm.model_executor.sampling_metadata import SamplingMetadata
51
from vllm.model_executor.utils import set_weight_attrs
52
from vllm.sequence import IntermediateTensors
53
55
-from .interfaces import SupportsLoRA
54
+from .interfaces import SupportsLoRA, SupportsPP
55
+from .utils import (is_pp_missing_parameter,
56
+ make_empty_intermediate_tensors_factory, make_layers)
57
58
59
class MiniCPMMoE(nn.Module):
@@ -264,7 +265,7 @@ class MiniCPMDecoderLayer(nn.Module):
265
266
def __init__(
267
self,
267
- config,
268
+ config: PretrainedConfig,
269
cache_config: Optional[CacheConfig] = None,
270
quant_config: Optional[QuantizationConfig] = None,
271
) -> None:
@@ -346,10 +347,11 @@ class MiniCPMModel(nn.Module):
347
348
def __init__(
349
self,
349
- config,
350
+ config: PretrainedConfig,
351
cache_config: Optional[CacheConfig] = None,
352
quant_config: Optional[QuantizationConfig] = None,
353
lora_config: Optional[LoRAConfig] = None,
354
+ prefix: str = "",
355
) -> None:
356
super().__init__()
357
self.config = config
@@ -365,15 +367,24 @@ class MiniCPMModel(nn.Module):
367
config.hidden_size,
368
org_num_embeddings=config.vocab_size,
369
)
368
- self._init_layers()
370
+ self._init_layers(prefix, config, cache_config, quant_config)
371
self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
372
+ self.make_empty_intermediate_tensors = (
373
+ make_empty_intermediate_tensors_factory(
374
+ ["hidden_states", "residual"], self.config.hidden_size))
375
371
- def _init_layers(self):
372
- self.layers = nn.ModuleList([
373
- MiniCPMDecoderLayer(self.config, self.cache_config,
374
- self.quant_config)
375
- for _ in range(self.config.num_hidden_layers)
376
- ])
376
+ def _init_layers(
377
+ self,
378
+ prefix: str,
379
+ config: PretrainedConfig,
380
+ cache_config: Optional[CacheConfig],
381
+ quant_config: Optional[QuantizationConfig],
382
+ ):
383
+ self.start_layer, self.end_layer, self.layers = make_layers(
384
+ config.num_hidden_layers,
385
+ lambda prefix: MiniCPMDecoderLayer(config, cache_config,
386
+ quant_config),
387
+ prefix=f"{prefix}.layers")
388
389
def get_input_embeddings(self, input_ids: torch.Tensor) -> torch.Tensor:
390
embedding = self.embed_tokens(input_ids)
@@ -387,27 +398,36 @@ class MiniCPMModel(nn.Module):
398
attn_metadata: AttentionMetadata,
399
intermediate_tensors: Optional[IntermediateTensors] = None,
400
inputs_embeds: Optional[torch.Tensor] = None,
390
- ) -> torch.Tensor:
391
- if inputs_embeds is not None:
392
- hidden_states = inputs_embeds
401
+ ) -> Union[torch.Tensor, IntermediateTensors]:
402
+ if get_pp_group().is_first_rank:
403
+ if inputs_embeds is not None:
404
+ hidden_states = inputs_embeds
405
+ else:
406
+ hidden_states = self.get_input_embeddings(input_ids)
407
+ residual = None
408
else:
394
- hidden_states = self.get_input_embeddings(input_ids)
395
- residual = None
409
+ hidden_states = intermediate_tensors["hidden_states"]
410
+ residual = intermediate_tensors["residual"]
411
397
- for i in range(len(self.layers)):
412
+ for i in range(self.start_layer, self.end_layer):
413
layer = self.layers[i]
414
hidden_states, residual = layer(
415
positions,
416
hidden_states,
402
- kv_caches[i],
417
+ kv_caches[i - self.start_layer],
418
attn_metadata,
419
residual,
420
)
421
+ if not get_pp_group().is_last_rank:
422
+ return IntermediateTensors({
423
+ "hidden_states": hidden_states,
424
+ "residual": residual
425
+ })
426
hidden_states = self.norm(hidden_states)
427
return hidden_states
428
429
410
-class MiniCPMForCausalLM(nn.Module, SupportsLoRA):
430
+class MiniCPMForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
431
packed_modules_mapping = {
432
"qkv_proj": [
433
"q_proj",
@@ -470,6 +490,8 @@ class MiniCPMForCausalLM(nn.Module, SupportsLoRA):
490
self.logits_processor = LogitsProcessor(unpadded_vocab_size,
491
config.vocab_size)
492
self.sampler = Sampler()
493
+ self.make_empty_intermediate_tensors = (
494
+ self.model.make_empty_intermediate_tensors)
495
496
def _init_model(self):
497
self.model = MiniCPMModel(config=self.config,
@@ -484,7 +506,7 @@ class MiniCPMForCausalLM(nn.Module, SupportsLoRA):
506
kv_caches: List[torch.Tensor],
507
attn_metadata: AttentionMetadata,
508
intermediate_tensors: Optional[IntermediateTensors] = None,
487
- ) -> torch.Tensor:
509
+ ) -> Union[torch.Tensor, IntermediateTensors]:
510
hidden_states = self.model(input_ids, positions, kv_caches,
511
attn_metadata, intermediate_tensors)
512
return hidden_states
@@ -548,6 +570,8 @@ class MiniCPMForCausalLM(nn.Module, SupportsLoRA):
570
# Skip loading extra bias for GPTQ models.
571
if name.endswith(".bias") and name not in params_dict:
572
continue
573
+ if is_pp_missing_parameter(name, self):
574
+ continue
575
param = params_dict[name]
576
weight_loader = param.weight_loader
577
weight_loader(param, loaded_weight, shard_id)
@@ -557,6 +581,8 @@ class MiniCPMForCausalLM(nn.Module, SupportsLoRA):
581
if weight_name not in name:
582
continue
583
name = name.replace(weight_name, param_name)
584
+ if is_pp_missing_parameter(name, self):
585
+ continue
586
param = params_dict[name]
587
weight_loader = param.weight_loader
588
weight_loader(param,
@@ -568,6 +594,8 @@ class MiniCPMForCausalLM(nn.Module, SupportsLoRA):
594
# Skip loading extra bias for GPTQ models.
595
if name.endswith(".bias") and name not in params_dict:
596
continue
597
+ if is_pp_missing_parameter(name, self):
598
+ continue
599
param = params_dict[name]
600
weight_loader = getattr(param, "weight_loader",
601
default_weight_loader)
vllm/model_executor/models/minicpm3.py
+17
-9
@@ -26,6 +26,7 @@ from typing import Any, Dict, Optional
26
27
import torch
28
from torch import nn
29
+from transformers import PretrainedConfig
30
31
from vllm.attention import Attention, AttentionMetadata
32
from vllm.config import CacheConfig
@@ -34,19 +35,20 @@ from vllm.model_executor.layers.layernorm import RMSNorm
35
from vllm.model_executor.layers.linear import (ColumnParallelLinear,
36
ReplicatedLinear,
37
RowParallelLinear)
37
-from vllm.model_executor.layers.quantization.base_config import (
38
- QuantizationConfig)
38
+from vllm.model_executor.layers.quantization import QuantizationConfig
39
from vllm.model_executor.layers.rotary_embedding import get_rope
40
from vllm.model_executor.models.minicpm import (MiniCPMDecoderLayer,
41
MiniCPMForCausalLM,
42
MiniCPMModel)
43
44
+from .utils import make_layers
45
+
46
47
class MiniCPM3Attention(nn.Module):
48
49
def __init__(
50
self,
49
- config,
51
+ config: PretrainedConfig,
52
hidden_size: int,
53
num_heads: int,
54
qk_nope_head_dim: int,
@@ -199,12 +201,18 @@ class MiniCPM3DecoderLayer(MiniCPMDecoderLayer):
201
202
class MiniCPM3Model(MiniCPMModel):
203
202
- def _init_layers(self):
203
- self.layers = nn.ModuleList([
204
- MiniCPM3DecoderLayer(self.config, self.cache_config,
205
- self.quant_config)
206
- for _ in range(self.config.num_hidden_layers)
207
- ])
204
+ def _init_layers(
205
+ self,
206
+ prefix: str,
207
+ config: PretrainedConfig,
208
+ cache_config: Optional[CacheConfig],
209
+ quant_config: Optional[QuantizationConfig],
210
+ ):
211
+ self.start_layer, self.end_layer, self.layers = make_layers(
212
+ config.num_hidden_layers,
213
+ lambda prefix: MiniCPM3DecoderLayer(config, cache_config,
214
+ quant_config),
215
+ prefix=f"{prefix}.layers")
216
217
218
class MiniCPM3ForCausalLM(MiniCPMForCausalLM):
vllm/model_executor/models/minicpmv.py
+16
-5
@@ -45,7 +45,6 @@ from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
45
from vllm.model_executor.layers.vocab_parallel_embedding import ParallelLMHead
46
from vllm.model_executor.model_loader.utils import set_default_torch_dtype
47
from vllm.model_executor.model_loader.weight_utils import default_weight_loader
48
-from vllm.model_executor.models.interfaces import SupportsMultiModal
48
from vllm.model_executor.models.llama import LlamaModel
49
from vllm.model_executor.models.minicpm import MiniCPMModel
50
from vllm.model_executor.models.module_mapping import MultiModelKeys
@@ -59,7 +58,8 @@ from vllm.multimodal.utils import cached_get_tokenizer
58
from vllm.sequence import IntermediateTensors, SequenceData
59
60
from .idefics2_vision_model import Idefics2VisionTransformer
62
-from .interfaces import SupportsLoRA
61
+from .interfaces import SupportsLoRA, SupportsMultiModal, SupportsPP
62
+from .utils import is_pp_missing_parameter
63
64
_KEYS_TO_MODIFY_MAPPING = {
65
"llm.lm_head": "lm_head",
@@ -337,7 +337,7 @@ def input_mapper_for_minicpmv(ctx: InputContext, data: object):
337
return MultiModalInputs(batch_data)
338
339
340
-class MiniCPMVBaseModel(nn.Module, SupportsMultiModal):
340
+class MiniCPMVBaseModel(nn.Module, SupportsMultiModal, SupportsPP):
341
"""
342
The abstract class of MiniCPMV can only be inherited, but cannot be
343
instantiated.
@@ -374,6 +374,9 @@ class MiniCPMVBaseModel(nn.Module, SupportsMultiModal):
374
self.logits_processor = LogitsProcessor(config.vocab_size)
375
self.sampler = Sampler()
376
377
+ self.make_empty_intermediate_tensors = (
378
+ self.llm.make_empty_intermediate_tensors)
379
+
380
def get_embedding(
381
self,
382
input_ids: torch.Tensor,
@@ -498,9 +501,12 @@ class MiniCPMVBaseModel(nn.Module, SupportsMultiModal):
501
intermediate_tensors: Optional[IntermediateTensors] = None,
502
**kwargs: Any,
503
) -> torch.Tensor:
501
- image_inputs = self._parse_and_validate_inputs(input_ids, **kwargs)
504
+ if intermediate_tensors is not None:
505
+ vlm_embeddings = None
506
+ else:
507
+ image_inputs = self._parse_and_validate_inputs(input_ids, **kwargs)
508
503
- vlm_embeddings, _ = self.get_embedding(input_ids, image_inputs)
509
+ vlm_embeddings, _ = self.get_embedding(input_ids, image_inputs)
510
511
output = self.llm(
512
input_ids=None,
@@ -557,6 +563,9 @@ class MiniCPMVBaseModel(nn.Module, SupportsMultiModal):
563
for param_name, weight_name, shard_id in stacked_params_mapping:
564
if weight_name not in name:
565
continue
566
+ if is_pp_missing_parameter(
567
+ name.replace(weight_name, param_name), self):
568
+ continue
569
param = params_dict[name.replace(weight_name, param_name)]
570
weight_loader = param.weight_loader
571
weight_loader(param, loaded_weight, shard_id)
@@ -564,6 +573,8 @@ class MiniCPMVBaseModel(nn.Module, SupportsMultiModal):
573
else:
574
use_default_weight_loading = True
575
if use_default_weight_loading:
576
+ if is_pp_missing_parameter(name, self):
577
+ continue
578
param = params_dict[name]
579
weight_loader = getattr(param, "weight_loader",
580
default_weight_loader)
vllm/model_executor/models/mixtral.py
+13
-22
@@ -21,7 +21,7 @@
21
# See the License for the specific language governing permissions and
22
# limitations under the License.
23
"""Inference-only Mixtral model."""
24
-from typing import Iterable, List, Optional, Tuple
24
+from typing import Iterable, List, Optional, Tuple, Union
25
26
import torch
27
from torch import nn
@@ -36,8 +36,7 @@ from vllm.model_executor.layers.linear import (QKVParallelLinear,
36
ReplicatedLinear,
37
RowParallelLinear)
38
from vllm.model_executor.layers.logits_processor import LogitsProcessor
39
-from vllm.model_executor.layers.quantization.base_config import (
40
- QuantizationConfig)
39
+from vllm.model_executor.layers.quantization import QuantizationConfig
40
from vllm.model_executor.layers.rotary_embedding import get_rope
41
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
42
from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -47,8 +46,9 @@ from vllm.model_executor.model_loader.weight_utils import (
46
from vllm.model_executor.sampling_metadata import SamplingMetadata
47
from vllm.sequence import IntermediateTensors
48
50
-from .interfaces import SupportsLoRA
51
-from .utils import is_pp_missing_parameter, make_layers
49
+from .interfaces import SupportsLoRA, SupportsPP
50
+from .utils import (is_pp_missing_parameter,
51
+ make_empty_intermediate_tensors_factory, make_layers)
52
53
54
class MixtralMoE(nn.Module):
@@ -276,6 +276,9 @@ class MixtralModel(nn.Module):
276
prefix=f"{prefix}.layers")
277
278
self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
279
+ self.make_empty_intermediate_tensors = (
280
+ make_empty_intermediate_tensors_factory(
281
+ ["hidden_states", "residual"], config.hidden_size))
282
283
def forward(
284
self,
@@ -284,7 +287,7 @@ class MixtralModel(nn.Module):
287
kv_caches: List[torch.Tensor],
288
attn_metadata: AttentionMetadata,
289
intermediate_tensors: Optional[IntermediateTensors],
287
- ) -> torch.Tensor:
290
+ ) -> Union[torch.Tensor, IntermediateTensors]:
291
if get_pp_group().is_first_rank:
292
hidden_states = self.embed_tokens(input_ids)
293
residual = None
@@ -306,7 +309,7 @@ class MixtralModel(nn.Module):
309
return hidden_states
310
311
309
-class MixtralForCausalLM(nn.Module, SupportsLoRA):
312
+class MixtralForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
313
fall_back_to_pt_during_load = False
314
315
packed_modules_mapping = {
@@ -365,6 +368,8 @@ class MixtralForCausalLM(nn.Module, SupportsLoRA):
368
self.logits_processor = LogitsProcessor(self.unpadded_vocab_size,
369
config.vocab_size)
370
self.sampler = Sampler()
371
+ self.make_empty_intermediate_tensors = (
372
+ self.model.make_empty_intermediate_tensors)
373
374
def forward(
375
self,
@@ -373,7 +378,7 @@ class MixtralForCausalLM(nn.Module, SupportsLoRA):
378
kv_caches: List[torch.Tensor],
379
attn_metadata: AttentionMetadata,
380
intermediate_tensors: Optional[IntermediateTensors] = None,
376
- ) -> torch.Tensor:
381
+ ) -> Union[torch.Tensor, IntermediateTensors]:
382
hidden_states = self.model(input_ids, positions, kv_caches,
383
attn_metadata, intermediate_tensors)
384
return hidden_states
@@ -387,20 +392,6 @@ class MixtralForCausalLM(nn.Module, SupportsLoRA):
392
sampling_metadata)
393
return logits
394
390
- def make_empty_intermediate_tensors(
391
- self, batch_size: int, dtype: torch.dtype,
392
- device: torch.device) -> IntermediateTensors:
393
- return IntermediateTensors({
394
- "hidden_states":
395
- torch.zeros((batch_size, self.config.hidden_size),
396
- dtype=dtype,
397
- device=device),
398
- "residual":
399
- torch.zeros((batch_size, self.config.hidden_size),
400
- dtype=dtype,
401
- device=device),
402
- })
403
-
395
def sample(
396
self,
397
logits: Optional[torch.Tensor],
vllm/model_executor/models/mixtral_quant.py
+42
-19
@@ -21,7 +21,7 @@
21
# See the License for the specific language governing permissions and
22
# limitations under the License.
23
"""Inference-only Mixtral model."""
24
-from typing import Iterable, List, Optional, Tuple
24
+from typing import Iterable, List, Optional, Tuple, Union
25
26
import numpy as np
27
import torch
@@ -31,7 +31,7 @@ from transformers import MixtralConfig
31
32
from vllm.attention import Attention, AttentionMetadata
33
from vllm.config import CacheConfig
34
-from vllm.distributed import (get_tensor_model_parallel_rank,
34
+from vllm.distributed import (get_pp_group, get_tensor_model_parallel_rank,
35
get_tensor_model_parallel_world_size,
36
tensor_model_parallel_all_reduce)
37
from vllm.model_executor.layers.layernorm import RMSNorm
@@ -39,8 +39,7 @@ from vllm.model_executor.layers.linear import (QKVParallelLinear,
39
ReplicatedLinear,
40
RowParallelLinear)
41
from vllm.model_executor.layers.logits_processor import LogitsProcessor
42
-from vllm.model_executor.layers.quantization.base_config import (
43
- QuantizationConfig)
42
+from vllm.model_executor.layers.quantization import QuantizationConfig
43
from vllm.model_executor.layers.rotary_embedding import get_rope
44
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
45
from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -49,6 +48,10 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
48
from vllm.model_executor.sampling_metadata import SamplingMetadata
49
from vllm.sequence import IntermediateTensors
50
51
+from .interfaces import SupportsPP
52
+from .utils import (is_pp_missing_parameter,
53
+ make_empty_intermediate_tensors_factory, make_layers)
54
+
55
56
class MixtralMLP(nn.Module):
57
@@ -296,6 +299,7 @@ class MixtralModel(nn.Module):
299
config: MixtralConfig,
300
cache_config: Optional[CacheConfig] = None,
301
quant_config: Optional[QuantizationConfig] = None,
302
+ prefix: str = "",
303
) -> None:
304
super().__init__()
305
self.padding_idx = config.pad_token_id
@@ -305,13 +309,15 @@ class MixtralModel(nn.Module):
309
config.vocab_size,
310
config.hidden_size,
311
)
308
- self.layers = nn.ModuleList([
309
- MixtralDecoderLayer(config,
310
- cache_config,
311
- quant_config=quant_config)
312
- for _ in range(config.num_hidden_layers)
313
- ])
312
+ self.start_layer, self.end_layer, self.layers = make_layers(
313
+ config.num_hidden_layers,
314
+ lambda prefix: MixtralDecoderLayer(
315
+ config, cache_config, quant_config=quant_config),
316
+ prefix=f"{prefix}.layers")
317
self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
318
+ self.make_empty_intermediate_tensors = (
319
+ make_empty_intermediate_tensors_factory(
320
+ ["hidden_states", "residual"], config.hidden_size))
321
322
def forward(
323
self,
@@ -319,19 +325,30 @@ class MixtralModel(nn.Module):
325
positions: torch.Tensor,
326
kv_caches: List[torch.Tensor],
327
attn_metadata: AttentionMetadata,
322
- ) -> torch.Tensor:
323
- hidden_states = self.embed_tokens(input_ids)
324
- residual = None
325
- for i in range(len(self.layers)):
328
+ intermediate_tensors: Optional[IntermediateTensors],
329
+ ) -> Union[torch.Tensor, IntermediateTensors]:
330
+ if get_pp_group().is_first_rank:
331
+ hidden_states = self.embed_tokens(input_ids)
332
+ residual = None
333
+ else:
334
+ assert intermediate_tensors is not None
335
+ hidden_states = intermediate_tensors["hidden_states"]
336
+ residual = intermediate_tensors["residual"]
337
+ for i in range(self.start_layer, self.end_layer):
338
layer = self.layers[i]
339
hidden_states, residual = layer(positions, hidden_states,
328
- kv_caches[i], attn_metadata,
329
- residual)
340
+ kv_caches[i - self.start_layer],
341
+ attn_metadata, residual)
342
+ if not get_pp_group().is_last_rank:
343
+ return IntermediateTensors({
344
+ "hidden_states": hidden_states,
345
+ "residual": residual
346
+ })
347
hidden_states, _ = self.norm(hidden_states, residual)
348
return hidden_states
349
350
334
-class MixtralForCausalLM(nn.Module):
351
+class MixtralForCausalLM(nn.Module, SupportsPP):
352
fall_back_to_pt_during_load = False
353
354
def __init__(
@@ -351,6 +368,8 @@ class MixtralForCausalLM(nn.Module):
368
self.lm_head.weight = self.model.embed_tokens.weight
369
self.logits_processor = LogitsProcessor(config.vocab_size)
370
self.sampler = Sampler()
371
+ self.make_empty_intermediate_tensors = (
372
+ self.model.make_empty_intermediate_tensors)
373
374
def forward(
375
self,
@@ -359,9 +378,9 @@ class MixtralForCausalLM(nn.Module):
378
kv_caches: List[torch.Tensor],
379
attn_metadata: AttentionMetadata,
380
intermediate_tensors: Optional[IntermediateTensors] = None,
362
- ) -> torch.Tensor:
381
+ ) -> Union[torch.Tensor, IntermediateTensors]:
382
hidden_states = self.model(input_ids, positions, kv_caches,
364
- attn_metadata)
383
+ attn_metadata, intermediate_tensors)
384
return hidden_states
385
386
def compute_logits(
@@ -400,6 +419,8 @@ class MixtralForCausalLM(nn.Module):
419
# Skip loading extra bias for GPTQ models.
420
if name.endswith(".bias") and name not in params_dict:
421
continue
422
+ if is_pp_missing_parameter(name, self):
423
+ continue
424
param = params_dict[name]
425
weight_loader = param.weight_loader
426
weight_loader(param, loaded_weight, shard_id)
@@ -412,6 +433,8 @@ class MixtralForCausalLM(nn.Module):
433
if ("block_sparse_moe.experts." in name
434
and name not in params_dict):
435
continue
436
+ if is_pp_missing_parameter(name, self):
437
+ continue
438
param = params_dict[name]
439
weight_loader = getattr(param, "weight_loader",
440
default_weight_loader)
vllm/model_executor/models/mpt.py
+34
-15
@@ -1,22 +1,21 @@
1
# coding=utf-8
2
# Adapted from https://huggingface.co/mosaicml/mpt-7b/tree/main
3
import math
4
-from typing import Iterable, List, Optional, Tuple
4
+from typing import Iterable, List, Optional, Tuple, Union
5
6
import torch
7
import torch.nn as nn
8
9
from vllm.attention import Attention, AttentionMetadata
10
from vllm.config import CacheConfig
11
-from vllm.distributed import (get_tensor_model_parallel_rank,
11
+from vllm.distributed import (get_pp_group, get_tensor_model_parallel_rank,
12
get_tensor_model_parallel_world_size)
13
from vllm.model_executor.layers.activation import get_act_fn
14
from vllm.model_executor.layers.linear import (ColumnParallelLinear,
15
QKVParallelLinear,
16
RowParallelLinear)
17
from vllm.model_executor.layers.logits_processor import LogitsProcessor
18
-from vllm.model_executor.layers.quantization.base_config import (
19
- QuantizationConfig)
18
+from vllm.model_executor.layers.quantization import QuantizationConfig
19
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
20
from vllm.model_executor.layers.vocab_parallel_embedding import (
21
VocabParallelEmbedding)
@@ -25,6 +24,10 @@ from vllm.model_executor.sampling_metadata import SamplingMetadata
24
from vllm.sequence import IntermediateTensors
25
from vllm.transformers_utils.configs.mpt import MPTConfig
26
27
+from .interfaces import SupportsPP
28
+from .utils import (is_pp_missing_parameter,
29
+ make_empty_intermediate_tensors_factory, make_layers)
30
+
31
32
def _get_alibi_slopes(
33
total_num_heads: int,
@@ -208,6 +211,7 @@ class MPTModel(nn.Module):
211
config: MPTConfig,
212
cache_config: Optional[CacheConfig] = None,
213
quant_config: Optional[QuantizationConfig] = None,
214
+ prefix: str = "",
215
):
216
super().__init__()
217
assert config.embedding_fraction == 1.0
@@ -217,10 +221,10 @@ class MPTModel(nn.Module):
221
config.vocab_size,
222
config.d_model,
223
)
220
- self.blocks = nn.ModuleList([
221
- MPTBlock(config, cache_config, quant_config)
222
- for _ in range(config.n_layers)
223
- ])
224
+ self.start_layer, self.end_layer, self.blocks = make_layers(
225
+ config.n_layers,
226
+ lambda prefix: MPTBlock(config, cache_config, quant_config),
227
+ prefix=f"{prefix}.blocks")
228
self.norm_f = nn.LayerNorm(config.d_model)
229
if config.no_bias:
230
for module in self.modules():
@@ -228,6 +232,9 @@ class MPTModel(nn.Module):
232
module.bias, nn.Parameter):
233
# Remove the bias term in Linear and LayerNorm.
234
module.register_parameter("bias", None)
235
+ self.make_empty_intermediate_tensors = (
236
+ make_empty_intermediate_tensors_factory(["hidden_states"],
237
+ config.d_model))
238
239
def forward(
240
self,
@@ -235,21 +242,29 @@ class MPTModel(nn.Module):
242
position_ids: torch.Tensor,
243
kv_caches: List[torch.Tensor],
244
attn_metadata: AttentionMetadata,
238
- ) -> torch.Tensor:
239
- hidden_states = self.wte(input_ids)
240
- for i in range(len(self.blocks)):
245
+ intermediate_tensors: Optional[IntermediateTensors],
246
+ ) -> Union[torch.Tensor, IntermediateTensors]:
247
+ if get_pp_group().is_first_rank:
248
+ hidden_states = self.wte(input_ids)
249
+ else:
250
+ assert intermediate_tensors is not None
251
+ hidden_states = intermediate_tensors["hidden_states"]
252
+
253
+ for i in range(self.start_layer, self.end_layer):
254
block = self.blocks[i]
255
hidden_states = block(
256
position_ids,
257
hidden_states,
245
- kv_caches[i],
258
+ kv_caches[i - self.start_layer],
259
attn_metadata,
260
)
261
+ if not get_pp_group().is_last_rank:
262
+ return IntermediateTensors({"hidden_states": hidden_states})
263
hidden_states = self.norm_f(hidden_states)
264
return hidden_states
265
266
252
-class MPTForCausalLM(nn.Module):
267
+class MPTForCausalLM(nn.Module, SupportsPP):
268
269
def __init__(
270
self,
@@ -266,6 +281,8 @@ class MPTForCausalLM(nn.Module):
281
self.lm_head = self.transformer.wte
282
self.logits_processor = LogitsProcessor(config.vocab_size)
283
self.sampler = Sampler()
284
+ self.make_empty_intermediate_tensors = (
285
+ self.transformer.make_empty_intermediate_tensors)
286
287
def forward(
288
self,
@@ -274,9 +291,9 @@ class MPTForCausalLM(nn.Module):
291
kv_caches: List[torch.Tensor],
292
attn_metadata: AttentionMetadata,
293
intermediate_tensors: Optional[IntermediateTensors] = None,
277
- ) -> torch.Tensor:
294
+ ) -> Union[torch.Tensor, IntermediateTensors]:
295
hidden_states = self.transformer(input_ids, positions, kv_caches,
279
- attn_metadata)
296
+ attn_metadata, intermediate_tensors)
297
return hidden_states
298
299
def compute_logits(
@@ -302,6 +319,8 @@ class MPTForCausalLM(nn.Module):
319
# Skip loading extra bias for GPTQ models.
320
if name.endswith(".bias") and name not in params_dict:
321
continue
322
+ if is_pp_missing_parameter(name, self):
323
+ continue
324
param = params_dict[name]
325
weight_loader = getattr(param, "weight_loader",
326
default_weight_loader)
vllm/model_executor/models/nemotron.py
+10
-19
@@ -34,8 +34,7 @@ from vllm.model_executor.layers.linear import (ColumnParallelLinear,
34
QKVParallelLinear,
35
RowParallelLinear)
36
from vllm.model_executor.layers.logits_processor import LogitsProcessor
37
-from vllm.model_executor.layers.quantization.base_config import (
38
- QuantizationConfig)
37
+from vllm.model_executor.layers.quantization import QuantizationConfig
38
from vllm.model_executor.layers.rotary_embedding import get_rope
39
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
40
from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -46,8 +45,9 @@ from vllm.model_executor.sampling_metadata import SamplingMetadata
45
from vllm.sequence import IntermediateTensors
46
from vllm.transformers_utils.configs import NemotronConfig
47
49
-from .interfaces import SupportsLoRA
50
-from .utils import PPMissingLayer, is_pp_missing_parameter, make_layers
48
+from .interfaces import SupportsLoRA, SupportsPP
49
+from .utils import (PPMissingLayer, is_pp_missing_parameter,
50
+ make_empty_intermediate_tensors_factory, make_layers)
51
52
# The architecture is pretty similar to Llama, with these changes:
53
# - There is no gate_proj, just up_proj
@@ -328,6 +328,9 @@ class NemotronModel(nn.Module):
328
eps=config.norm_eps)
329
else:
330
self.norm = PPMissingLayer()
331
+ self.make_empty_intermediate_tensors = (
332
+ make_empty_intermediate_tensors_factory(
333
+ ["hidden_states", "residual"], config.hidden_size))
334
335
def get_input_embeddings(self, input_ids: torch.Tensor) -> torch.Tensor:
336
return self.embed_tokens(input_ids)
@@ -372,7 +375,7 @@ class NemotronModel(nn.Module):
375
return hidden_states
376
377
375
-class NemotronForCausalLM(nn.Module, SupportsLoRA):
378
+class NemotronForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
379
packed_modules_mapping = {
380
"qkv_proj": [
381
"q_proj",
@@ -440,6 +443,8 @@ class NemotronForCausalLM(nn.Module, SupportsLoRA):
443
self.sampler = Sampler()
444
else:
445
self.lm_head = PPMissingLayer()
446
+ self.make_empty_intermediate_tensors = (
447
+ self.model.make_empty_intermediate_tensors)
448
449
def forward(
450
self,
@@ -470,20 +475,6 @@ class NemotronForCausalLM(nn.Module, SupportsLoRA):
475
next_tokens = self.sampler(logits, sampling_metadata)
476
return next_tokens
477
473
- def make_empty_intermediate_tensors(
474
- self, batch_size: int, dtype: torch.dtype,
475
- device: torch.device) -> IntermediateTensors:
476
- return IntermediateTensors({
477
- "hidden_states":
478
- torch.zeros((batch_size, self.config.hidden_size),
479
- dtype=dtype,
480
- device=device),
481
- "residual":
482
- torch.zeros((batch_size, self.config.hidden_size),
483
- dtype=dtype,
484
- device=device),
485
- })
486
-
478
def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
479
stacked_params_mapping = [
480
# (param_name, shard_name, shard_id)
vllm/model_executor/models/olmo.py
+42
-20
@@ -21,7 +21,7 @@
21
# See the License for the specific language governing permissions and
22
# limitations under the License.
23
"""Inference-only OLMo model compatible with HuggingFace weights."""
24
-from typing import Iterable, List, Optional, Tuple
24
+from typing import Iterable, List, Optional, Tuple, Union
25
26
import torch
27
from torch import nn
@@ -29,14 +29,13 @@ from transformers import OlmoConfig
29
30
from vllm.attention import Attention, AttentionMetadata
31
from vllm.config import CacheConfig
32
-from vllm.distributed import get_tensor_model_parallel_world_size
32
+from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
33
from vllm.model_executor.layers.activation import SiluAndMul
34
from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
35
QKVParallelLinear,
36
RowParallelLinear)
37
from vllm.model_executor.layers.logits_processor import LogitsProcessor
38
-from vllm.model_executor.layers.quantization.base_config import (
39
- QuantizationConfig)
38
+from vllm.model_executor.layers.quantization import QuantizationConfig
39
from vllm.model_executor.layers.rotary_embedding import get_rope
40
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
41
from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -45,6 +44,10 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
44
from vllm.model_executor.sampling_metadata import SamplingMetadata
45
from vllm.sequence import IntermediateTensors
46
47
+from .interfaces import SupportsPP
48
+from .utils import (is_pp_missing_parameter,
49
+ make_empty_intermediate_tensors_factory, make_layers)
50
+
51
52
class OlmoAttention(nn.Module):
53
"""
@@ -223,19 +226,24 @@ class OlmoModel(nn.Module):
226
def __init__(self,
227
config: OlmoConfig,
228
cache_config: Optional[CacheConfig] = None,
226
- quant_config: Optional[QuantizationConfig] = None):
229
+ quant_config: Optional[QuantizationConfig] = None,
230
+ prefix: str = ""):
231
super().__init__()
232
self.config = config
233
234
self.embed_tokens = VocabParallelEmbedding(config.vocab_size,
235
config.hidden_size)
232
- self.layers = nn.ModuleList([
233
- OlmoDecoderLayer(config, cache_config, quant_config)
234
- for layer_idx in range(config.num_hidden_layers)
235
- ])
236
+ self.start_layer, self.end_layer, self.layers = make_layers(
237
+ config.num_hidden_layers,
238
+ lambda prefix: OlmoDecoderLayer(config, cache_config, quant_config
239
+ ),
240
+ prefix=f"{prefix}.layers")
241
self.norm = nn.LayerNorm(config.hidden_size,
242
elementwise_affine=False,
243
bias=False)
244
+ self.make_empty_intermediate_tensors = (
245
+ make_empty_intermediate_tensors_factory(["hidden_states"],
246
+ config.hidden_size))
247
248
def forward(
249
self,
@@ -243,34 +251,41 @@ class OlmoModel(nn.Module):
251
positions: torch.Tensor,
252
kv_caches: List[torch.Tensor],
253
attn_metadata: AttentionMetadata,
246
- ) -> torch.Tensor:
254
+ intermediate_tensors: Optional[IntermediateTensors],
255
+ ) -> Union[torch.Tensor, IntermediateTensors]:
256
"""
257
:param input_ids: A tensor of shape `(batch_size, seq_len)`.
258
"""
250
- # Get embeddings of input.
251
- # shape: (batch_size, seq_len, d_model)
252
- inputs_embeds = self.embed_tokens(input_ids)
259
+ if get_pp_group().is_first_rank:
260
+ # Get embeddings of input.
261
+ # shape: (batch_size, seq_len, d_model)
262
+ inputs_embeds = self.embed_tokens(input_ids)
263
254
- # embed positions
255
- hidden_states = inputs_embeds
264
+ # embed positions
265
+ hidden_states = inputs_embeds
266
+ else:
267
+ assert intermediate_tensors is not None
268
+ hidden_states = intermediate_tensors["hidden_states"]
269
270
# Apply blocks one-by-one.
258
- for layer_idx, decoder_layer in enumerate(self.layers):
271
+ for i in range(self.start_layer, self.end_layer):
272
# shape: (batch_size, seq_len, d_model)
260
- hidden_states = decoder_layer(
273
+ hidden_states = self.layers[i](
274
positions,
275
hidden_states,
263
- kv_caches[layer_idx],
276
+ kv_caches[i - self.start_layer],
277
attn_metadata,
278
)
279
280
+ if not get_pp_group().is_last_rank:
281
+ return IntermediateTensors({"hidden_states": hidden_states})
282
# Apply final layer norm.
283
# shape: (batch_size, seq_len or 1, d_model)
284
hidden_states = self.norm(hidden_states)
285
return hidden_states
286
287
273
-class OlmoForCausalLM(nn.Module):
288
+class OlmoForCausalLM(nn.Module, SupportsPP):
289
"""
290
Extremely barebones HF model wrapper.
291
"""
@@ -294,6 +309,8 @@ class OlmoForCausalLM(nn.Module):
309
)
310
self.logits_processor = LogitsProcessor(config.vocab_size)
311
self.sampler = Sampler()
312
+ self.make_empty_intermediate_tensors = (
313
+ self.model.make_empty_intermediate_tensors)
314
315
def forward(
316
self,
@@ -302,12 +319,13 @@ class OlmoForCausalLM(nn.Module):
319
kv_caches: List[torch.Tensor],
320
attn_metadata: AttentionMetadata,
321
intermediate_tensors: Optional[IntermediateTensors] = None,
305
- ) -> torch.Tensor:
322
+ ) -> Union[torch.Tensor, IntermediateTensors]:
323
hidden_states = self.model(
324
input_ids=input_ids,
325
positions=positions,
326
kv_caches=kv_caches,
327
attn_metadata=attn_metadata,
328
+ intermediate_tensors=intermediate_tensors,
329
)
330
return hidden_states
331
@@ -358,6 +376,8 @@ class OlmoForCausalLM(nn.Module):
376
# Skip loading extra bias for GPTQ models.
377
if name.endswith(".bias") and name not in params_dict:
378
continue
379
+ if is_pp_missing_parameter(name, self):
380
+ continue
381
param = params_dict[name]
382
weight_loader = param.weight_loader
383
weight_loader(param, loaded_weight, shard_id)
@@ -366,6 +386,8 @@ class OlmoForCausalLM(nn.Module):
386
# Skip loading extra bias for GPTQ models.
387
if name.endswith(".bias") and name not in params_dict:
388
continue
389
+ if is_pp_missing_parameter(name, self):
390
+ continue
391
param = params_dict[name]
392
weight_loader = getattr(param, "weight_loader",
393
default_weight_loader)
vllm/model_executor/models/olmoe.py
+57
-21
@@ -10,7 +10,7 @@
10
# See the License for the specific language governing permissions and
11
# limitations under the License.
12
"""Inference-only OLMoE model compatible with HuggingFace weights."""
13
-from typing import Any, Dict, Iterable, List, Optional, Tuple
13
+from typing import Any, Dict, Iterable, List, Optional, Tuple, Union
14
15
import torch
16
from torch import nn
@@ -18,15 +18,14 @@ from transformers import PretrainedConfig
18
19
from vllm.attention import Attention, AttentionMetadata
20
from vllm.config import CacheConfig
21
-from vllm.distributed import get_tensor_model_parallel_world_size
21
+from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
22
from vllm.model_executor.layers.fused_moe import FusedMoE
23
from vllm.model_executor.layers.layernorm import RMSNorm
24
from vllm.model_executor.layers.linear import (QKVParallelLinear,
25
ReplicatedLinear,
26
RowParallelLinear)
27
from vllm.model_executor.layers.logits_processor import LogitsProcessor
28
-from vllm.model_executor.layers.quantization.base_config import (
29
- QuantizationConfig)
28
+from vllm.model_executor.layers.quantization import QuantizationConfig
29
from vllm.model_executor.layers.rotary_embedding import get_rope
30
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
31
from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -36,6 +35,10 @@ from vllm.model_executor.sampling_metadata import SamplingMetadata
35
from vllm.sequence import IntermediateTensors
36
from vllm.utils import print_warning_once
37
38
+from .interfaces import SupportsPP
39
+from .utils import (is_pp_missing_parameter,
40
+ make_empty_intermediate_tensors_factory, make_layers)
41
+
42
43
class OlmoeMoE(nn.Module):
44
"""A tensor-parallel MoE implementation for Olmoe that shards each expert
@@ -243,6 +246,7 @@ class OlmoeModel(nn.Module):
246
config: PretrainedConfig,
247
cache_config: Optional[CacheConfig] = None,
248
quant_config: Optional[QuantizationConfig] = None,
249
+ prefix: str = "",
250
) -> None:
251
super().__init__()
252
self.padding_idx = config.pad_token_id
@@ -252,34 +256,54 @@ class OlmoeModel(nn.Module):
256
config.vocab_size,
257
config.hidden_size,
258
)
255
- self.layers = nn.ModuleList([
256
- OlmoeDecoderLayer(config,
257
- layer_idx,
258
- cache_config,
259
- quant_config=quant_config)
260
- for layer_idx in range(config.num_hidden_layers)
261
- ])
259
+ self.start_layer, self.end_layer, self.layers = make_layers(
260
+ config.num_hidden_layers,
261
+ lambda prefix: OlmoeDecoderLayer(config, int(
262
+ prefix.split(".")[-1]), cache_config, quant_config),
263
+ prefix=f"{prefix}.layers")
264
self.norm = RMSNorm(config.hidden_size, eps=1e-5)
265
266
+ self.make_empty_intermediate_tensors = (
267
+ make_empty_intermediate_tensors_factory(
268
+ ["hidden_states", "residual"], config.hidden_size))
269
+
270
def forward(
271
self,
272
input_ids: torch.Tensor,
273
positions: torch.Tensor,
274
kv_caches: List[torch.Tensor],
275
attn_metadata: AttentionMetadata,
270
- ) -> torch.Tensor:
271
- hidden_states = self.embed_tokens(input_ids)
272
- residual = None
273
- for i in range(len(self.layers)):
276
+ intermediate_tensors: Optional[IntermediateTensors],
277
+ ) -> Union[torch.Tensor, IntermediateTensors]:
278
+ if get_pp_group().is_first_rank:
279
+ hidden_states = self.embed_tokens(input_ids)
280
+ residual = None
281
+ else:
282
+ assert intermediate_tensors is not None
283
+ hidden_states = intermediate_tensors["hidden_states"]
284
+ residual = intermediate_tensors["residual"]
285
+
286
+ for i in range(self.start_layer, self.end_layer):
287
layer = self.layers[i]
275
- hidden_states, residual = layer(positions, hidden_states,
276
- kv_caches[i], attn_metadata,
277
- residual)
288
+ hidden_states, residual = layer(
289
+ positions,
290
+ hidden_states,
291
+ kv_caches[i - self.start_layer],
292
+ attn_metadata,
293
+ residual,
294
+ )
295
+
296
+ if not get_pp_group().is_last_rank:
297
+ return IntermediateTensors({
298
+ "hidden_states": hidden_states,
299
+ "residual": residual
300
+ })
301
+
302
hidden_states, _ = self.norm(hidden_states, residual)
303
return hidden_states
304
305
282
-class OlmoeForCausalLM(nn.Module):
306
+class OlmoeForCausalLM(nn.Module, SupportsPP):
307
308
fall_back_to_pt_during_load = False
309
@@ -299,6 +323,9 @@ class OlmoeForCausalLM(nn.Module):
323
self.logits_processor = LogitsProcessor(config.vocab_size)
324
self.sampler = Sampler()
325
326
+ self.make_empty_intermediate_tensors = (
327
+ self.model.make_empty_intermediate_tensors)
328
+
329
def forward(
330
self,
331
input_ids: torch.Tensor,
@@ -306,9 +333,9 @@ class OlmoeForCausalLM(nn.Module):
333
kv_caches: List[torch.Tensor],
334
attn_metadata: AttentionMetadata,
335
intermediate_tensors: Optional[IntermediateTensors] = None,
309
- ) -> torch.Tensor:
336
+ ) -> Union[torch.Tensor, IntermediateTensors]:
337
hidden_states = self.model(input_ids, positions, kv_caches,
311
- attn_metadata)
338
+ attn_metadata, intermediate_tensors)
339
return hidden_states
340
341
def compute_logits(self, hidden_states: torch.Tensor,
@@ -363,6 +390,9 @@ class OlmoeForCausalLM(nn.Module):
390
# Skip loading extra bias for GPTQ models.
391
if name.endswith(".bias") and name not in params_dict:
392
continue
393
+ # Skip layers on other devices.
394
+ if is_pp_missing_parameter(name, self):
395
+ continue
396
if name not in params_dict:
397
continue
398
@@ -376,6 +406,9 @@ class OlmoeForCausalLM(nn.Module):
406
if weight_name not in name:
407
continue
408
name = name.replace(weight_name, param_name)
409
+ # Skip layers on other devices.
410
+ if is_pp_missing_parameter(name, self):
411
+ continue
412
param = params_dict[name]
413
weight_loader = param.weight_loader
414
weight_loader(param,
@@ -388,6 +421,9 @@ class OlmoeForCausalLM(nn.Module):
421
# Skip loading extra bias for GPTQ models.
422
if name.endswith(".bias") and name not in params_dict:
423
continue
424
+ # Skip layers on other devices.
425
+ if is_pp_missing_parameter(name, self):
426
+ continue
427
# Remapping the name of FP8 kv-scale.
428
if name.endswith("kv_scale"):
429
remapped_kv_scale_name = name.replace(
vllm/model_executor/models/opt.py
+47
-23
@@ -17,7 +17,7 @@
17
# See the License for the specific language governing permissions and
18
# limitations under the License.
19
"""Inference-only OPT model compatible with HuggingFace weights."""
20
-from typing import Iterable, List, Optional, Tuple
20
+from typing import Iterable, List, Optional, Tuple, Union
21
22
import torch
23
from torch import nn
@@ -25,15 +25,14 @@ from transformers import OPTConfig
25
26
from vllm.attention import Attention, AttentionMetadata
27
from vllm.config import CacheConfig
28
-from vllm.distributed import get_tensor_model_parallel_world_size
28
+from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
29
from vllm.model_executor.layers.activation import get_act_fn
30
from vllm.model_executor.layers.linear import (ColumnParallelLinear,
31
QKVParallelLinear,
32
ReplicatedLinear,
33
RowParallelLinear)
34
from vllm.model_executor.layers.logits_processor import LogitsProcessor
35
-from vllm.model_executor.layers.quantization.base_config import (
36
- QuantizationConfig)
35
+from vllm.model_executor.layers.quantization import QuantizationConfig
36
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
37
from vllm.model_executor.layers.vocab_parallel_embedding import (
38
ParallelLMHead, VocabParallelEmbedding)
@@ -41,6 +40,10 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
40
from vllm.model_executor.sampling_metadata import SamplingMetadata
41
from vllm.sequence import IntermediateTensors
42
43
+from .interfaces import SupportsPP
44
+from .utils import (is_pp_missing_parameter,
45
+ make_empty_intermediate_tensors_factory, make_layers)
46
+
47
48
class OPTLearnedPositionalEmbedding(nn.Embedding):
49
@@ -189,6 +192,7 @@ class OPTDecoder(nn.Module):
192
config: OPTConfig,
193
cache_config: Optional[CacheConfig] = None,
194
quant_config: Optional[QuantizationConfig] = None,
195
+ prefix: str = "",
196
):
197
super().__init__()
198
self.config = config
@@ -232,10 +236,10 @@ class OPTDecoder(nn.Module):
236
else:
237
self.final_layer_norm = None
238
235
- self.layers = nn.ModuleList([
236
- OPTDecoderLayer(config, cache_config, quant_config)
237
- for _ in range(config.num_hidden_layers)
238
- ])
239
+ self.start_layer, self.end_layer, self.layers = make_layers(
240
+ config.num_hidden_layers,
241
+ lambda prefix: OPTDecoderLayer(config, cache_config, quant_config),
242
+ prefix=f"{prefix}.layers")
243
244
def get_input_embeddings(self, input_ids: torch.Tensor) -> torch.Tensor:
245
return self.embed_tokens(input_ids)
@@ -246,19 +250,28 @@ class OPTDecoder(nn.Module):
250
positions: torch.Tensor,
251
kv_caches: List[torch.Tensor],
252
attn_metadata: AttentionMetadata,
253
+ intermediate_tensors: Optional[IntermediateTensors],
254
inputs_embeds: Optional[torch.Tensor] = None,
250
- ) -> torch.Tensor:
251
- if inputs_embeds is None:
252
- inputs_embeds = self.get_input_embeddings(input_ids)
253
- pos_embeds = self.embed_positions(positions)
254
- if self.project_in is not None:
255
- inputs_embeds, _ = self.project_in(inputs_embeds)
256
- hidden_states = inputs_embeds + pos_embeds
257
-
258
- for i in range(len(self.layers)):
255
+ ) -> Union[torch.Tensor, IntermediateTensors]:
256
+ if get_pp_group().is_first_rank:
257
+ if inputs_embeds is None:
258
+ inputs_embeds = self.get_input_embeddings(input_ids)
259
+ pos_embeds = self.embed_positions(positions)
260
+ if self.project_in is not None:
261
+ inputs_embeds, _ = self.project_in(inputs_embeds)
262
+ hidden_states = inputs_embeds + pos_embeds
263
+ else:
264
+ assert intermediate_tensors is not None
265
+ hidden_states = intermediate_tensors["hidden_states"]
266
+
267
+ for i in range(self.start_layer, self.end_layer):
268
layer = self.layers[i]
260
- hidden_states = layer(hidden_states, kv_caches[i], attn_metadata)
269
+ hidden_states = layer(hidden_states,
270
+ kv_caches[i - self.start_layer],
271
+ attn_metadata)
272
273
+ if not get_pp_group().is_last_rank:
274
+ return IntermediateTensors({"hidden_states": hidden_states})
275
if self.final_layer_norm is not None:
276
hidden_states = self.final_layer_norm(hidden_states)
277
if self.project_out is not None:
@@ -276,6 +289,9 @@ class OPTModel(nn.Module):
289
):
290
super().__init__()
291
self.decoder = OPTDecoder(config, cache_config, quant_config)
292
+ self.make_empty_intermediate_tensors = (
293
+ make_empty_intermediate_tensors_factory(["hidden_states"],
294
+ config.hidden_size))
295
296
def get_input_embeddings(self, input_ids: torch.Tensor) -> torch.Tensor:
297
return self.decoder.get_input_embeddings(input_ids)
@@ -286,20 +302,22 @@ class OPTModel(nn.Module):
302
positions: torch.Tensor,
303
kv_caches: List[torch.Tensor],
304
attn_metadata: AttentionMetadata,
305
+ intermediate_tensors: Optional[IntermediateTensors],
306
inputs_embeds: Optional[torch.Tensor] = None,
290
- ) -> torch.Tensor:
307
+ ) -> Union[torch.Tensor, IntermediateTensors]:
308
return self.decoder(input_ids,
309
positions,
310
kv_caches,
311
attn_metadata,
312
+ intermediate_tensors,
313
inputs_embeds=inputs_embeds)
314
315
298
-class OPTForCausalLM(nn.Module):
316
+class OPTForCausalLM(nn.Module, SupportsPP):
317
318
def __init__(
319
self,
302
- config,
320
+ config: OPTConfig,
321
cache_config: Optional[CacheConfig] = None,
322
quant_config: Optional[QuantizationConfig] = None,
323
):
@@ -314,6 +332,8 @@ class OPTForCausalLM(nn.Module):
332
config.word_embed_proj_dim)
333
self.logits_processor = LogitsProcessor(config.vocab_size)
334
self.sampler = Sampler()
335
+ self.make_empty_intermediate_tensors = (
336
+ self.model.make_empty_intermediate_tensors)
337
338
def forward(
339
self,
@@ -322,9 +342,9 @@ class OPTForCausalLM(nn.Module):
342
kv_caches: List[torch.Tensor],
343
attn_metadata: AttentionMetadata,
344
intermediate_tensors: Optional[IntermediateTensors] = None,
325
- ) -> torch.Tensor:
345
+ ) -> Union[torch.Tensor, IntermediateTensors]:
346
hidden_states = self.model(input_ids, positions, kv_caches,
327
- attn_metadata)
347
+ attn_metadata, intermediate_tensors)
348
return hidden_states
349
350
def compute_logits(
@@ -365,6 +385,8 @@ class OPTForCausalLM(nn.Module):
385
# Skip loading extra bias for GPTQ models.
386
if name.endswith(".bias") and name not in params_dict:
387
continue
388
+ if is_pp_missing_parameter(name, self):
389
+ continue
390
param = params_dict[name]
391
weight_loader = param.weight_loader
392
weight_loader(param, loaded_weight, shard_id)
@@ -373,6 +395,8 @@ class OPTForCausalLM(nn.Module):
395
# Skip loading extra bias for GPTQ models.
396
if name.endswith(".bias") and name not in params_dict:
397
continue
398
+ if is_pp_missing_parameter(name, self):
399
+ continue
400
param = params_dict[name]
401
weight_loader = getattr(param, "weight_loader",
402
default_weight_loader)
vllm/model_executor/models/orion.py
+44
-16
@@ -4,7 +4,7 @@
4
# Copyright (c) OrionStar Inc.
5
# LICENSE: https://huggingface.co/OrionStarAI/Orion-14B-Base/blob/main/LICENSE
6
"""Inference-only Orion-14B model compatible with HuggingFace weights."""
7
-from typing import Any, Dict, Iterable, List, Optional, Tuple
7
+from typing import Any, Dict, Iterable, List, Optional, Tuple, Union
8
9
import torch
10
from torch import nn
@@ -12,14 +12,13 @@ from transformers import PretrainedConfig
12
13
from vllm.attention import Attention, AttentionMetadata
14
from vllm.config import CacheConfig
15
-from vllm.distributed import get_tensor_model_parallel_world_size
15
+from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
16
from vllm.model_executor.layers.activation import SiluAndMul
17
from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
18
QKVParallelLinear,
19
RowParallelLinear)
20
from vllm.model_executor.layers.logits_processor import LogitsProcessor
21
-from vllm.model_executor.layers.quantization.base_config import (
22
- QuantizationConfig)
21
+from vllm.model_executor.layers.quantization import QuantizationConfig
22
from vllm.model_executor.layers.rotary_embedding import get_rope
23
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
24
from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -28,6 +27,10 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
27
from vllm.model_executor.sampling_metadata import SamplingMetadata
28
from vllm.sequence import IntermediateTensors
29
30
+from .interfaces import SupportsPP
31
+from .utils import (is_pp_missing_parameter,
32
+ make_empty_intermediate_tensors_factory, make_layers)
33
+
34
35
class OrionMLP(nn.Module):
36
@@ -210,6 +213,7 @@ class OrionModel(nn.Module):
213
config: PretrainedConfig,
214
cache_config: Optional[CacheConfig] = None,
215
quant_config: Optional[QuantizationConfig] = None,
216
+ prefix: str = "",
217
) -> None:
218
super().__init__()
219
self.config = config
@@ -219,11 +223,18 @@ class OrionModel(nn.Module):
223
config.vocab_size,
224
config.hidden_size,
225
)
222
- self.layers = nn.ModuleList([
223
- OrionDecoderLayer(config, cache_config, quant_config)
224
- for _ in range(config.num_hidden_layers)
225
- ])
226
+ self.start_layer, self.end_layer, self.layers = make_layers(
227
+ config.num_hidden_layers,
228
+ lambda prefix: OrionDecoderLayer(
229
+ config,
230
+ cache_config,
231
+ quant_config,
232
+ ),
233
+ prefix=f"{prefix}.layers")
234
self.norm = nn.LayerNorm(config.hidden_size, eps=config.rms_norm_eps)
235
+ self.make_empty_intermediate_tensors = (
236
+ make_empty_intermediate_tensors_factory(
237
+ ["hidden_states", "residual"], config.hidden_size))
238
239
def forward(
240
self,
@@ -231,23 +242,34 @@ class OrionModel(nn.Module):
242
positions: torch.Tensor,
243
kv_caches: List[torch.Tensor],
244
attn_metadata: AttentionMetadata,
234
- ) -> torch.Tensor:
235
- hidden_states = self.embed_tokens(input_ids)
236
- residual = None
237
- for i in range(len(self.layers)):
245
+ intermediate_tensors: Optional[IntermediateTensors],
246
+ ) -> Union[torch.Tensor, IntermediateTensors]:
247
+ if get_pp_group().is_first_rank:
248
+ hidden_states = self.embed_tokens(input_ids)
249
+ residual = None
250
+ else:
251
+ assert intermediate_tensors
252
+ hidden_states = intermediate_tensors["hidden_states"]
253
+ residual = intermediate_tensors["residual"]
254
+ for i in range(self.start_layer, self.end_layer):
255
layer = self.layers[i]
256
hidden_states, residual = layer(
257
positions,
258
hidden_states,
242
- kv_caches[i],
259
+ kv_caches[i - self.start_layer],
260
attn_metadata,
261
residual,
262
)
263
+ if not get_pp_group().is_last_rank:
264
+ return IntermediateTensors({
265
+ "hidden_states": hidden_states,
266
+ "residual": residual
267
+ })
268
hidden_states = self.norm(hidden_states)
269
return hidden_states
270
271
250
-class OrionForCausalLM(nn.Module):
272
+class OrionForCausalLM(nn.Module, SupportsPP):
273
274
def __init__(
275
self,
@@ -266,6 +288,8 @@ class OrionForCausalLM(nn.Module):
288
self.lm_head.weight = self.model.embed_tokens.weight
289
self.logits_processor = LogitsProcessor(config.vocab_size)
290
self.sampler = Sampler()
291
+ self.make_empty_intermediate_tensors = (
292
+ self.model.make_empty_intermediate_tensors)
293
294
def forward(
295
self,
@@ -274,9 +298,9 @@ class OrionForCausalLM(nn.Module):
298
kv_caches: List[torch.Tensor],
299
attn_metadata: AttentionMetadata,
300
intermediate_tensors: Optional[IntermediateTensors] = None,
277
- ) -> torch.Tensor:
301
+ ) -> Union[torch.Tensor, IntermediateTensors]:
302
hidden_states = self.model(input_ids, positions, kv_caches,
279
- attn_metadata)
303
+ attn_metadata, intermediate_tensors)
304
return hidden_states
305
306
def compute_logits(
@@ -321,6 +345,8 @@ class OrionForCausalLM(nn.Module):
345
# Skip loading extra bias for GPTQ models.
346
if name.endswith(".bias") and name not in params_dict:
347
continue
348
+ if is_pp_missing_parameter(name, self):
349
+ continue
350
param = params_dict[name]
351
weight_loader = param.weight_loader
352
weight_loader(param, loaded_weight, shard_id)
@@ -329,6 +355,8 @@ class OrionForCausalLM(nn.Module):
355
# Skip loading extra bias for GPTQ models.
356
if name.endswith(".bias") and name not in params_dict:
357
continue
358
+ if is_pp_missing_parameter(name, self):
359
+ continue
360
param = params_dict[name]
361
weight_loader = getattr(param, "weight_loader",
362
default_weight_loader)
vllm/model_executor/models/paligemma.py
+33
-26
@@ -9,9 +9,8 @@ from vllm.attention import AttentionMetadata
9
from vllm.config import CacheConfig, MultiModalConfig
10
from vllm.inputs import INPUT_REGISTRY, InputContext, LLMInputs
11
from vllm.logger import init_logger
12
-from vllm.model_executor.layers.logits_processor import LogitsProcessor
12
from vllm.model_executor.layers.quantization import QuantizationConfig
14
-from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
13
+from vllm.model_executor.layers.sampler import SamplerOutput
14
from vllm.model_executor.model_loader.weight_utils import default_weight_loader
15
from vllm.model_executor.models.gemma import GemmaForCausalLM
16
from vllm.model_executor.sampling_metadata import SamplingMetadata
@@ -19,7 +18,7 @@ from vllm.multimodal import MULTIMODAL_REGISTRY
18
from vllm.multimodal.utils import cached_get_tokenizer
19
from vllm.sequence import IntermediateTensors
20
22
-from .interfaces import SupportsMultiModal
21
+from .interfaces import SupportsMultiModal, SupportsPP
22
from .siglip import (SiglipVisionModel, dummy_image_for_siglip,
23
dummy_seq_data_for_siglip, get_max_siglip_image_tokens)
24
from .utils import group_weights_with_prefix, merge_multimodal_embeddings
@@ -129,7 +128,8 @@ class PaliGemmaMultiModalProjector(nn.Module):
128
@MULTIMODAL_REGISTRY.register_max_image_tokens(get_max_paligemma_image_tokens)
129
@INPUT_REGISTRY.register_dummy_data(dummy_data_for_paligemma)
130
@INPUT_REGISTRY.register_input_processor(input_processor_for_paligemma)
132
-class PaliGemmaForConditionalGeneration(nn.Module, SupportsMultiModal):
131
+class PaliGemmaForConditionalGeneration(nn.Module, SupportsMultiModal,
132
+ SupportsPP):
133
134
def __init__(self,
135
config: PaliGemmaConfig,
@@ -149,12 +149,15 @@ class PaliGemmaForConditionalGeneration(nn.Module, SupportsMultiModal):
149
self.quant_config = quant_config
150
self.language_model = GemmaForCausalLM(config.text_config,
151
cache_config, quant_config)
152
- self.unpadded_vocab_size = config.text_config.vocab_size
152
logit_scale = getattr(config, "logit_scale", 1.0)
154
- self.logits_processor = LogitsProcessor(self.unpadded_vocab_size,
155
- config.text_config.vocab_size,
156
- logit_scale)
157
- self.sampler = Sampler()
153
+ self.language_model.logits_processor.scale *= logit_scale
154
+
155
+ self.make_empty_intermediate_tensors = (
156
+ self.language_model.make_empty_intermediate_tensors)
157
+
158
+ @property
159
+ def sampler(self):
160
+ return self.language_model.sampler
161
162
def _validate_pixel_values(self, data: torch.Tensor) -> torch.Tensor:
163
h = w = self.config.vision_config.image_size
@@ -239,32 +242,36 @@ class PaliGemmaForConditionalGeneration(nn.Module, SupportsMultiModal):
242
kv_caches: List[torch.Tensor],
243
attn_metadata: AttentionMetadata,
244
intermediate_tensors: Optional[IntermediateTensors] = None,
242
- **kwargs: object) -> SamplerOutput:
243
-
244
- parsed_image_input = self._parse_and_validate_image_input(**kwargs)
245
+ **kwargs: object) -> Union[SamplerOutput, IntermediateTensors]:
246
+ if intermediate_tensors is not None:
247
+ input_ids = None
248
+ inputs_embeds = None
249
+ else:
250
+ parsed_image_input = self._parse_and_validate_image_input(**kwargs)
251
246
- if parsed_image_input is not None:
247
- vision_embeddings = self._process_image_input(parsed_image_input)
248
- # https://github.com/huggingface/transformers/blob/main/src/transformers/models/paligemma/modeling_paligemma.py#L294 # noqa
249
- vision_embeddings = vision_embeddings * (self.config.hidden_size**
250
- -0.5)
252
+ if parsed_image_input is not None:
253
+ vision_embeddings = self._process_image_input(
254
+ parsed_image_input)
255
+ # https://github.com/huggingface/transformers/blob/main/src/transformers/models/paligemma/modeling_paligemma.py#L294 # noqa
256
+ vision_embeddings = vision_embeddings * (
257
+ self.config.hidden_size**-0.5)
258
252
- inputs_embeds = self.language_model.model.get_input_embeddings(
253
- input_ids)
259
+ inputs_embeds = self.language_model.model.get_input_embeddings(
260
+ input_ids)
261
255
- inputs_embeds = merge_multimodal_embeddings(
256
- input_ids, inputs_embeds, vision_embeddings,
257
- self.config.image_token_index)
262
+ inputs_embeds = merge_multimodal_embeddings(
263
+ input_ids, inputs_embeds, vision_embeddings,
264
+ self.config.image_token_index)
265
259
- input_ids = None
260
- else:
261
- inputs_embeds = None
266
+ input_ids = None
267
+ else:
268
+ inputs_embeds = None
269
270
hidden_states = self.language_model.model(input_ids,
271
positions,
272
kv_caches,
273
attn_metadata,
267
- None,
274
+ intermediate_tensors,
275
inputs_embeds=inputs_embeds)
276
277
return hidden_states
vllm/model_executor/models/persimmon.py
+36
-18
@@ -20,7 +20,7 @@
20
# See the License for the specific language governing permissions and
21
# limitations under the License.
22
"""Inference-only persimmon model compatible with HuggingFace weights."""
23
-from typing import Iterable, List, Optional, Tuple
23
+from typing import Iterable, List, Optional, Tuple, Union
24
25
import torch
26
from torch import nn
@@ -28,14 +28,13 @@ from transformers import PersimmonConfig
28
29
from vllm.attention import Attention, AttentionMetadata
30
from vllm.config import CacheConfig
31
-from vllm.distributed import get_tensor_model_parallel_world_size
31
+from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
32
from vllm.model_executor.layers.activation import get_act_fn
33
from vllm.model_executor.layers.linear import (ColumnParallelLinear,
34
QKVParallelLinear,
35
RowParallelLinear)
36
from vllm.model_executor.layers.logits_processor import LogitsProcessor
37
-from vllm.model_executor.layers.quantization.base_config import (
38
- QuantizationConfig)
37
+from vllm.model_executor.layers.quantization import QuantizationConfig
38
from vllm.model_executor.layers.rotary_embedding import get_rope
39
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
40
from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -44,6 +43,10 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
43
from vllm.model_executor.sampling_metadata import SamplingMetadata
44
from vllm.sequence import IntermediateTensors
45
46
+from .interfaces import SupportsPP
47
+from .utils import (is_pp_missing_parameter,
48
+ make_empty_intermediate_tensors_factory, make_layers)
49
+
50
51
class PersimmonMLP(nn.Module):
52
@@ -211,20 +214,23 @@ class PersimmonModel(nn.Module):
214
def __init__(self,
215
config: PersimmonConfig,
216
cache_config: Optional[CacheConfig] = None,
214
- quant_config: Optional[QuantizationConfig] = None):
217
+ quant_config: Optional[QuantizationConfig] = None,
218
+ prefix: str = ""):
219
super().__init__()
220
self.vocab_size = config.vocab_size
221
222
self.embed_tokens = VocabParallelEmbedding(config.vocab_size,
223
config.hidden_size)
220
- self.layers = nn.ModuleList([
221
- PersimmonDecoderLayer(config,
222
- cache_config=cache_config,
223
- quant_config=quant_config)
224
- for _ in range(config.num_hidden_layers)
225
- ])
224
+ self.start_layer, self.end_layer, self.layers = make_layers(
225
+ config.num_hidden_layers,
226
+ lambda prefix: PersimmonDecoderLayer(config, cache_config,
227
+ quant_config),
228
+ prefix=f"{prefix}.layers")
229
self.final_layernorm = nn.LayerNorm(config.hidden_size,
230
eps=config.layer_norm_eps)
231
+ self.make_empty_intermediate_tensors = (
232
+ make_empty_intermediate_tensors_factory(["hidden_states"],
233
+ config.hidden_size))
234
235
def forward(
236
self,
@@ -232,24 +238,31 @@ class PersimmonModel(nn.Module):
238
positions: torch.Tensor,
239
kv_caches: List[torch.Tensor],
240
attn_metadata: AttentionMetadata,
241
+ intermediate_tensors: Optional[IntermediateTensors],
242
inputs_embeds: Optional[torch.Tensor] = None,
236
- ) -> torch.Tensor:
237
- if inputs_embeds is not None:
238
- hidden_states = inputs_embeds
243
+ ) -> Union[torch.Tensor, IntermediateTensors]:
244
+ if get_pp_group().is_first_rank:
245
+ if inputs_embeds is not None:
246
+ hidden_states = inputs_embeds
247
+ else:
248
+ hidden_states = self.embed_tokens(input_ids)
249
else:
240
- hidden_states = self.embed_tokens(input_ids)
241
- for i in range(len(self.layers)):
250
+ assert intermediate_tensors is not None
251
+ hidden_states = intermediate_tensors["hidden_states"]
252
+ for i in range(self.start_layer, self.end_layer):
253
hidden_states = self.layers[i](
254
positions,
255
hidden_states,
245
- kv_caches[i],
256
+ kv_caches[i - self.start_layer],
257
attn_metadata,
258
)
259
+ if not get_pp_group().is_last_rank:
260
+ return IntermediateTensors({"hidden_states": hidden_states})
261
hidden_states = self.final_layernorm(hidden_states)
262
return hidden_states
263
264
252
-class PersimmonForCausalLM(nn.Module):
265
+class PersimmonForCausalLM(nn.Module, SupportsPP):
266
267
def __init__(self,
268
config: PersimmonConfig,
@@ -266,6 +279,8 @@ class PersimmonForCausalLM(nn.Module):
279
bias=False)
280
self.logits_processor = LogitsProcessor(config.vocab_size)
281
self.sampler = Sampler()
282
+ self.make_empty_intermediate_tensors = (
283
+ self.model.make_empty_intermediate_tensors)
284
285
def forward(
286
self,
@@ -281,6 +296,7 @@ class PersimmonForCausalLM(nn.Module):
296
positions=positions,
297
kv_caches=kv_caches,
298
attn_metadata=attn_metadata,
299
+ intermediate_tensors=intermediate_tensors,
300
inputs_embeds=inputs_embeds,
301
)
302
return hidden_states
@@ -312,6 +328,8 @@ class PersimmonForCausalLM(nn.Module):
328
# Models trained using ColossalAI may include these tensors in
329
# the checkpoint. Skip them.
330
continue
331
+ if is_pp_missing_parameter(name, self):
332
+ continue
333
param = params_dict[name]
334
335
if "query_key_value" in name:
vllm/model_executor/models/phi.py
+36
-17
@@ -35,7 +35,7 @@
35
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
36
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
37
"""Inference-only Phi-1.5 model compatible with HuggingFace weights."""
38
-from typing import Iterable, List, Optional, Tuple
38
+from typing import Iterable, List, Optional, Tuple, Union
39
40
import torch
41
from torch import nn
@@ -43,14 +43,13 @@ from transformers import PhiConfig
43
44
from vllm.attention import Attention, AttentionMetadata
45
from vllm.config import CacheConfig, LoRAConfig
46
-from vllm.distributed import get_tensor_model_parallel_world_size
46
+from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
47
from vllm.model_executor.layers.activation import get_act_fn
48
from vllm.model_executor.layers.linear import (ColumnParallelLinear,
49
QKVParallelLinear,
50
RowParallelLinear)
51
from vllm.model_executor.layers.logits_processor import LogitsProcessor
52
-from vllm.model_executor.layers.quantization.base_config import (
53
- QuantizationConfig)
52
+from vllm.model_executor.layers.quantization import QuantizationConfig
53
from vllm.model_executor.layers.rotary_embedding import get_rope
54
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
55
from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -59,7 +58,9 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
58
from vllm.model_executor.sampling_metadata import SamplingMetadata
59
from vllm.sequence import IntermediateTensors
60
62
-from .interfaces import SupportsLoRA
61
+from .interfaces import SupportsLoRA, SupportsPP
62
+from .utils import (is_pp_missing_parameter,
63
+ make_empty_intermediate_tensors_factory, make_layers)
64
65
66
class PhiAttention(nn.Module):
@@ -196,18 +197,22 @@ class PhiModel(nn.Module):
197
def __init__(self,
198
config: PhiConfig,
199
cache_config: Optional[CacheConfig] = None,
199
- quant_config: Optional[QuantizationConfig] = None):
200
+ quant_config: Optional[QuantizationConfig] = None,
201
+ prefix: str = ""):
202
super().__init__()
203
self.config = config
204
self.quant_config = quant_config
205
self.embed_tokens = VocabParallelEmbedding(config.vocab_size,
206
config.hidden_size)
205
- self.layers = nn.ModuleList([
206
- PhiLayer(config, cache_config, quant_config)
207
- for _ in range(config.num_hidden_layers)
208
- ])
207
+ self.start_layer, self.end_layer, self.layers = make_layers(
208
+ config.num_hidden_layers,
209
+ lambda prefix: PhiLayer(config, cache_config, quant_config),
210
+ prefix=f"{prefix}.layers")
211
self.final_layernorm = nn.LayerNorm(config.hidden_size,
212
eps=config.layer_norm_eps)
213
+ self.make_empty_intermediate_tensors = (
214
+ make_empty_intermediate_tensors_factory(["hidden_states"],
215
+ config.hidden_size))
216
217
def forward(
218
self,
@@ -215,23 +220,31 @@ class PhiModel(nn.Module):
220
positions: torch.Tensor,
221
kv_caches: List[torch.Tensor],
222
attn_metadata: AttentionMetadata,
218
- ) -> torch.Tensor:
219
- hidden_states = self.embed_tokens(input_ids)
220
- for i in range(self.config.num_hidden_layers):
223
+ intermediate_tensors: Optional[IntermediateTensors],
224
+ ) -> Union[torch.Tensor, IntermediateTensors]:
225
+ if get_pp_group().is_first_rank:
226
+ hidden_states = self.embed_tokens(input_ids)
227
+ else:
228
+ assert intermediate_tensors is not None
229
+ hidden_states = intermediate_tensors["hidden_states"]
230
+ for i in range(self.start_layer, self.end_layer):
231
layer = self.layers[i]
232
hidden_states = layer(
233
positions,
234
hidden_states,
225
- kv_caches[i],
235
+ kv_caches[i - self.start_layer],
236
attn_metadata,
237
)
238
239
+ if not get_pp_group().is_last_rank:
240
+ return IntermediateTensors({"hidden_states": hidden_states})
241
+
242
hidden_states = self.final_layernorm(hidden_states)
243
244
return hidden_states
245
246
234
-class PhiForCausalLM(nn.Module, SupportsLoRA):
247
+class PhiForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
248
packed_modules_mapping = {
249
"qkv_proj": [
250
"q_proj",
@@ -274,6 +287,8 @@ class PhiForCausalLM(nn.Module, SupportsLoRA):
287
quant_config=quant_config)
288
self.logits_processor = LogitsProcessor(config.vocab_size)
289
self.sampler = Sampler()
290
+ self.make_empty_intermediate_tensors = (
291
+ self.model.make_empty_intermediate_tensors)
292
293
def forward(
294
self,
@@ -282,9 +297,9 @@ class PhiForCausalLM(nn.Module, SupportsLoRA):
297
kv_caches: List[torch.Tensor],
298
attn_metadata: AttentionMetadata,
299
intermediate_tensors: Optional[IntermediateTensors] = None,
285
- ) -> torch.Tensor:
300
+ ) -> Union[torch.Tensor, IntermediateTensors]:
301
hidden_states = self.model(input_ids, positions, kv_caches,
287
- attn_metadata)
302
+ attn_metadata, intermediate_tensors)
303
304
return hidden_states
305
@@ -325,6 +340,8 @@ class PhiForCausalLM(nn.Module, SupportsLoRA):
340
# Skip loading extra bias for GPTQ models.
341
if name.endswith(".bias") and name not in params_dict:
342
continue
343
+ if is_pp_missing_parameter(name, self):
344
+ continue
345
param = params_dict[name]
346
weight_loader = param.weight_loader
347
weight_loader(param, loaded_weight, shard_id)
@@ -335,6 +352,8 @@ class PhiForCausalLM(nn.Module, SupportsLoRA):
352
continue
353
# pylint: disable=E1136
354
355
+ if is_pp_missing_parameter(name, self):
356
+ continue
357
param = params_dict[name]
358
weight_loader = getattr(param, "weight_loader",
359
default_weight_loader)
vllm/model_executor/models/phi3_small.py
+40
-20
@@ -1,5 +1,5 @@
1
import math
2
-from typing import Iterable, List, Optional, Tuple
2
+from typing import Iterable, List, Optional, Tuple, Union
3
4
import torch
5
from torch import nn
@@ -7,14 +7,13 @@ from transformers.configuration_utils import PretrainedConfig
7
8
from vllm.attention import Attention, AttentionMetadata
9
from vllm.config import CacheConfig, LoRAConfig
10
-from vllm.distributed import (get_tensor_model_parallel_rank,
10
+from vllm.distributed import (get_pp_group, get_tensor_model_parallel_rank,
11
get_tensor_model_parallel_world_size)
12
from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
13
QKVParallelLinear,
14
RowParallelLinear)
15
from vllm.model_executor.layers.logits_processor import LogitsProcessor
16
-from vllm.model_executor.layers.quantization.base_config import (
17
- QuantizationConfig)
16
+from vllm.model_executor.layers.quantization import QuantizationConfig
17
from vllm.model_executor.layers.rotary_embedding import get_rope
18
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
19
from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -23,6 +22,10 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
22
from vllm.model_executor.sampling_metadata import SamplingMetadata
23
from vllm.sequence import IntermediateTensors
24
25
+from .interfaces import SupportsPP
26
+from .utils import (is_pp_missing_parameter,
27
+ make_empty_intermediate_tensors_factory, make_layers)
28
+
29
30
def load_column_parallel_weight(param: torch.nn.Parameter,
31
loaded_weight: torch.Tensor):
@@ -301,20 +304,25 @@ class Phi3SmallModel(nn.Module):
304
config: PretrainedConfig,
305
cache_config: Optional[CacheConfig] = None,
306
quant_config: Optional[QuantizationConfig] = None,
307
+ prefix: str = "",
308
):
309
super().__init__()
310
self.config = config
311
self.embed_tokens = VocabParallelEmbedding(config.vocab_size,
312
config.hidden_size)
313
self.mup_embedding_multiplier = config.mup_embedding_multiplier
310
- self.layers = nn.ModuleList([
311
- Phi3SmallDecoderLayer(config, layer_idx, cache_config,
312
- quant_config)
313
- for layer_idx in range(config.num_hidden_layers)
314
- ])
314
+ self.start_layer, self.end_layer, self.layers = make_layers(
315
+ config.num_hidden_layers,
316
+ lambda prefix: Phi3SmallDecoderLayer(config,
317
+ int(prefix.split('.')[-1]),
318
+ cache_config, quant_config),
319
+ prefix=f"{prefix}.layers")
320
321
self.final_layernorm = nn.LayerNorm(config.hidden_size,
322
eps=config.layer_norm_epsilon)
323
+ self.make_empty_intermediate_tensors = (
324
+ make_empty_intermediate_tensors_factory(["hidden_states"],
325
+ config.hidden_size))
326
327
def get_input_embeddings(self):
328
return self.embed_tokens
@@ -327,30 +335,37 @@ class Phi3SmallModel(nn.Module):
335
input_ids: torch.LongTensor,
336
positions: Optional[torch.LongTensor],
337
kv_caches: List[torch.Tensor],
330
- attn_metadata: AttentionMetadata = None,
331
- ):
332
- hidden_states = self.embed_tokens(input_ids)
333
- if (self.mup_embedding_multiplier is not None
334
- and self.mup_embedding_multiplier > 0.0):
335
- hidden_states = hidden_states * self.mup_embedding_multiplier
336
- for i in range(len(self.layers)):
338
+ attn_metadata: AttentionMetadata,
339
+ intermediate_tensors: Optional[IntermediateTensors],
340
+ ) -> Union[torch.Tensor, IntermediateTensors]:
341
+ if get_pp_group().is_first_rank:
342
+ hidden_states = self.embed_tokens(input_ids)
343
+ if (self.mup_embedding_multiplier is not None
344
+ and self.mup_embedding_multiplier > 0.0):
345
+ hidden_states = hidden_states * self.mup_embedding_multiplier
346
+ else:
347
+ assert intermediate_tensors
348
+ hidden_states = intermediate_tensors["hidden_states"]
349
+ for i in range(self.start_layer, self.end_layer):
350
layer = self.layers[i]
351
hidden_states = layer(
352
positions,
353
hidden_states,
341
- kv_caches[i],
354
+ kv_caches[i - self.start_layer],
355
attn_metadata,
356
)
357
+ if not get_pp_group().is_last_rank:
358
+ return IntermediateTensors({"hidden_states": hidden_states})
359
hidden_states = self.final_layernorm(hidden_states)
360
return hidden_states
361
362
348
-class Phi3SmallForCausalLM(nn.Module):
363
+class Phi3SmallForCausalLM(nn.Module, SupportsPP):
364
_tied_weights_keys = ["lm_head.weight"]
365
366
def __init__(
367
self,
353
- config,
368
+ config: PretrainedConfig,
369
cache_config: Optional[CacheConfig] = None,
370
quant_config: Optional[QuantizationConfig] = None,
371
lora_config: Optional[LoRAConfig] = None,
@@ -372,6 +387,8 @@ class Phi3SmallForCausalLM(nn.Module):
387
self.lm_head.weight = self.model.embed_tokens.weight
388
self.logits_processor = LogitsProcessor(config.vocab_size)
389
self.sampler = Sampler()
390
+ self.make_empty_intermediate_tensors = (
391
+ self.model.make_empty_intermediate_tensors)
392
393
# tokens in tiktoken but not used
394
if hasattr(config, 'dummy_token_indices'):
@@ -419,12 +436,13 @@ class Phi3SmallForCausalLM(nn.Module):
436
kv_caches: List[torch.Tensor],
437
attn_metadata: AttentionMetadata,
438
intermediate_tensors: Optional[IntermediateTensors] = None,
422
- ) -> torch.Tensor:
439
+ ) -> Union[torch.Tensor, IntermediateTensors]:
440
output_hidden_states = self.model(
441
input_ids=input_ids,
442
positions=positions,
443
kv_caches=kv_caches,
444
attn_metadata=attn_metadata,
445
+ intermediate_tensors=intermediate_tensors,
446
)
447
output_hidden_states = output_hidden_states
448
return output_hidden_states
@@ -447,6 +465,8 @@ class Phi3SmallForCausalLM(nn.Module):
465
continue
466
if name.endswith(".bias") and name not in params_dict:
467
continue
468
+ if is_pp_missing_parameter(name, self):
469
+ continue
470
param = params_dict[name]
471
weight_loader = getattr(param, "weight_loader",
472
default_weight_loader)
vllm/model_executor/models/phi3v.py
+96
-85
@@ -15,7 +15,7 @@
15
# limitations under the License.
16
import itertools
17
import re
18
-from functools import lru_cache
18
+from functools import cached_property, lru_cache
19
from typing import (Any, Dict, Iterable, List, Literal, Mapping, Optional,
20
Tuple, TypedDict, Union)
21
@@ -29,13 +29,11 @@ from vllm.attention import AttentionMetadata
29
from vllm.config import CacheConfig, ModelConfig, MultiModalConfig
30
from vllm.inputs import INPUT_REGISTRY, InputContext, LLMInputs
31
from vllm.logger import init_logger
32
-from vllm.model_executor.layers.logits_processor import LogitsProcessor
32
from vllm.model_executor.layers.quantization import QuantizationConfig
33
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
35
-from vllm.model_executor.layers.vocab_parallel_embedding import ParallelLMHead
34
from vllm.model_executor.model_loader.weight_utils import default_weight_loader
35
from vllm.model_executor.models.clip import CLIPVisionModel
38
-from vllm.model_executor.models.llama import LlamaModel
36
+from vllm.model_executor.models.llama import LlamaForCausalLM
37
from vllm.model_executor.sampling_metadata import SamplingMetadata
38
from vllm.multimodal import MULTIMODAL_REGISTRY
39
from vllm.multimodal.utils import cached_get_tokenizer, repeat_and_pad_token
@@ -43,8 +41,9 @@ from vllm.sequence import IntermediateTensors
41
from vllm.utils import is_list_of
42
43
from .clip import dummy_image_for_clip, dummy_seq_data_for_clip
46
-from .interfaces import SupportsMultiModal
47
-from .utils import flatten_bn, merge_multimodal_embeddings
44
+from .interfaces import SupportsMultiModal, SupportsPP
45
+from .utils import (flatten_bn, group_weights_with_prefix,
46
+ merge_multimodal_embeddings)
47
48
logger = init_logger(__name__)
49
@@ -295,6 +294,37 @@ class Phi3HDImageEmbedding(Phi3ImageEmbeddingBase):
294
dim=2).reshape(num_images, -1, hid_dim)
295
return image_features_hd_newline
296
297
+ def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
298
+ # prepare weight iterators for components
299
+ weights_group = group_weights_with_prefix(weights)
300
+
301
+ # load vision encoder
302
+ self.img_processor.load_weights(weights_group["img_processor"])
303
+
304
+ # load glb_GN
305
+ for name, loaded_weight in weights_group["glb_GN"]:
306
+ assert name == ""
307
+ param = self.glb_GN
308
+ weight_loader = getattr(param, "weight_loader",
309
+ default_weight_loader)
310
+ weight_loader(param, loaded_weight)
311
+
312
+ # load sub_GN
313
+ for name, loaded_weight in weights_group["sub_GN"]:
314
+ assert name == ""
315
+ param = self.sub_GN
316
+ weight_loader = getattr(param, "weight_loader",
317
+ default_weight_loader)
318
+ weight_loader(param, loaded_weight)
319
+
320
+ # load mlp projector
321
+ mlp_params_dict = dict(self.img_projection.named_parameters())
322
+ for name, loaded_weight in weights_group["img_projection"]:
323
+ param = mlp_params_dict[name]
324
+ weight_loader = getattr(param, "weight_loader",
325
+ default_weight_loader)
326
+ weight_loader(param, loaded_weight)
327
+
328
329
# Based on https://huggingface.co/microsoft/Phi-3-vision-128k-instruct/blob/main/image_processing_phi3_v.py#L57
330
def _calc_padded_size(*, width: int, height: int, padding_unit: int = 336):
@@ -508,7 +538,7 @@ def input_processor_for_phi3v(ctx: InputContext,
538
@MULTIMODAL_REGISTRY.register_max_image_tokens(get_max_phi3v_image_tokens)
539
@INPUT_REGISTRY.register_dummy_data(dummy_data_for_phi3v)
540
@INPUT_REGISTRY.register_input_processor(input_processor_for_phi3v)
511
-class Phi3VForCausalLM(nn.Module, SupportsMultiModal):
541
+class Phi3VForCausalLM(nn.Module, SupportsMultiModal, SupportsPP):
542
543
def __init__(self,
544
config: PretrainedConfig,
@@ -521,17 +551,21 @@ class Phi3VForCausalLM(nn.Module, SupportsMultiModal):
551
self.multimodal_config = multimodal_config
552
self.image_token_id = _IMAGE_TOKEN_ID
553
524
- self.model = LlamaModel(config, cache_config, quant_config)
525
-
554
# TODO: Optionally initializes this for supporting embeddings.
555
self.vision_embed_tokens = Phi3HDImageEmbedding(config)
528
- self.lm_head = ParallelLMHead(config.vocab_size,
529
- config.hidden_size,
530
- quant_config=quant_config)
531
- if self.config.tie_word_embeddings:
532
- self.lm_head.weight = self.model.embed_tokens.weight
533
- self.logits_processor = LogitsProcessor(config.vocab_size)
534
- self.sampler = Sampler()
556
+
557
+ self.language_model = LlamaForCausalLM(config, cache_config,
558
+ quant_config)
559
+
560
+ self.make_empty_intermediate_tensors = (
561
+ self.language_model.make_empty_intermediate_tensors)
562
+
563
+ @cached_property
564
+ def sampler(self):
565
+ if hasattr(self.language_model, "sampler"):
566
+ return self.language_model.sampler
567
+
568
+ return Sampler()
569
570
def _validate_image_sizes(self, data: torch.Tensor) -> torch.Tensor:
571
expected_dims = (2, )
@@ -631,24 +665,29 @@ class Phi3VForCausalLM(nn.Module, SupportsMultiModal):
665
attn_metadata: AttentionMetadata,
666
intermediate_tensors: Optional[IntermediateTensors] = None,
667
**kwargs: object):
634
- image_input = self._parse_and_validate_image_input(**kwargs)
635
-
636
- if image_input is not None:
637
- vision_embeddings = self._process_image_input(image_input)
638
- inputs_embeds = self.model.get_input_embeddings(input_ids)
639
- inputs_embeds = merge_multimodal_embeddings(
640
- input_ids, inputs_embeds, vision_embeddings,
641
- self.image_token_id)
668
+ if intermediate_tensors is not None:
669
input_ids = None
643
- else:
670
inputs_embeds = None
671
+ else:
672
+ image_input = self._parse_and_validate_image_input(**kwargs)
673
+
674
+ if image_input is not None:
675
+ vision_embeddings = self._process_image_input(image_input)
676
+ inputs_embeds = self.language_model.model.get_input_embeddings(
677
+ input_ids)
678
+ inputs_embeds = merge_multimodal_embeddings(
679
+ input_ids, inputs_embeds, vision_embeddings,
680
+ self.image_token_id)
681
+ input_ids = None
682
+ else:
683
+ inputs_embeds = None
684
646
- hidden_states = self.model(input_ids,
647
- positions,
648
- kv_caches,
649
- attn_metadata,
650
- intermediate_tensors,
651
- inputs_embeds=inputs_embeds)
685
+ hidden_states = self.language_model.model(input_ids,
686
+ positions,
687
+ kv_caches,
688
+ attn_metadata,
689
+ intermediate_tensors,
690
+ inputs_embeds=inputs_embeds)
691
692
return hidden_states
693
@@ -657,66 +696,38 @@ class Phi3VForCausalLM(nn.Module, SupportsMultiModal):
696
hidden_states: torch.Tensor,
697
sampling_metadata: SamplingMetadata,
698
) -> Optional[torch.Tensor]:
660
- logits = self.logits_processor(self.lm_head, hidden_states,
661
- sampling_metadata)
662
- return logits
699
+ return self.language_model.compute_logits(hidden_states,
700
+ sampling_metadata)
701
702
def sample(
703
self,
704
logits: torch.Tensor,
705
sampling_metadata: SamplingMetadata,
706
) -> Optional[SamplerOutput]:
669
- next_tokens = self.sampler(logits, sampling_metadata)
670
- return next_tokens
707
+ return self.language_model.sample(logits, sampling_metadata)
708
709
def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
673
- stacked_params_mapping = [
674
- # (param_name, shard_name, shard_id)
675
- (".qkv_proj", ".q_proj", "q"),
676
- (".qkv_proj", ".k_proj", "k"),
677
- (".qkv_proj", ".v_proj", "v"),
678
- (".gate_up_proj", ".gate_proj", 0),
679
- (".gate_up_proj", ".up_proj", 1),
680
- ]
710
+ hf_to_vllm_mapping = {
711
+ "model.vision_embed_tokens.": "vision_embed_tokens.",
712
+ "lm_head.": "language_model.lm_head.",
713
+ "model.": "language_model.model.",
714
+ }
715
682
- # TODO(ChristopherCho): This is a temporary fix to load
683
- # the vision weights with CLIPVisionModel.load_weights()
684
- vision_weights = []
685
- params_dict = dict(self.named_parameters())
686
- for name, loaded_weight in weights:
687
- if "rotary_emb.inv_freq" in name:
688
- continue
689
- # Skip loading the img_processor weights since they are
690
- # loaded separately.
691
- if "vision_embed_tokens.img_processor" in name:
692
- vision_weights.append((name, loaded_weight))
693
- continue
694
-
695
- for key_to_modify, new_key in _KEYS_TO_MODIFY_MAPPING.items():
696
- if key_to_modify in name:
697
- name = name.replace(key_to_modify, new_key)
698
- for (param_name, weight_name, shard_id) in stacked_params_mapping:
699
- if weight_name not in name:
700
- continue
701
-
702
- param = params_dict[name.replace(weight_name, param_name)]
703
- weight_loader = param.weight_loader
704
- weight_loader(param, loaded_weight, shard_id)
705
- break
706
- else:
707
- # Skip loading extra bias for GPTQ models.
708
- if name.endswith(".bias") and name not in params_dict:
709
- continue
710
- if name in params_dict:
711
- param = params_dict[name]
712
- weight_loader = getattr(param, "weight_loader",
713
- default_weight_loader)
714
- weight_loader(param, loaded_weight)
715
-
716
- # We use regex to extract the sub-module name
717
- # from "model.vision_embed_tokens.img_processor.*"
718
- vision_weights = [
719
- (re.search(r"vision_embed_tokens\.img_processor\.(.*)",
720
- n).group(1), w) for n, w in vision_weights
721
- ]
722
- self.vision_embed_tokens.img_processor.load_weights(vision_weights)
716
+ def hf_to_vllm_name(key: str) -> str:
717
+ for hf_name, vllm_name in hf_to_vllm_mapping.items():
718
+ if key.startswith(hf_name):
719
+ return key.replace(hf_name, vllm_name, 1)
720
+
721
+ return key
722
+
723
+ vllm_weights = {hf_to_vllm_name(k): v for k, v in weights}
724
+
725
+ # prepare weight iterators for components
726
+ weights_group = group_weights_with_prefix(vllm_weights.items())
727
+
728
+ # load vision embeddings and encoder
729
+ self.vision_embed_tokens.load_weights(
730
+ weights_group["vision_embed_tokens"])
731
+
732
+ # load llm backbone
733
+ self.language_model.load_weights(weights_group["language_model"])
vllm/model_executor/models/phimoe.py
+55
-17
@@ -21,7 +21,7 @@
21
# See the License for the specific language governing permissions and
22
# limitations under the License.
23
"""Inference-only PhiMoE model."""
24
-from typing import Iterable, List, Optional, Tuple
24
+from typing import Iterable, List, Optional, Tuple, Union
25
26
import torch
27
from torch import nn
@@ -29,7 +29,7 @@ from transformers.configuration_utils import PretrainedConfig
29
30
from vllm.attention import Attention, AttentionMetadata
31
from vllm.config import CacheConfig, LoRAConfig
32
-from vllm.distributed import get_tensor_model_parallel_world_size
32
+from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
33
from vllm.model_executor.layers.fused_moe import FusedMoE
34
from vllm.model_executor.layers.linear import (QKVParallelLinear,
35
ReplicatedLinear,
@@ -46,7 +46,9 @@ from vllm.model_executor.model_loader.weight_utils import (
46
from vllm.model_executor.sampling_metadata import SamplingMetadata
47
from vllm.sequence import IntermediateTensors
48
49
-from .interfaces import SupportsLoRA
49
+from .interfaces import SupportsLoRA, SupportsPP
50
+from .utils import (is_pp_missing_parameter,
51
+ make_empty_intermediate_tensors_factory, make_layers)
52
53
54
class PhiMoEConfig(PretrainedConfig):
@@ -435,6 +437,7 @@ class PhiMoEModel(nn.Module):
437
cache_config: Optional[CacheConfig] = None,
438
quant_config: Optional[QuantizationConfig] = None,
439
lora_config: Optional[LoRAConfig] = None,
440
+ prefix: str = "",
441
) -> None:
442
super().__init__()
443
self.padding_idx = config.pad_token_id
@@ -448,33 +451,56 @@ class PhiMoEModel(nn.Module):
451
config.hidden_size,
452
org_num_embeddings=config.vocab_size,
453
)
451
- self.layers = nn.ModuleList([
452
- PhiMoEDecoderLayer(config, cache_config, quant_config=quant_config)
453
- for _ in range(config.num_hidden_layers)
454
- ])
454
+ self.start_layer, self.end_layer, self.layers = make_layers(
455
+ config.num_hidden_layers,
456
+ lambda prefix: PhiMoEDecoderLayer(config, cache_config,
457
+ quant_config),
458
+ prefix=f"{prefix}.layers")
459
self.norm = nn.LayerNorm(config.hidden_size,
460
eps=config.rms_norm_eps,
461
elementwise_affine=True)
462
463
+ self.make_empty_intermediate_tensors = (
464
+ make_empty_intermediate_tensors_factory(
465
+ ["hidden_states", "residual"], config.hidden_size))
466
+
467
def forward(
468
self,
469
input_ids: torch.Tensor,
470
positions: torch.Tensor,
471
kv_caches: List[torch.Tensor],
472
attn_metadata: AttentionMetadata,
465
- ) -> torch.Tensor:
466
- hidden_states = self.embed_tokens(input_ids)
467
- residual = None
468
- for i in range(len(self.layers)):
473
+ intermediate_tensors: Optional[IntermediateTensors],
474
+ ) -> Union[torch.Tensor, IntermediateTensors]:
475
+ if get_pp_group().is_first_rank:
476
+ hidden_states = self.embed_tokens(input_ids)
477
+ residual = None
478
+ else:
479
+ assert intermediate_tensors is not None
480
+ hidden_states = intermediate_tensors["hidden_states"]
481
+ residual = intermediate_tensors["residual"]
482
+
483
+ for i in range(self.start_layer, self.end_layer):
484
layer = self.layers[i]
470
- hidden_states, residual = layer(positions, hidden_states,
471
- kv_caches[i], attn_metadata,
472
- residual)
485
+ hidden_states, residual = layer(
486
+ positions,
487
+ hidden_states,
488
+ kv_caches[i - self.start_layer],
489
+ attn_metadata,
490
+ residual,
491
+ )
492
+
493
+ if not get_pp_group().is_last_rank:
494
+ return IntermediateTensors({
495
+ "hidden_states": hidden_states,
496
+ "residual": residual
497
+ })
498
+
499
hidden_states = self.norm(hidden_states)
500
return hidden_states
501
502
477
-class PhiMoEForCausalLM(nn.Module, SupportsLoRA):
503
+class PhiMoEForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
504
fall_back_to_pt_during_load = False
505
506
packed_modules_mapping = {
@@ -537,6 +563,9 @@ class PhiMoEForCausalLM(nn.Module, SupportsLoRA):
563
config.vocab_size)
564
self.sampler = Sampler()
565
566
+ self.make_empty_intermediate_tensors = (
567
+ self.model.make_empty_intermediate_tensors)
568
+
569
def forward(
570
self,
571
input_ids: torch.Tensor,
@@ -544,9 +573,9 @@ class PhiMoEForCausalLM(nn.Module, SupportsLoRA):
573
kv_caches: List[torch.Tensor],
574
attn_metadata: AttentionMetadata,
575
intermediate_tensors: Optional[IntermediateTensors] = None,
547
- ) -> torch.Tensor:
576
+ ) -> Union[torch.Tensor, IntermediateTensors]:
577
hidden_states = self.model(input_ids, positions, kv_caches,
549
- attn_metadata)
578
+ attn_metadata, intermediate_tensors)
579
return hidden_states
580
581
def compute_logits(self, hidden_states: torch.Tensor,
@@ -589,6 +618,9 @@ class PhiMoEForCausalLM(nn.Module, SupportsLoRA):
618
# Skip loading extra bias for GPTQ models.
619
if name.endswith(".bias") and name not in params_dict:
620
continue
621
+ # Skip layers on other devices.
622
+ if is_pp_missing_parameter(name, self):
623
+ continue
624
param = params_dict[name]
625
weight_loader = param.weight_loader
626
weight_loader(param, loaded_weight, shard_id)
@@ -599,6 +631,9 @@ class PhiMoEForCausalLM(nn.Module, SupportsLoRA):
631
if weight_name not in name:
632
continue
633
name = name.replace(weight_name, param_name)
634
+ # Skip layers on other devices.
635
+ if is_pp_missing_parameter(name, self):
636
+ continue
637
param = params_dict[name]
638
weight_loader = param.weight_loader
639
weight_loader(
@@ -613,6 +648,9 @@ class PhiMoEForCausalLM(nn.Module, SupportsLoRA):
648
# Skip loading extra bias for GPTQ models.
649
if name.endswith(".bias") and name not in params_dict:
650
continue
651
+ # Skip layers on other devices.
652
+ if is_pp_missing_parameter(name, self):
653
+ continue
654
# Remapping the name of FP8 kv-scale.
655
name = maybe_remap_kv_scale_name(name, params_dict)
656
if name is None:
vllm/model_executor/models/pixtral.py
+32
-16
@@ -1,4 +1,5 @@
1
from dataclasses import dataclass, fields
2
+from functools import cached_property
3
from itertools import tee
4
from typing import Iterable, List, Mapping, Optional, Tuple, Union
5
@@ -16,7 +17,7 @@ from vllm.config import CacheConfig, MultiModalConfig
17
from vllm.inputs import INPUT_REGISTRY, InputContext, LLMInputs
18
from vllm.model_executor.layers.layernorm import RMSNorm
19
from vllm.model_executor.layers.quantization import QuantizationConfig
19
-from vllm.model_executor.layers.sampler import SamplerOutput
20
+from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
21
from vllm.model_executor.model_loader.weight_utils import default_weight_loader
22
from vllm.model_executor.models.utils import merge_multimodal_embeddings
23
from vllm.model_executor.sampling_metadata import SamplingMetadata
@@ -25,7 +26,7 @@ from vllm.multimodal.base import MultiModalInputs
26
from vllm.multimodal.utils import cached_get_tokenizer
27
from vllm.sequence import IntermediateTensors, SequenceData
28
28
-from .interfaces import SupportsMultiModal
29
+from .interfaces import SupportsMultiModal, SupportsPP
30
from .utils import init_vllm_registered_model
31
32
@@ -126,7 +127,8 @@ def input_processor_for_pixtral(ctx: InputContext, llm_inputs: LLMInputs):
127
@MULTIMODAL_REGISTRY.register_max_image_tokens(get_max_pixtral_image_tokens)
128
@INPUT_REGISTRY.register_dummy_data(dummy_data_for_pixtral)
129
@INPUT_REGISTRY.register_input_processor(input_processor_for_pixtral)
129
-class PixtralForConditionalGeneration(nn.Module, SupportsMultiModal):
130
+class PixtralForConditionalGeneration(nn.Module, SupportsMultiModal,
131
+ SupportsPP):
132
133
def __init__(self,
134
config: PretrainedConfig,
@@ -155,6 +157,16 @@ class PixtralForConditionalGeneration(nn.Module, SupportsMultiModal):
157
self.vision_language_adapter = VisionLanguageAdapter(
158
self.vision_args, dim=config.text_config.hidden_size)
159
160
+ self.make_empty_intermediate_tensors = (
161
+ self.language_model.make_empty_intermediate_tensors)
162
+
163
+ @cached_property
164
+ def sampler(self):
165
+ if hasattr(self.language_model, "sampler"):
166
+ return self.language_model.sampler
167
+
168
+ return Sampler()
169
+
170
def forward(
171
self,
172
input_ids: torch.Tensor,
@@ -163,32 +175,36 @@ class PixtralForConditionalGeneration(nn.Module, SupportsMultiModal):
175
attn_metadata: AttentionMetadata,
176
intermediate_tensors: Optional[IntermediateTensors] = None,
177
**kwargs: object,
166
- ) -> SamplerOutput:
178
+ ) -> Union[torch.Tensor, IntermediateTensors]:
179
"""Run forward pass for pixtral.
180
181
TODO
182
183
"""
172
- image_input = self._parse_and_validate_image_input(**kwargs)
184
+ if intermediate_tensors is not None:
185
+ input_ids = None
186
+ inputs_embeds = None
187
+ else:
188
+ image_input = self._parse_and_validate_image_input(**kwargs)
189
174
- if image_input is not None:
175
- vision_embeddings = self._process_image_input(image_input)
176
- inputs_embeds = self.language_model.model.get_input_embeddings(
177
- input_ids)
190
+ if image_input is not None:
191
+ vision_embeddings = self._process_image_input(image_input)
192
+ inputs_embeds = self.language_model.model.get_input_embeddings(
193
+ input_ids)
194
179
- inputs_embeds = merge_multimodal_embeddings(
180
- input_ids, inputs_embeds, vision_embeddings,
181
- self.vision_args.image_token_id)
195
+ inputs_embeds = merge_multimodal_embeddings(
196
+ input_ids, inputs_embeds, vision_embeddings,
197
+ self.vision_args.image_token_id)
198
183
- input_ids = None
184
- else:
185
- inputs_embeds = None
199
+ input_ids = None
200
+ else:
201
+ inputs_embeds = None
202
203
hidden_states = self.language_model.model(input_ids,
204
positions,
205
kv_caches,
206
attn_metadata,
191
- None,
207
+ intermediate_tensors,
208
inputs_embeds=inputs_embeds)
209
210
return hidden_states
vllm/model_executor/models/qwen.py
+26
-28
@@ -31,15 +31,13 @@ from vllm.model_executor.layers.linear import (ColumnParallelLinear,
31
QKVParallelLinear,
32
RowParallelLinear)
33
from vllm.model_executor.layers.logits_processor import LogitsProcessor
34
-from vllm.model_executor.layers.quantization.base_config import (
35
- QuantizationConfig)
34
+from vllm.model_executor.layers.quantization import QuantizationConfig
35
from vllm.model_executor.layers.resampler import Resampler2, get_abs_pos
36
from vllm.model_executor.layers.rotary_embedding import get_rope
37
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
38
from vllm.model_executor.layers.vocab_parallel_embedding import (
39
ParallelLMHead, VocabParallelEmbedding)
40
from vllm.model_executor.model_loader.weight_utils import default_weight_loader
42
-from vllm.model_executor.models.interfaces import SupportsMultiModal
41
from vllm.model_executor.sampling_metadata import SamplingMetadata
42
from vllm.multimodal import MULTIMODAL_REGISTRY
43
from vllm.multimodal.base import MultiModalInputs
@@ -47,7 +45,9 @@ from vllm.multimodal.utils import cached_get_tokenizer
45
from vllm.sequence import IntermediateTensors, SequenceData
46
from vllm.utils import is_list_of
47
50
-from .utils import flatten_bn, is_pp_missing_parameter, make_layers
48
+from .interfaces import SupportsMultiModal, SupportsPP
49
+from .utils import (flatten_bn, is_pp_missing_parameter,
50
+ make_empty_intermediate_tensors_factory, make_layers)
51
52
logger = init_logger(__name__)
53
@@ -568,6 +568,9 @@ class QWenModel(nn.Module):
568
lambda prefix: QWenBlock(config, cache_config, quant_config),
569
prefix=f"{prefix}.h")
570
self.ln_f = RMSNorm(config.hidden_size, eps=config.layer_norm_epsilon)
571
+ self.make_empty_intermediate_tensors = (
572
+ make_empty_intermediate_tensors_factory(
573
+ ["hidden_states", "residual"], config.hidden_size))
574
self.visual = VisionTransformer(**config.visual,
575
quant_config=quant_config) if hasattr(
576
config, "visual") else None
@@ -580,7 +583,7 @@ class QWenModel(nn.Module):
583
attn_metadata: AttentionMetadata,
584
intermediate_tensors: Optional[IntermediateTensors],
585
pixel_values: Optional[QwenImageInputs],
583
- ) -> torch.Tensor:
586
+ ) -> Union[torch.Tensor, IntermediateTensors]:
587
img_pos = None
588
# If pixel / visual embeddings are provided, this is a visual model
589
if pixel_values is not None and self.visual is not None:
@@ -860,7 +863,7 @@ def dummy_data_for_qwen(
863
@MULTIMODAL_REGISTRY.register_max_image_tokens(MAX_QWEN_IMG_TOKENS)
864
@INPUT_REGISTRY.register_dummy_data(dummy_data_for_qwen)
865
@INPUT_REGISTRY.register_input_processor(input_processor_for_qwen)
863
-class QWenLMHeadModel(nn.Module, SupportsMultiModal):
866
+class QWenLMHeadModel(nn.Module, SupportsMultiModal, SupportsPP):
867
868
def __init__(
869
self,
@@ -881,6 +884,8 @@ class QWenLMHeadModel(nn.Module, SupportsMultiModal):
884
self.lm_head.weight = self.transformer.wte.weight
885
self.logits_processor = LogitsProcessor(config.vocab_size)
886
self.sampler = Sampler()
887
+ self.make_empty_intermediate_tensors = (
888
+ self.transformer.make_empty_intermediate_tensors)
889
890
def _get_image_input_type(
891
self,
@@ -912,33 +917,26 @@ class QWenLMHeadModel(nn.Module, SupportsMultiModal):
917
)
918
return None
919
915
- def forward(self,
916
- input_ids: torch.Tensor,
917
- positions: torch.Tensor,
918
- kv_caches: List[torch.Tensor],
919
- attn_metadata: AttentionMetadata,
920
- intermediate_tensors: Optional[IntermediateTensors] = None,
921
- pixel_values: Optional[torch.Tensor] = None) -> torch.Tensor:
922
- pixel_values = self._get_image_input_type(pixel_values)
920
+ def forward(
921
+ self,
922
+ input_ids: torch.Tensor,
923
+ positions: torch.Tensor,
924
+ kv_caches: List[torch.Tensor],
925
+ attn_metadata: AttentionMetadata,
926
+ intermediate_tensors: Optional[IntermediateTensors] = None,
927
+ pixel_values: Optional[torch.Tensor] = None
928
+ ) -> Union[torch.Tensor, IntermediateTensors]:
929
+ if intermediate_tensors is not None:
930
+ input_ids = None
931
+ pixel_values = None
932
+ else:
933
+ pixel_values = self._get_image_input_type(pixel_values)
934
+
935
hidden_states = self.transformer(input_ids, positions, kv_caches,
936
attn_metadata, intermediate_tensors,
937
pixel_values)
938
return hidden_states
939
928
- def make_empty_intermediate_tensors(
929
- self, batch_size: int, dtype: torch.dtype,
930
- device: torch.device) -> IntermediateTensors:
931
- return IntermediateTensors({
932
- "hidden_states":
933
- torch.zeros((batch_size, self.config.hidden_size),
934
- dtype=dtype,
935
- device=device),
936
- "residual":
937
- torch.zeros((batch_size, self.config.hidden_size),
938
- dtype=dtype,
939
- device=device),
940
- })
941
-
940
def compute_logits(
941
self,
942
hidden_states: torch.Tensor,
vllm/model_executor/models/qwen2.py
+13
-22
@@ -22,7 +22,7 @@
22
# See the License for the specific language governing permissions and
23
# limitations under the License.
24
"""Inference-only Qwen2 model compatible with HuggingFace weights."""
25
-from typing import Iterable, List, Optional, Tuple
25
+from typing import Iterable, List, Optional, Tuple, Union
26
27
import torch
28
from torch import nn
@@ -37,8 +37,7 @@ from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
37
QKVParallelLinear,
38
RowParallelLinear)
39
from vllm.model_executor.layers.logits_processor import LogitsProcessor
40
-from vllm.model_executor.layers.quantization.base_config import (
41
- QuantizationConfig)
40
+from vllm.model_executor.layers.quantization import QuantizationConfig
41
from vllm.model_executor.layers.rotary_embedding import get_rope
42
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
43
from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -48,8 +47,9 @@ from vllm.model_executor.model_loader.weight_utils import (
47
from vllm.model_executor.sampling_metadata import SamplingMetadata
48
from vllm.sequence import IntermediateTensors
49
51
-from .interfaces import SupportsLoRA
52
-from .utils import PPMissingLayer, is_pp_missing_parameter, make_layers
50
+from .interfaces import SupportsLoRA, SupportsPP
51
+from .utils import (PPMissingLayer, is_pp_missing_parameter,
52
+ make_empty_intermediate_tensors_factory, make_layers)
53
54
55
class Qwen2MLP(nn.Module):
@@ -253,6 +253,9 @@ class Qwen2Model(nn.Module):
253
prefix=f"{prefix}.layers",
254
)
255
256
+ self.make_empty_intermediate_tensors = (
257
+ make_empty_intermediate_tensors_factory(
258
+ ["hidden_states", "residual"], config.hidden_size))
259
if get_pp_group().is_last_rank:
260
self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
261
else:
@@ -269,7 +272,7 @@ class Qwen2Model(nn.Module):
272
attn_metadata: AttentionMetadata,
273
intermediate_tensors: Optional[IntermediateTensors] = None,
274
inputs_embeds: Optional[torch.Tensor] = None,
272
- ) -> torch.Tensor:
275
+ ) -> Union[torch.Tensor, IntermediateTensors]:
276
if get_pp_group().is_first_rank:
277
if inputs_embeds is not None:
278
hidden_states = inputs_embeds
@@ -298,7 +301,7 @@ class Qwen2Model(nn.Module):
301
return hidden_states
302
303
301
-class Qwen2ForCausalLM(nn.Module, SupportsLoRA):
304
+class Qwen2ForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
305
packed_modules_mapping = {
306
"qkv_proj": [
307
"q_proj",
@@ -357,6 +360,8 @@ class Qwen2ForCausalLM(nn.Module, SupportsLoRA):
360
361
self.logits_processor = LogitsProcessor(config.vocab_size)
362
self.sampler = Sampler()
363
+ self.make_empty_intermediate_tensors = (
364
+ self.model.make_empty_intermediate_tensors)
365
366
def forward(
367
self,
@@ -365,7 +370,7 @@ class Qwen2ForCausalLM(nn.Module, SupportsLoRA):
370
kv_caches: List[torch.Tensor],
371
attn_metadata: AttentionMetadata,
372
intermediate_tensors: Optional[IntermediateTensors] = None,
368
- ) -> torch.Tensor:
373
+ ) -> Union[torch.Tensor, IntermediateTensors]:
374
hidden_states = self.model(input_ids, positions, kv_caches,
375
attn_metadata, intermediate_tensors)
376
return hidden_states
@@ -379,20 +384,6 @@ class Qwen2ForCausalLM(nn.Module, SupportsLoRA):
384
sampling_metadata)
385
return logits
386
382
- def make_empty_intermediate_tensors(
383
- self, batch_size: int, dtype: torch.dtype,
384
- device: torch.device) -> IntermediateTensors:
385
- return IntermediateTensors({
386
- "hidden_states":
387
- torch.zeros((batch_size, self.config.hidden_size),
388
- dtype=dtype,
389
- device=device),
390
- "residual":
391
- torch.zeros((batch_size, self.config.hidden_size),
392
- dtype=dtype,
393
- device=device),
394
- })
395
-
387
def sample(
388
self,
389
logits: torch.Tensor,
vllm/model_executor/models/qwen2_moe.py
+13
-21
@@ -22,7 +22,7 @@
22
# See the License for the specific language governing permissions and
23
# limitations under the License.
24
"""Inference-only Qwen2MoE model compatible with HuggingFace weights."""
25
-from typing import Any, Dict, Iterable, List, Optional, Tuple
25
+from typing import Any, Dict, Iterable, List, Optional, Tuple, Union
26
27
import torch
28
import torch.nn.functional as F
@@ -42,8 +42,7 @@ from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
42
ReplicatedLinear,
43
RowParallelLinear)
44
from vllm.model_executor.layers.logits_processor import LogitsProcessor
45
-from vllm.model_executor.layers.quantization.base_config import (
46
- QuantizationConfig)
45
+from vllm.model_executor.layers.quantization import QuantizationConfig
46
from vllm.model_executor.layers.rotary_embedding import get_rope
47
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
48
from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -53,7 +52,9 @@ from vllm.model_executor.sampling_metadata import SamplingMetadata
52
from vllm.sequence import IntermediateTensors
53
from vllm.utils import print_warning_once
54
56
-from .utils import is_pp_missing_parameter, make_layers
55
+from .interfaces import SupportsPP
56
+from .utils import (is_pp_missing_parameter,
57
+ make_empty_intermediate_tensors_factory, make_layers)
58
59
60
class Qwen2MoeMLP(nn.Module):
@@ -338,6 +339,9 @@ class Qwen2MoeModel(nn.Module):
339
prefix=f"{prefix}.layers",
340
)
341
self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
342
+ self.make_empty_intermediate_tensors = (
343
+ make_empty_intermediate_tensors_factory(
344
+ ["hidden_states", "residual"], config.hidden_size))
345
346
def forward(
347
self,
@@ -346,7 +350,7 @@ class Qwen2MoeModel(nn.Module):
350
kv_caches: List[torch.Tensor],
351
attn_metadata: AttentionMetadata,
352
intermediate_tensors: Optional[IntermediateTensors] = None,
349
- ) -> torch.Tensor:
353
+ ) -> Union[torch.Tensor, IntermediateTensors]:
354
if get_pp_group().is_first_rank:
355
hidden_states = self.embed_tokens(input_ids)
356
residual = None
@@ -368,7 +372,7 @@ class Qwen2MoeModel(nn.Module):
372
return hidden_states
373
374
371
-class Qwen2MoeForCausalLM(nn.Module):
375
+class Qwen2MoeForCausalLM(nn.Module, SupportsPP):
376
377
fall_back_to_pt_during_load = False
378
@@ -389,6 +393,8 @@ class Qwen2MoeForCausalLM(nn.Module):
393
self.lm_head.weight = self.model.embed_tokens.weight
394
self.logits_processor = LogitsProcessor(config.vocab_size)
395
self.sampler = Sampler()
396
+ self.make_empty_intermediate_tensors = (
397
+ self.model.make_empty_intermediate_tensors)
398
399
def forward(
400
self,
@@ -397,7 +403,7 @@ class Qwen2MoeForCausalLM(nn.Module):
403
kv_caches: List[torch.Tensor],
404
attn_metadata: AttentionMetadata,
405
intermediate_tensors: Optional[IntermediateTensors] = None,
400
- ) -> torch.Tensor:
406
+ ) -> Union[torch.Tensor, IntermediateTensors]:
407
hidden_states = self.model(input_ids, positions, kv_caches,
408
attn_metadata, intermediate_tensors)
409
return hidden_states
@@ -411,20 +417,6 @@ class Qwen2MoeForCausalLM(nn.Module):
417
sampling_metadata)
418
return logits
419
414
- def make_empty_intermediate_tensors(
415
- self, batch_size: int, dtype: torch.dtype,
416
- device: torch.device) -> IntermediateTensors:
417
- return IntermediateTensors({
418
- "hidden_states":
419
- torch.zeros((batch_size, self.config.hidden_size),
420
- dtype=dtype,
421
- device=device),
422
- "residual":
423
- torch.zeros((batch_size, self.config.hidden_size),
424
- dtype=dtype,
425
- device=device),
426
- })
427
-
420
def sample(
421
self,
422
logits: Optional[torch.Tensor],
vllm/model_executor/models/qwen2_vl.py
+38
-35
@@ -55,7 +55,6 @@ from vllm.model_executor.layers.quantization import QuantizationConfig
55
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
56
from vllm.model_executor.layers.vocab_parallel_embedding import ParallelLMHead
57
from vllm.model_executor.model_loader.weight_utils import default_weight_loader
58
-from vllm.model_executor.models.interfaces import SupportsMultiModal
58
from vllm.model_executor.models.qwen2 import Qwen2Model
59
from vllm.multimodal import (MULTIMODAL_REGISTRY, MultiModalDataDict,
60
MultiModalInputs)
@@ -68,6 +67,7 @@ from vllm.transformers_utils.configs.qwen2vl import (Qwen2VLConfig,
67
from vllm.transformers_utils.processor import get_processor
68
from vllm.utils import is_cpu
69
70
+from .interfaces import SupportsMultiModal, SupportsPP
71
from .utils import (PPMissingLayer, is_pp_missing_parameter,
72
make_empty_intermediate_tensors_factory)
73
@@ -883,7 +883,8 @@ def input_processor_for_qwen2_vl(ctx: InputContext,
883
"video", get_max_qwen2_vl_video_tokens)
884
@INPUT_REGISTRY.register_dummy_data(dummy_data_for_qwen2_vl)
885
@INPUT_REGISTRY.register_input_processor(input_processor_for_qwen2_vl)
886
-class Qwen2VLForConditionalGeneration(nn.Module, SupportsMultiModal):
886
+class Qwen2VLForConditionalGeneration(nn.Module, SupportsMultiModal,
887
+ SupportsPP):
888
889
def __init__(self,
890
config: Qwen2VLConfig,
@@ -1027,7 +1028,7 @@ class Qwen2VLForConditionalGeneration(nn.Module, SupportsMultiModal):
1028
attn_metadata: AttentionMetadata,
1029
intermediate_tensors: Optional[IntermediateTensors] = None,
1030
**kwargs: object,
1030
- ) -> SamplerOutput:
1031
+ ) -> Union[torch.Tensor, IntermediateTensors]:
1032
"""Run forward pass for Qwen2-VL.
1033
1034
Args:
@@ -1047,41 +1048,43 @@ class Qwen2VLForConditionalGeneration(nn.Module, SupportsMultiModal):
1048
video_grid_thw: Tensor `(n_videos, 3)` of video 3D grid in LLM.
1049
`None` if no videos are passed.
1050
"""
1050
-
1051
- image_input = self._parse_and_validate_image_input(**kwargs)
1052
- video_input = self._parse_and_validate_video_input(**kwargs)
1053
-
1054
- if (image_input is None
1055
- and video_input is None) or not get_pp_group().is_first_rank:
1051
+ if intermediate_tensors is not None:
1052
+ input_ids = None
1053
inputs_embeds = None
1054
else:
1058
- if getattr(self.config, "rope_scaling", {}).get("type",
1059
- None) == "mrope":
1060
- assert positions.ndim == 2 and positions.size(0) == 3, (
1061
- "multimodal section rotary embedding requires "
1062
- f"(3, seq_len) positions, but got {positions.size()}")
1063
-
1064
- inputs_embeds = self.model.embed_tokens(input_ids)
1065
-
1066
- if image_input is not None:
1067
- image_embeds = self._process_image_input(image_input)
1068
- inputs_embeds = self._merge_multimodal_embeddings(
1069
- input_ids,
1070
- inputs_embeds,
1071
- image_embeds,
1072
- placeholder_token_id=self.config.image_token_id,
1073
- )
1055
+ image_input = self._parse_and_validate_image_input(**kwargs)
1056
+ video_input = self._parse_and_validate_video_input(**kwargs)
1057
1075
- if video_input is not None:
1076
- video_embeds = self._process_video_input(video_input)
1077
- inputs_embeds = self._merge_multimodal_embeddings(
1078
- input_ids,
1079
- inputs_embeds,
1080
- video_embeds,
1081
- placeholder_token_id=self.config.video_token_id,
1082
- )
1083
-
1084
- input_ids = None
1058
+ if image_input is None and video_input is None:
1059
+ inputs_embeds = None
1060
+ else:
1061
+ rope_scaling = getattr(self.config, "rope_scaling", {})
1062
+ if rope_scaling.get("type", None) == "mrope":
1063
+ assert positions.ndim == 2 and positions.size(0) == 3, (
1064
+ "multimodal section rotary embedding requires "
1065
+ f"(3, seq_len) positions, but got {positions.size()}")
1066
+
1067
+ inputs_embeds = self.model.embed_tokens(input_ids)
1068
+
1069
+ if image_input is not None:
1070
+ image_embeds = self._process_image_input(image_input)
1071
+ inputs_embeds = self._merge_multimodal_embeddings(
1072
+ input_ids,
1073
+ inputs_embeds,
1074
+ image_embeds,
1075
+ placeholder_token_id=self.config.image_token_id,
1076
+ )
1077
+
1078
+ if video_input is not None:
1079
+ video_embeds = self._process_video_input(video_input)
1080
+ inputs_embeds = self._merge_multimodal_embeddings(
1081
+ input_ids,
1082
+ inputs_embeds,
1083
+ video_embeds,
1084
+ placeholder_token_id=self.config.video_token_id,
1085
+ )
1086
+
1087
+ input_ids = None
1088
1089
hidden_states = self.model(
1090
input_ids=input_ids,
vllm/model_executor/models/siglip.py
+2
-2
@@ -246,7 +246,7 @@ class SiglipParallelAttention(nn.Module):
246
247
def __init__(
248
self,
249
- config,
249
+ config: SiglipVisionConfig,
250
quant_config: Optional[QuantizationConfig] = None,
251
):
252
super().__init__()
@@ -312,7 +312,7 @@ class SiglipMLP(nn.Module):
312
313
def __init__(
314
self,
315
- config,
315
+ config: SiglipVisionConfig,
316
quant_config: Optional[QuantizationConfig] = None,
317
):
318
super().__init__()
vllm/model_executor/models/solar.py
+18
-29
@@ -26,6 +26,7 @@ from typing import Any, Dict, Iterable, List, Optional, Tuple, Union
26
27
import torch
28
from torch import nn
29
+from transformers import PretrainedConfig
30
31
from vllm.attention import Attention, AttentionMetadata
32
from vllm.config import CacheConfig, LoRAConfig
@@ -37,8 +38,7 @@ from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
38
QKVParallelLinear,
39
RowParallelLinear)
40
from vllm.model_executor.layers.logits_processor import LogitsProcessor
40
-from vllm.model_executor.layers.quantization.base_config import (
41
- QuantizationConfig)
41
+from vllm.model_executor.layers.quantization import QuantizationConfig
42
from vllm.model_executor.layers.quantization.compressed_tensors.utils import (
43
get_compressed_tensors_cache_scale)
44
from vllm.model_executor.layers.rotary_embedding import get_rope
@@ -47,14 +47,14 @@ from vllm.model_executor.layers.vocab_parallel_embedding import (
47
DEFAULT_VOCAB_PADDING_SIZE, ParallelLMHead, VocabParallelEmbedding)
48
from vllm.model_executor.model_loader.weight_utils import (
49
default_weight_loader, kv_cache_scales_loader, maybe_remap_kv_scale_name)
50
-from vllm.model_executor.models.interfaces import SupportsLoRA
51
-from vllm.model_executor.models.utils import (PPMissingLayer,
52
- is_pp_missing_parameter,
53
- make_layers)
50
from vllm.model_executor.sampling_metadata import SamplingMetadata
51
from vllm.sequence import IntermediateTensors
52
from vllm.utils import is_hip
53
54
+from .interfaces import SupportsLoRA, SupportsPP
55
+from .utils import (PPMissingLayer, is_pp_missing_parameter,
56
+ make_empty_intermediate_tensors_factory, make_layers)
57
+
58
59
class SolarMLP(nn.Module):
60
@@ -98,7 +98,7 @@ class SolarAttention(nn.Module):
98
99
def __init__(
100
self,
101
- config,
101
+ config: PretrainedConfig,
102
hidden_size: int,
103
num_heads: int,
104
num_kv_heads: int,
@@ -187,7 +187,7 @@ class SolarDecoderLayer(nn.Module):
187
188
def __init__(
189
self,
190
- config,
190
+ config: PretrainedConfig,
191
cache_config: Optional[CacheConfig] = None,
192
quant_config: Optional[QuantizationConfig] = None,
193
prefix: str = "",
@@ -267,7 +267,7 @@ class SolarModel(nn.Module):
267
268
def __init__(
269
self,
270
- config,
270
+ config: PretrainedConfig,
271
cache_config: Optional[CacheConfig] = None,
272
quant_config: Optional[QuantizationConfig] = None,
273
lora_config: Optional[LoRAConfig] = None,
@@ -304,6 +304,10 @@ class SolarModel(nn.Module):
304
else:
305
self.norm = PPMissingLayer()
306
307
+ self.make_empty_intermediate_tensors = (
308
+ make_empty_intermediate_tensors_factory(
309
+ ["hidden_states", "residual"], config.hidden_size))
310
+
311
def get_input_embeddings(self, input_ids: torch.Tensor) -> torch.Tensor:
312
return self.embed_tokens(input_ids)
313
@@ -368,7 +372,7 @@ class SolarModel(nn.Module):
372
return hidden_states
373
374
371
-class SolarForCausalLM(nn.Module, SupportsLoRA):
375
+class SolarForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
376
packed_modules_mapping = {
377
"qkv_proj": [
378
"q_proj",
@@ -406,7 +410,7 @@ class SolarForCausalLM(nn.Module, SupportsLoRA):
410
411
def __init__(
412
self,
409
- config,
413
+ config: PretrainedConfig,
414
cache_config: Optional[CacheConfig] = None,
415
quant_config: Optional[QuantizationConfig] = None,
416
lora_config: Optional[LoRAConfig] = None,
@@ -448,6 +452,9 @@ class SolarForCausalLM(nn.Module, SupportsLoRA):
452
else:
453
self.lm_head = PPMissingLayer()
454
455
+ self.make_empty_intermediate_tensors = (
456
+ self.model.make_empty_intermediate_tensors)
457
+
458
def forward(
459
self,
460
input_ids: torch.Tensor,
@@ -474,24 +481,6 @@ class SolarForCausalLM(nn.Module, SupportsLoRA):
481
next_tokens = self.sampler(logits, sampling_metadata)
482
return next_tokens
483
477
- def make_empty_intermediate_tensors(
478
- self, batch_size: int, dtype: torch.dtype,
479
- device: torch.device) -> IntermediateTensors:
480
- return IntermediateTensors({
481
- "hidden_states":
482
- torch.zeros(
483
- (batch_size, self.config.hidden_size),
484
- dtype=dtype,
485
- device=device,
486
- ),
487
- "residual":
488
- torch.zeros(
489
- (batch_size, self.config.hidden_size),
490
- dtype=dtype,
491
- device=device,
492
- ),
493
- })
494
-
484
def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
485
stacked_params_mapping = [
486
# (param_name, shard_name, shard_id)
vllm/model_executor/models/stablelm.py
+38
-16
@@ -19,7 +19,7 @@
19
# https://huggingface.co/stabilityai/stablelm-3b-4e1t/blob/main/config.json
20
"""Inference-only StabeLM (https://github.com/Stability-AI/StableLM)
21
model compatible with HuggingFace weights."""
22
-from typing import Iterable, List, Optional, Tuple
22
+from typing import Iterable, List, Optional, Tuple, Union
23
24
import torch
25
from torch import nn
@@ -27,14 +27,13 @@ from transformers import PretrainedConfig
27
28
from vllm.attention import Attention, AttentionMetadata
29
from vllm.config import CacheConfig
30
-from vllm.distributed import get_tensor_model_parallel_world_size
30
+from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
31
from vllm.model_executor.layers.activation import SiluAndMul
32
from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
33
QKVParallelLinear,
34
RowParallelLinear)
35
from vllm.model_executor.layers.logits_processor import LogitsProcessor
36
-from vllm.model_executor.layers.quantization.base_config import (
37
- QuantizationConfig)
36
+from vllm.model_executor.layers.quantization import QuantizationConfig
37
from vllm.model_executor.layers.rotary_embedding import get_rope
38
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
39
from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -43,6 +42,10 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
42
from vllm.model_executor.sampling_metadata import SamplingMetadata
43
from vllm.sequence import IntermediateTensors
44
45
+from .interfaces import SupportsPP
46
+from .utils import (is_pp_missing_parameter,
47
+ make_empty_intermediate_tensors_factory, make_layers)
48
+
49
50
class StablelmMLP(nn.Module):
51
@@ -194,19 +197,25 @@ class StableLMEpochModel(nn.Module):
197
def __init__(self,
198
config: PretrainedConfig,
199
cache_config: Optional[CacheConfig] = None,
197
- quant_config: Optional[QuantizationConfig] = None) -> None:
200
+ quant_config: Optional[QuantizationConfig] = None,
201
+ prefix: str = '') -> None:
202
super().__init__()
203
self.embed_tokens = VocabParallelEmbedding(
204
config.vocab_size,
205
config.hidden_size,
206
)
203
- self.layers = nn.ModuleList([
204
- StablelmDecoderLayer(config, cache_config, quant_config)
205
- for _ in range(config.num_hidden_layers)
206
- ])
207
+ self.start_layer, self.end_layer, self.layers = make_layers(
208
+ config.num_hidden_layers,
209
+ lambda prefix: StablelmDecoderLayer(config, cache_config,
210
+ quant_config),
211
+ prefix=f"{prefix}.layers",
212
+ )
213
norm_eps = getattr(config, "norm_eps",
214
getattr(config, "layer_norm_eps", 1e-05))
215
self.norm = nn.LayerNorm(config.hidden_size, eps=norm_eps)
216
+ self.make_empty_intermediate_tensors = (
217
+ make_empty_intermediate_tensors_factory(["hidden_states"],
218
+ config.hidden_size))
219
220
def forward(
221
self,
@@ -214,21 +223,28 @@ class StableLMEpochModel(nn.Module):
223
positions: torch.Tensor,
224
kv_caches: List[torch.Tensor],
225
attn_metadata: AttentionMetadata,
217
- ) -> torch.Tensor:
218
- hidden_states = self.embed_tokens(input_ids)
219
- for i in range(len(self.layers)):
226
+ intermediate_tensors: Optional[IntermediateTensors],
227
+ ) -> Union[torch.Tensor, IntermediateTensors]:
228
+ if get_pp_group().is_first_rank:
229
+ hidden_states = self.embed_tokens(input_ids)
230
+ else:
231
+ assert intermediate_tensors is not None
232
+ hidden_states = intermediate_tensors["hidden_states"]
233
+ for i in range(self.start_layer, self.end_layer):
234
layer = self.layers[i]
235
hidden_states, residual = layer(
236
positions,
237
hidden_states,
224
- kv_caches[i],
238
+ kv_caches[i - self.start_layer],
239
attn_metadata,
240
)
241
+ if not get_pp_group().is_last_rank:
242
+ return IntermediateTensors({"hidden_states": hidden_states})
243
hidden_states = self.norm(hidden_states)
244
return hidden_states
245
246
231
-class StablelmForCausalLM(nn.Module):
247
+class StablelmForCausalLM(nn.Module, SupportsPP):
248
249
def __init__(
250
self,
@@ -247,6 +263,8 @@ class StablelmForCausalLM(nn.Module):
263
self.lm_head.weight = self.model.embed_tokens.weight
264
self.logits_processor = LogitsProcessor(config.vocab_size)
265
self.sampler = Sampler()
266
+ self.make_empty_intermediate_tensors = (
267
+ self.model.make_empty_intermediate_tensors)
268
269
def forward(
270
self,
@@ -255,9 +273,9 @@ class StablelmForCausalLM(nn.Module):
273
kv_caches: List[torch.Tensor],
274
attn_metadata: AttentionMetadata,
275
intermediate_tensors: Optional[IntermediateTensors] = None,
258
- ) -> torch.Tensor:
276
+ ) -> Union[torch.Tensor, IntermediateTensors]:
277
hidden_states = self.model(input_ids, positions, kv_caches,
260
- attn_metadata)
278
+ attn_metadata, intermediate_tensors)
279
return hidden_states
280
281
def compute_logits(
@@ -302,6 +320,8 @@ class StablelmForCausalLM(nn.Module):
320
# Skip loading extra bias for GPTQ models.
321
if name.endswith(".bias") and name not in params_dict:
322
continue
323
+ if is_pp_missing_parameter(name, self):
324
+ continue
325
param = params_dict[name]
326
weight_loader = param.weight_loader
327
weight_loader(param, loaded_weight, shard_id)
@@ -310,6 +330,8 @@ class StablelmForCausalLM(nn.Module):
330
# Skip loading extra bias for GPTQ models.
331
if name.endswith(".bias") and name not in params_dict:
332
continue
333
+ if is_pp_missing_parameter(name, self):
334
+ continue
335
param = params_dict[name]
336
weight_loader = getattr(param, "weight_loader",
337
default_weight_loader)
vllm/model_executor/models/starcoder2.py
+39
-18
@@ -18,7 +18,7 @@
18
# See the License for the specific language governing permissions and
19
# limitations under the License.
20
""" PyTorch Starcoder2 model."""
21
-from typing import Iterable, List, Optional, Tuple
21
+from typing import Iterable, List, Optional, Tuple, Union
22
23
import torch
24
from torch import nn
@@ -26,14 +26,13 @@ from transformers import Starcoder2Config
26
27
from vllm.attention import Attention, AttentionMetadata
28
from vllm.config import CacheConfig
29
-from vllm.distributed import get_tensor_model_parallel_world_size
29
+from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
30
from vllm.model_executor.layers.activation import get_act_fn
31
from vllm.model_executor.layers.linear import (ColumnParallelLinear,
32
QKVParallelLinear,
33
RowParallelLinear)
34
from vllm.model_executor.layers.logits_processor import LogitsProcessor
35
-from vllm.model_executor.layers.quantization.base_config import (
36
- QuantizationConfig)
35
+from vllm.model_executor.layers.quantization import QuantizationConfig
36
from vllm.model_executor.layers.rotary_embedding import get_rope
37
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
38
from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -42,6 +41,10 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
41
from vllm.model_executor.sampling_metadata import SamplingMetadata
42
from vllm.sequence import IntermediateTensors
43
44
+from .interfaces import SupportsPP
45
+from .utils import (is_pp_missing_parameter,
46
+ make_empty_intermediate_tensors_factory, make_layers)
47
+
48
49
class Starcoder2Attention(nn.Module):
50
@@ -195,7 +198,8 @@ class Starcoder2Model(nn.Module):
198
def __init__(self,
199
config: Starcoder2Config,
200
cache_config: Optional[CacheConfig] = None,
198
- quant_config: Optional[QuantizationConfig] = None):
201
+ quant_config: Optional[QuantizationConfig] = None,
202
+ prefix: str = ""):
203
super().__init__()
204
self.config = config
205
self.padding_idx = config.pad_token_id
@@ -204,13 +208,16 @@ class Starcoder2Model(nn.Module):
208
# TODO: consider padding_idx (currently removed)
209
self.embed_tokens = VocabParallelEmbedding(config.vocab_size,
210
config.hidden_size)
207
- self.layers = nn.ModuleList([
208
- Starcoder2DecoderLayer(config,
209
- cache_config,
210
- quant_config=quant_config)
211
- for _ in range(config.num_hidden_layers)
212
- ])
211
+ self.start_layer, self.end_layer, self.layers = make_layers(
212
+ config.num_hidden_layers,
213
+ lambda prefix: Starcoder2DecoderLayer(
214
+ config, cache_config, quant_config=quant_config),
215
+ prefix=f"{prefix}.layers",
216
+ )
217
self.norm = nn.LayerNorm(config.hidden_size, eps=config.norm_epsilon)
218
+ self.make_empty_intermediate_tensors = (
219
+ make_empty_intermediate_tensors_factory(["hidden_states"],
220
+ config.hidden_size))
221
222
def forward(
223
self,
@@ -218,17 +225,25 @@ class Starcoder2Model(nn.Module):
225
positions: torch.Tensor,
226
kv_caches: List[torch.Tensor],
227
attn_metadata: AttentionMetadata,
221
- ) -> torch.Tensor:
222
- hidden_states = self.embed_tokens(input_ids)
223
- for i in range(len(self.layers)):
228
+ intermediate_tensors: Optional[IntermediateTensors],
229
+ ) -> Union[torch.Tensor, IntermediateTensors]:
230
+ if get_pp_group().is_first_rank:
231
+ hidden_states = self.embed_tokens(input_ids)
232
+ else:
233
+ assert intermediate_tensors is not None
234
+ hidden_states = intermediate_tensors["hidden_states"]
235
+ for i in range(self.start_layer, self.end_layer):
236
layer = self.layers[i]
225
- hidden_states = layer(positions, hidden_states, kv_caches[i],
237
+ hidden_states = layer(positions, hidden_states,
238
+ kv_caches[i - self.start_layer],
239
attn_metadata)
240
+ if not get_pp_group().is_last_rank:
241
+ return IntermediateTensors({"hidden_states": hidden_states})
242
hidden_states = self.norm(hidden_states)
243
return hidden_states
244
245
231
-class Starcoder2ForCausalLM(nn.Module):
246
+class Starcoder2ForCausalLM(nn.Module, SupportsPP):
247
248
def __init__(self,
249
config: Starcoder2Config,
@@ -255,6 +270,8 @@ class Starcoder2ForCausalLM(nn.Module):
270
self.logits_processor = LogitsProcessor(self.unpadded_vocab_size,
271
config.vocab_size)
272
self.sampler = Sampler()
273
+ self.make_empty_intermediate_tensors = (
274
+ self.model.make_empty_intermediate_tensors)
275
276
def forward(
277
self,
@@ -263,9 +280,9 @@ class Starcoder2ForCausalLM(nn.Module):
280
kv_caches: List[torch.Tensor],
281
attn_metadata: AttentionMetadata,
282
intermediate_tensors: Optional[IntermediateTensors] = None,
266
- ) -> torch.Tensor:
283
+ ) -> Union[torch.Tensor, IntermediateTensors]:
284
hidden_states = self.model(input_ids, positions, kv_caches,
268
- attn_metadata)
285
+ attn_metadata, intermediate_tensors)
286
return hidden_states
287
288
def compute_logits(
@@ -302,6 +319,8 @@ class Starcoder2ForCausalLM(nn.Module):
319
if weight_name not in name:
320
continue
321
name = name.replace(weight_name, param_name)
322
+ if is_pp_missing_parameter(name, self):
323
+ continue
324
param = params_dict[name]
325
weight_loader = param.weight_loader
326
weight_loader(param, loaded_weight, shard_id)
@@ -309,6 +328,8 @@ class Starcoder2ForCausalLM(nn.Module):
328
else:
329
if self.config.tie_word_embeddings and "lm_head.weight" in name:
330
continue
331
+ if is_pp_missing_parameter(name, self):
332
+ continue
333
param = params_dict[name]
334
weight_loader = getattr(param, "weight_loader",
335
default_weight_loader)
vllm/model_executor/models/ultravox.py
+33
-18
@@ -3,7 +3,7 @@
3
4
import math
5
from array import array
6
-from functools import lru_cache
6
+from functools import cached_property, lru_cache
7
from typing import (Iterable, List, Literal, Mapping, Optional, Tuple,
8
TypedDict, Union, cast)
9
@@ -22,12 +22,10 @@ from vllm.inputs.data import LLMInputs
22
from vllm.inputs.registry import InputContext
23
from vllm.model_executor.layers.activation import SiluAndMul, get_act_fn
24
from vllm.model_executor.layers.layernorm import RMSNorm
25
-from vllm.model_executor.layers.quantization.base_config import (
26
- QuantizationConfig)
27
-from vllm.model_executor.layers.sampler import SamplerOutput
25
+from vllm.model_executor.layers.quantization import QuantizationConfig
26
+from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
27
from vllm.model_executor.model_loader.loader import DefaultModelLoader
28
from vllm.model_executor.model_loader.weight_utils import default_weight_loader
30
-from vllm.model_executor.models.interfaces import SupportsMultiModal
29
from vllm.model_executor.models.utils import (flatten_bn,
30
group_weights_with_prefix,
31
init_vllm_registered_model,
@@ -37,9 +35,12 @@ from vllm.multimodal import MULTIMODAL_REGISTRY
35
from vllm.multimodal.base import MultiModalInputs, NestedTensors
36
from vllm.multimodal.utils import (cached_get_tokenizer,
37
repeat_and_pad_placeholder_tokens)
40
-from vllm.sequence import VLLM_TOKEN_ID_ARRAY_TYPE, SequenceData
38
+from vllm.sequence import (VLLM_TOKEN_ID_ARRAY_TYPE, IntermediateTensors,
39
+ SequenceData)
40
from vllm.transformers_utils.configs.ultravox import UltravoxConfig
41
42
+from .interfaces import SupportsMultiModal, SupportsPP
43
+
44
_AUDIO_PLACEHOLDER_TOKEN = 128002
45
_AUDIO_TOKENS_PER_SECOND = 6.25
46
@@ -323,7 +324,7 @@ class ModifiedWhisperEncoder(WhisperEncoder):
324
"audio", get_ultravox_max_audio_tokens)
325
@INPUT_REGISTRY.register_dummy_data(dummy_data_for_ultravox)
326
@INPUT_REGISTRY.register_input_processor(input_processor_for_ultravox)
326
-class UltravoxModel(nn.Module, SupportsMultiModal):
327
+class UltravoxModel(nn.Module, SupportsMultiModal, SupportsPP):
328
329
def __init__(self,
330
config: UltravoxConfig,
@@ -353,6 +354,16 @@ class UltravoxModel(nn.Module, SupportsMultiModal):
354
revision=None,
355
prefix="language_model."))
356
357
+ self.make_empty_intermediate_tensors = (
358
+ self.language_model.make_empty_intermediate_tensors)
359
+
360
+ @cached_property
361
+ def sampler(self):
362
+ if hasattr(self.language_model, "sampler"):
363
+ return self.language_model.sampler
364
+
365
+ return Sampler()
366
+
367
def _audio_features_to_embeddings(
368
self, input_features: torch.Tensor) -> torch.Tensor:
369
audio_input = input_features.to(self.audio_tower.dtype)
@@ -425,7 +436,7 @@ class UltravoxModel(nn.Module, SupportsMultiModal):
436
kv_caches: List[torch.Tensor],
437
attn_metadata: AttentionMetadata,
438
intermediate_tensors: Optional[torch.Tensor],
428
- **kwargs) -> SamplerOutput:
439
+ **kwargs) -> Union[torch.Tensor, IntermediateTensors]:
440
"""Run forward pass for Ultravox
441
442
One key thing to understand is the `input_ids` already accounts for the
@@ -438,18 +449,22 @@ class UltravoxModel(nn.Module, SupportsMultiModal):
449
Args:
450
audio_features: A batch of audio inputs [B, N, 80, M].
451
"""
441
- audio_input = self._parse_and_validate_audio_input(**kwargs)
442
- if audio_input is not None:
443
- audio_embeddings = self._process_audio_input(audio_input)
444
- inputs_embeds = self.language_model.model.get_input_embeddings(
445
- input_ids)
446
-
447
- inputs_embeds = merge_multimodal_embeddings(
448
- input_ids, inputs_embeds, audio_embeddings,
449
- _AUDIO_PLACEHOLDER_TOKEN)
452
+ if intermediate_tensors is not None:
453
input_ids = None
451
- else:
454
inputs_embeds = None
455
+ else:
456
+ audio_input = self._parse_and_validate_audio_input(**kwargs)
457
+ if audio_input is not None:
458
+ audio_embeddings = self._process_audio_input(audio_input)
459
+ inputs_embeds = self.language_model.model.get_input_embeddings(
460
+ input_ids)
461
+
462
+ inputs_embeds = merge_multimodal_embeddings(
463
+ input_ids, inputs_embeds, audio_embeddings,
464
+ _AUDIO_PLACEHOLDER_TOKEN)
465
+ input_ids = None
466
+ else:
467
+ inputs_embeds = None
468
469
hidden_states = self.language_model.model(
470
input_ids=input_ids,
vllm/model_executor/models/utils.py
+17
-13
@@ -24,7 +24,7 @@ class WeightsGroup(UserDict):
24
when attempting to access a weight component that does not exist.
25
"""
26
27
- def __getitem__(self, key: str) -> int:
27
+ def __getitem__(self, key: str) -> Iterable[Tuple[str, torch.Tensor]]:
28
try:
29
return super().__getitem__(key)
30
except KeyError as exc:
@@ -49,8 +49,7 @@ def filter_weights(weights: Iterable[Tuple[str, torch.Tensor]],
49
50
51
def group_weights_with_prefix(
52
- weights: Iterable[Tuple[str, torch.Tensor]]
53
-) -> Dict[str, Iterable[Tuple[str, torch.Tensor]]]:
52
+ weights: Iterable[Tuple[str, torch.Tensor]], ) -> WeightsGroup:
53
"""
54
Helper function to group weights with prefix
55
"""
@@ -183,10 +182,7 @@ def merge_multimodal_embeddings(input_ids: torch.Tensor,
182
183
class LayerFn(Protocol):
184
186
- def __call__(
187
- self,
188
- prefix="",
189
- ) -> torch.nn.Module:
185
+ def __call__(self, prefix: str) -> torch.nn.Module:
186
...
187
188
@@ -319,8 +315,10 @@ def is_pp_missing_parameter(name: str, model: torch.nn.Module) -> bool:
315
def make_empty_intermediate_tensors_factory(keys: List[str], hidden_size: int):
316
317
def make_empty_intermediate_tensors(
322
- batch_size: int, dtype: torch.dtype,
323
- device: torch.device) -> IntermediateTensors:
318
+ batch_size: int,
319
+ dtype: torch.dtype,
320
+ device: torch.device,
321
+ ) -> IntermediateTensors:
322
return IntermediateTensors({
323
key: torch.zeros((batch_size, hidden_size),
324
dtype=dtype,
@@ -342,8 +340,14 @@ class LLMWrapper(nn.Module):
340
self.model_name = name
341
setattr(self, name, llm)
342
345
- def forward(self, *args, **kwargs) -> Any:
346
- return getattr(self, self.model_name)(*args, **kwargs)
343
+ def __getattr__(self, key: str):
344
+ llm = super().__getattr__(self.model_name)
345
+ if key == self.model_name:
346
+ return llm
347
348
- def embed_tokens(self, *args, **kwargs) -> Any:
349
- return getattr(self, self.model_name).embed_tokens(*args, **kwargs)
348
+ return getattr(llm, key)
349
+
350
+ # We need to explicitly override this
351
+ def __call__(self, *args: Any, **kwargs: Any) -> Any:
352
+ llm = super().__getattr__(self.model_name)
353
+ return llm(*args, **kwargs)
vllm/model_executor/models/xverse.py
+39
-17
@@ -20,7 +20,7 @@
20
# See the License for the specific language governing permissions and
21
# limitations under the License.
22
"""Inference-only Xverse model compatible with HuggingFace weights."""
23
-from typing import Any, Dict, Iterable, List, Optional, Tuple
23
+from typing import Any, Dict, Iterable, List, Optional, Tuple, Union
24
25
import torch
26
from torch import nn
@@ -28,15 +28,14 @@ from transformers import PretrainedConfig
28
29
from vllm.attention import Attention, AttentionMetadata
30
from vllm.config import CacheConfig, LoRAConfig
31
-from vllm.distributed import get_tensor_model_parallel_world_size
31
+from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
32
from vllm.model_executor.layers.activation import SiluAndMul
33
from vllm.model_executor.layers.layernorm import RMSNorm
34
from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
35
QKVParallelLinear,
36
RowParallelLinear)
37
from vllm.model_executor.layers.logits_processor import LogitsProcessor
38
-from vllm.model_executor.layers.quantization.base_config import (
39
- QuantizationConfig)
38
+from vllm.model_executor.layers.quantization import QuantizationConfig
39
from vllm.model_executor.layers.rotary_embedding import get_rope
40
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
41
from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -45,7 +44,9 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
44
from vllm.model_executor.sampling_metadata import SamplingMetadata
45
from vllm.sequence import IntermediateTensors
46
48
-from .interfaces import SupportsLoRA
47
+from .interfaces import SupportsLoRA, SupportsPP
48
+from .utils import (is_pp_missing_parameter,
49
+ make_empty_intermediate_tensors_factory, make_layers)
50
51
52
class XverseMLP(nn.Module):
@@ -227,6 +228,7 @@ class XverseModel(nn.Module):
228
cache_config: Optional[CacheConfig] = None,
229
quant_config: Optional[QuantizationConfig] = None,
230
lora_config: Optional[LoRAConfig] = None,
231
+ prefix: str = "",
232
) -> None:
233
super().__init__()
234
self.config = config
@@ -240,11 +242,16 @@ class XverseModel(nn.Module):
242
config.hidden_size,
243
org_num_embeddings=config.vocab_size,
244
)
243
- self.layers = nn.ModuleList([
244
- XverseDecoderLayer(config, cache_config, quant_config)
245
- for _ in range(config.num_hidden_layers)
246
- ])
245
+ self.start_layer, self.end_layer, self.layers = make_layers(
246
+ config.num_hidden_layers,
247
+ lambda prefix: XverseDecoderLayer(config, cache_config,
248
+ quant_config),
249
+ prefix=f"{prefix}.layers",
250
+ )
251
self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
252
+ self.make_empty_intermediate_tensors = (
253
+ make_empty_intermediate_tensors_factory(
254
+ ["hidden_states", "residual"], config.hidden_size))
255
256
def forward(
257
self,
@@ -252,23 +259,32 @@ class XverseModel(nn.Module):
259
positions: torch.Tensor,
260
kv_caches: List[torch.Tensor],
261
attn_metadata: AttentionMetadata,
255
- ) -> torch.Tensor:
256
- hidden_states = self.embed_tokens(input_ids)
257
- residual = None
258
- for i in range(len(self.layers)):
262
+ intermediate_tensors: Optional[IntermediateTensors],
263
+ ) -> Union[torch.Tensor, IntermediateTensors]:
264
+ if get_pp_group().is_first_rank:
265
+ hidden_states = self.embed_tokens(input_ids)
266
+ residual = None
267
+ else:
268
+ hidden_states = intermediate_tensors["hidden_states"]
269
+ for i in range(self.start_layer, self.end_layer):
270
layer = self.layers[i]
271
hidden_states, residual = layer(
272
positions,
273
hidden_states,
263
- kv_caches[i],
274
+ kv_caches[i - self.start_layer],
275
attn_metadata,
276
residual,
277
)
278
+ if not get_pp_group().is_last_rank:
279
+ return IntermediateTensors({
280
+ "hidden_states": hidden_states,
281
+ "residual": residual
282
+ })
283
hidden_states, _ = self.norm(hidden_states, residual)
284
return hidden_states
285
286
271
-class XverseForCausalLM(nn.Module, SupportsLoRA):
287
+class XverseForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
288
packed_modules_mapping = {
289
"qkv_proj": [
290
"q_proj",
@@ -317,6 +333,8 @@ class XverseForCausalLM(nn.Module, SupportsLoRA):
333
self.lm_head.weight = self.model.embed_tokens.weight
334
self.logits_processor = LogitsProcessor(config.vocab_size)
335
self.sampler = Sampler()
336
+ self.make_empty_intermediate_tensors = (
337
+ self.model.make_empty_intermediate_tensors)
338
339
def forward(
340
self,
@@ -325,9 +343,9 @@ class XverseForCausalLM(nn.Module, SupportsLoRA):
343
kv_caches: List[torch.Tensor],
344
attn_metadata: AttentionMetadata,
345
intermediate_tensors: Optional[IntermediateTensors] = None,
328
- ) -> torch.Tensor:
346
+ ) -> Union[torch.Tensor, IntermediateTensors]:
347
hidden_states = self.model(input_ids, positions, kv_caches,
330
- attn_metadata)
348
+ attn_metadata, intermediate_tensors)
349
return hidden_states
350
351
def compute_logits(
@@ -368,6 +386,8 @@ class XverseForCausalLM(nn.Module, SupportsLoRA):
386
# Skip loading extra bias for GPTQ models.
387
if name.endswith(".bias") and name not in params_dict:
388
continue
389
+ if is_pp_missing_parameter(name, self):
390
+ continue
391
param = params_dict[name]
392
weight_loader = param.weight_loader
393
weight_loader(param, loaded_weight, shard_id)
@@ -376,6 +396,8 @@ class XverseForCausalLM(nn.Module, SupportsLoRA):
396
# Skip loading extra bias for GPTQ models.
397
if name.endswith(".bias") and name not in params_dict:
398
continue
399
+ if is_pp_missing_parameter(name, self):
400
+ continue
401
param = params_dict[name]
402
weight_loader = getattr(param, "weight_loader",
403
default_weight_loader)