@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 0f6d7a9a34
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 10:45 UTC

[Models] Add remaining model PP support (#7168)

Signed-off-by: Muralidhar Andoorveedu <muralidhar.andoorveedu@centml.ai> Signed-off-by: Murali Andoorveedu <muralidhar.andoorveedu@centml.ai> Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>

Murali Andoorveedu committed Oct 3, 2024 at 19:56 UTC 0f6d7a9a347944bffd2204cbf9686299e9dd6557
69 files changed +2583 -1342
.buildkite/test-pipeline.yaml
+3 -1
@@ -146,7 +146,9 @@ steps:
146 source_file_dependencies:
147 - vllm/
148 - tests/test_regression
149 - command: pytest -v -s test_regression.py
149 + commands:
150 + - pip install modelscope
151 + - pytest -v -s test_regression.py
152 working_dir: "/vllm-workspace/tests" # optional
153
154 - label: Engine Test # 10min
docs/source/models/supported_models.rst
+81 -16
@@ -12,201 +12,249 @@ Alongside each architecture, we include some popular models that use it.
12 Decoder-only Language Models
13 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
14 .. list-table::
15 - :widths: 25 25 50 5
15 + :widths: 25 25 50 5 5
16 :header-rows: 1
17
18 * - Architecture
19 - Models
20 - Example HuggingFace Models
21 - :ref:`LoRA <lora>`
22 + - :ref:`PP <distributed_serving>`
23 * - :code:`AquilaForCausalLM`
24 - Aquila, Aquila2
25 - :code:`BAAI/Aquila-7B`, :code:`BAAI/AquilaChat-7B`, etc.
26 - ✅︎
27 + - ✅︎
28 * - :code:`ArcticForCausalLM`
29 - Arctic
30 - :code:`Snowflake/snowflake-arctic-base`, :code:`Snowflake/snowflake-arctic-instruct`, etc.
31 -
32 + - ✅︎
33 * - :code:`BaiChuanForCausalLM`
34 - Baichuan2, Baichuan
35 - :code:`baichuan-inc/Baichuan2-13B-Chat`, :code:`baichuan-inc/Baichuan-7B`, etc.
36 - ✅︎
37 + - ✅︎
38 * - :code:`BloomForCausalLM`
39 - BLOOM, BLOOMZ, BLOOMChat
40 - :code:`bigscience/bloom`, :code:`bigscience/bloomz`, etc.
41 -
42 + - ✅︎
43 * - :code:`ChatGLMModel`
44 - ChatGLM
45 - :code:`THUDM/chatglm2-6b`, :code:`THUDM/chatglm3-6b`, etc.
46 - ✅︎
47 + - ✅︎
48 * - :code:`CohereForCausalLM`
49 - Command-R
50 - :code:`CohereForAI/c4ai-command-r-v01`, etc.
45 - -
51 + - ✅︎
52 + - ✅︎
53 * - :code:`DbrxForCausalLM`
54 - DBRX
55 - :code:`databricks/dbrx-base`, :code:`databricks/dbrx-instruct`, etc.
56 -
57 + - ✅︎
58 * - :code:`DeciLMForCausalLM`
59 - DeciLM
60 - :code:`Deci/DeciLM-7B`, :code:`Deci/DeciLM-7B-instruct`, etc.
61 -
62 + - ✅︎
63 * - :code:`DeepseekForCausalLM`
64 - DeepSeek
65 - :code:`deepseek-ai/deepseek-llm-67b-base`, :code:`deepseek-ai/deepseek-llm-7b-chat` etc.
66 -
67 + - ✅︎
68 * - :code:`DeepseekV2ForCausalLM`
69 - DeepSeek-V2
70 - :code:`deepseek-ai/DeepSeek-V2`, :code:`deepseek-ai/DeepSeek-V2-Chat` etc.
71 -
72 + - ✅︎
73 * - :code:`ExaoneForCausalLM`
74 - EXAONE-3
75 - :code:`LGAI-EXAONE/EXAONE-3.0-7.8B-Instruct`, etc.
76 - ✅︎
77 + - ✅︎
78 * - :code:`FalconForCausalLM`
79 - Falcon
80 - :code:`tiiuae/falcon-7b`, :code:`tiiuae/falcon-40b`, :code:`tiiuae/falcon-rw-7b`, etc.
81 -
82 + - ✅︎
83 * - :code:`GemmaForCausalLM`
84 - Gemma
85 - :code:`google/gemma-2b`, :code:`google/gemma-7b`, etc.
86 - ✅︎
87 + - ✅︎
88 * - :code:`Gemma2ForCausalLM`
89 - Gemma2
90 - :code:`google/gemma-2-9b`, :code:`google/gemma-2-27b`, etc.
91 - ✅︎
92 + - ✅︎
93 * - :code:`GPT2LMHeadModel`
94 - GPT-2
95 - :code:`gpt2`, :code:`gpt2-xl`, etc.
96 -
97 + - ✅︎
98 * - :code:`GPTBigCodeForCausalLM`
99 - StarCoder, SantaCoder, WizardCoder
100 - :code:`bigcode/starcoder`, :code:`bigcode/gpt_bigcode-santacoder`, :code:`WizardLM/WizardCoder-15B-V1.0`, etc.
101 - ✅︎
102 + - ✅︎
103 * - :code:`GPTJForCausalLM`
104 - GPT-J
105 - :code:`EleutherAI/gpt-j-6b`, :code:`nomic-ai/gpt4all-j`, etc.
106 -
107 + - ✅︎
108 * - :code:`GPTNeoXForCausalLM`
109 - GPT-NeoX, Pythia, OpenAssistant, Dolly V2, StableLM
110 - :code:`EleutherAI/gpt-neox-20b`, :code:`EleutherAI/pythia-12b`, :code:`OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5`, :code:`databricks/dolly-v2-12b`, :code:`stabilityai/stablelm-tuned-alpha-7b`, etc.
111 -
112 + - ✅︎
113 * - :code:`GraniteForCausalLM`
114 - PowerLM
115 - :code:`ibm/PowerLM-3b` etc.
116 - ✅︎
117 + - ✅︎
118 * - :code:`GraniteMoeForCausalLM`
119 - PowerMoE
120 - :code:`ibm/PowerMoE-3b` etc.
121 - ✅︎
122 + - ✅︎
123 * - :code:`InternLMForCausalLM`
124 - InternLM
125 - :code:`internlm/internlm-7b`, :code:`internlm/internlm-chat-7b`, etc.
126 - ✅︎
127 + - ✅︎
128 * - :code:`InternLM2ForCausalLM`
129 - InternLM2
130 - :code:`internlm/internlm2-7b`, :code:`internlm/internlm2-chat-7b`, etc.
131 -
132 + - ✅︎
133 * - :code:`JAISLMHeadModel`
134 - Jais
135 - :code:`core42/jais-13b`, :code:`core42/jais-13b-chat`, :code:`core42/jais-30b-v3`, :code:`core42/jais-30b-chat-v3`, etc.
136 -
137 + - ✅︎
138 * - :code:`JambaForCausalLM`
139 - Jamba
140 - :code:`ai21labs/AI21-Jamba-1.5-Large`, :code:`ai21labs/AI21-Jamba-1.5-Mini`, :code:`ai21labs/Jamba-v0.1`, etc.
141 - ✅︎
142 + -
143 * - :code:`LlamaForCausalLM`
144 - Llama 3.1, Llama 3, Llama 2, LLaMA, Yi
145 - :code:`meta-llama/Meta-Llama-3.1-405B-Instruct`, :code:`meta-llama/Meta-Llama-3.1-70B`, :code:`meta-llama/Meta-Llama-3-70B-Instruct`, :code:`meta-llama/Llama-2-70b-hf`, :code:`01-ai/Yi-34B`, etc.
146 - ✅︎
147 + - ✅︎
148 * - :code:`MiniCPMForCausalLM`
149 - MiniCPM
150 - :code:`openbmb/MiniCPM-2B-sft-bf16`, :code:`openbmb/MiniCPM-2B-dpo-bf16`, etc.
125 - -
151 + - ✅︎
152 + - ✅︎
153 * - :code:`MiniCPM3ForCausalLM`
154 - MiniCPM3
155 - :code:`openbmb/MiniCPM3-4B`, etc.
129 - -
156 + - ✅︎
157 + - ✅︎
158 * - :code:`MistralForCausalLM`
159 - Mistral, Mistral-Instruct
160 - :code:`mistralai/Mistral-7B-v0.1`, :code:`mistralai/Mistral-7B-Instruct-v0.1`, etc.
161 - ✅︎
162 + - ✅︎
163 * - :code:`MixtralForCausalLM`
164 - Mixtral-8x7B, Mixtral-8x7B-Instruct
165 - :code:`mistralai/Mixtral-8x7B-v0.1`, :code:`mistralai/Mixtral-8x7B-Instruct-v0.1`, :code:`mistral-community/Mixtral-8x22B-v0.1`, etc.
166 - ✅︎
167 + - ✅︎
168 * - :code:`MPTForCausalLM`
169 - MPT, MPT-Instruct, MPT-Chat, MPT-StoryWriter
170 - :code:`mosaicml/mpt-7b`, :code:`mosaicml/mpt-7b-storywriter`, :code:`mosaicml/mpt-30b`, etc.
171 -
172 + - ✅︎
173 * - :code:`NemotronForCausalLM`
174 - Nemotron-3, Nemotron-4, Minitron
175 - :code:`nvidia/Minitron-8B-Base`, :code:`mgoin/Nemotron-4-340B-Base-hf-FP8`, etc.
176 - ✅︎
146 - * - :code:`OLMoEForCausalLM`
147 - - OLMoE
148 - - :code:`allenai/OLMoE-1B-7B-0924`, :code:`allenai/OLMoE-1B-7B-0924-Instruct`, etc.
149 - -
177 + - ✅︎
178 * - :code:`OLMoForCausalLM`
179 - OLMo
180 - :code:`allenai/OLMo-1B-hf`, :code:`allenai/OLMo-7B-hf`, etc.
181 -
182 + - ✅︎
183 + * - :code:`OLMoEForCausalLM`
184 + - OLMoE
185 + - :code:`allenai/OLMoE-1B-7B-0924`, :code:`allenai/OLMoE-1B-7B-0924-Instruct`, etc.
186 + - ✅︎
187 + - ✅︎
188 * - :code:`OPTForCausalLM`
189 - OPT, OPT-IML
190 - :code:`facebook/opt-66b`, :code:`facebook/opt-iml-max-30b`, etc.
191 -
192 + - ✅︎
193 * - :code:`OrionForCausalLM`
194 - Orion
195 - :code:`OrionStarAI/Orion-14B-Base`, :code:`OrionStarAI/Orion-14B-Chat`, etc.
196 -
197 + - ✅︎
198 * - :code:`PhiForCausalLM`
199 - Phi
200 - :code:`microsoft/phi-1_5`, :code:`microsoft/phi-2`, etc.
201 - ✅︎
202 + - ✅︎
203 * - :code:`Phi3ForCausalLM`
204 - Phi-3
205 - :code:`microsoft/Phi-3-mini-4k-instruct`, :code:`microsoft/Phi-3-mini-128k-instruct`, :code:`microsoft/Phi-3-medium-128k-instruct`, etc.
169 - -
206 + - ✅︎
207 + - ✅︎
208 * - :code:`Phi3SmallForCausalLM`
209 - Phi-3-Small
210 - :code:`microsoft/Phi-3-small-8k-instruct`, :code:`microsoft/Phi-3-small-128k-instruct`, etc.
211 -
212 + - ✅︎
213 * - :code:`PhiMoEForCausalLM`
214 - Phi-3.5-MoE
215 - :code:`microsoft/Phi-3.5-MoE-instruct`, etc.
177 - -
216 + - ✅︎
217 + - ✅︎
218 * - :code:`PersimmonForCausalLM`
219 - Persimmon
220 - :code:`adept/persimmon-8b-base`, :code:`adept/persimmon-8b-chat`, etc.
221 -
222 + - ✅︎
223 * - :code:`QWenLMHeadModel`
224 - Qwen
225 - :code:`Qwen/Qwen-7B`, :code:`Qwen/Qwen-7B-Chat`, etc.
226 -
227 + - ✅︎
228 * - :code:`Qwen2ForCausalLM`
229 - Qwen2
230 - :code:`Qwen/Qwen2-beta-7B`, :code:`Qwen/Qwen2-beta-7B-Chat`, etc.
231 - ✅︎
232 + - ✅︎
233 * - :code:`Qwen2MoeForCausalLM`
234 - Qwen2MoE
235 - :code:`Qwen/Qwen1.5-MoE-A2.7B`, :code:`Qwen/Qwen1.5-MoE-A2.7B-Chat`, etc.
236 -
237 + - ✅︎
238 * - :code:`StableLmForCausalLM`
239 - StableLM
240 - :code:`stabilityai/stablelm-3b-4e1t`, :code:`stabilityai/stablelm-base-alpha-7b-v2`, etc.
241 -
242 + - ✅︎
243 * - :code:`Starcoder2ForCausalLM`
244 - Starcoder2
245 - :code:`bigcode/starcoder2-3b`, :code:`bigcode/starcoder2-7b`, :code:`bigcode/starcoder2-15b`, etc.
246 -
247 + - ✅︎
248 * - :code:`SolarForCausalLM`
203 - - EXAONE-3
249 + - Solar Pro
250 - :code:`upstage/solar-pro-preview-instruct`, etc.
205 - -
251 + - ✅︎
252 + - ✅︎
253 * - :code:`XverseForCausalLM`
207 - - Xverse
254 + - XVERSE
255 - :code:`xverse/XVERSE-7B-Chat`, :code:`xverse/XVERSE-13B-Chat`, :code:`xverse/XVERSE-65B-Chat`, etc.
209 - -
256 + - ✅︎
257 + - ✅︎
258
259 .. note::
260 Currently, the ROCm version of vLLM supports Mistral and Mixtral only for context lengths up to 4096.
@@ -217,7 +265,7 @@ Multimodal Language Models
265 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
266
267 .. list-table::
220 - :widths: 25 25 25 25 5
268 + :widths: 25 25 25 25 5 5
269 :header-rows: 1
270
271 * - Architecture
@@ -225,86 +273,103 @@ Multimodal Language Models
273 - Modalities
274 - Example HuggingFace Models
275 - :ref:`LoRA <lora>`
276 + - :ref:`PP <distributed_serving>`
277 * - :code:`Blip2ForConditionalGeneration`
278 - BLIP-2
279 - Image\ :sup:`E`
280 - :code:`Salesforce/blip2-opt-2.7b`, :code:`Salesforce/blip2-opt-6.7b`, etc.
281 -
282 + - ✅︎
283 * - :code:`ChameleonForConditionalGeneration`
284 - Chameleon
285 - Image
286 - :code:`facebook/chameleon-7b` etc.
287 -
288 + - ✅︎
289 * - :code:`FuyuForCausalLM`
290 - Fuyu
291 - Image
292 - :code:`adept/fuyu-8b` etc.
293 -
294 + - ✅︎
295 * - :code:`InternVLChatModel`
296 - InternVL2
297 - Image\ :sup:`E+`
298 - :code:`OpenGVLab/InternVL2-4B`, :code:`OpenGVLab/InternVL2-8B`, etc.
299 -
300 + - ✅︎
301 * - :code:`LlavaForConditionalGeneration`
302 - LLaVA-1.5
303 - Image\ :sup:`E+`
304 - :code:`llava-hf/llava-1.5-7b-hf`, :code:`llava-hf/llava-1.5-13b-hf`, etc.
305 -
306 + - ✅︎
307 * - :code:`LlavaNextForConditionalGeneration`
308 - LLaVA-NeXT
309 - Image\ :sup:`E+`
310 - :code:`llava-hf/llava-v1.6-mistral-7b-hf`, :code:`llava-hf/llava-v1.6-vicuna-7b-hf`, etc.
311 -
312 + - ✅︎
313 * - :code:`LlavaNextVideoForConditionalGeneration`
314 - LLaVA-NeXT-Video
315 - Video
316 - :code:`llava-hf/LLaVA-NeXT-Video-7B-hf`, etc.
317 -
318 + - ✅︎
319 * - :code:`LlavaOnevisionForConditionalGeneration`
320 - LLaVA-Onevision
321 - Image\ :sup:`+` / Video
322 - :code:`llava-hf/llava-onevision-qwen2-7b-ov-hf`, :code:`llava-hf/llava-onevision-qwen2-0.5b-ov-hf`, etc.
323 -
324 + - ✅︎
325 * - :code:`MiniCPMV`
326 - MiniCPM-V
327 - Image\ :sup:`+`
328 - :code:`openbmb/MiniCPM-V-2` (see note), :code:`openbmb/MiniCPM-Llama3-V-2_5`, :code:`openbmb/MiniCPM-V-2_6`, etc.
272 - -
329 + - ✅︎
330 + - ✅︎
331 * - :code:`MllamaForConditionalGeneration`
332 - Llama 3.2
333 - Image
334 - :code:`meta-llama/Llama-3.2-90B-Vision-Instruct`, :code:`meta-llama/Llama-3.2-11B-Vision`, etc.
335 -
336 + -
337 * - :code:`PaliGemmaForConditionalGeneration`
338 - PaliGemma
339 - Image\ :sup:`E`
340 - :code:`google/paligemma-3b-pt-224`, :code:`google/paligemma-3b-mix-224`, etc.
341 -
342 + - ✅︎
343 * - :code:`Phi3VForCausalLM`
344 - Phi-3-Vision, Phi-3.5-Vision
345 - Image\ :sup:`E+`
346 - :code:`microsoft/Phi-3-vision-128k-instruct`, :code:`microsoft/Phi-3.5-vision-instruct` etc.
347 -
348 + - ✅︎
349 * - :code:`PixtralForConditionalGeneration`
350 - Pixtral
351 - Image\ :sup:`+`
352 - :code:`mistralai/Pixtral-12B-2409`
353 -
354 + - ✅︎
355 * - :code:`QWenLMHeadModel`
356 - Qwen-VL
357 - Image\ :sup:`E+`
358 - :code:`Qwen/Qwen-VL`, :code:`Qwen/Qwen-VL-Chat`, etc.
359 -
360 + - ✅︎
361 * - :code:`Qwen2VLForConditionalGeneration`
362 - Qwen2-VL
363 - Image\ :sup:`E+` / Video\ :sup:`+`
364 - :code:`Qwen/Qwen2-VL-2B-Instruct`, :code:`Qwen/Qwen2-VL-7B-Instruct`, :code:`Qwen/Qwen2-VL-72B-Instruct`, etc.
365 -
366 + - ✅︎
367 * - :code:`UltravoxModel`
368 - Ultravox
369 - Audio\ :sup:`E+`
370 - :code:`fixie-ai/ultravox-v0_3`
371 -
372 + - ✅︎
373
374 | :sup:`E` Pre-computed embeddings can be inputted for this modality.
375 | :sup:`+` Multiple items can be inputted per text prompt for this modality.
requirements-test.txt
+2 -2
@@ -10,8 +10,8 @@ pytest-shard
10 awscli
11 einops # required for MPT, qwen-vl and Mamba
12 httpx
13 -librosa # required for audio test
14 -opencv-python # required for video test
13 +librosa # required for audio tests
14 +opencv-python # required for video tests
15 peft
16 requests
17 ray[adag]==2.35
tests/distributed/test_pipeline_parallel.py
+240 -57
@@ -6,6 +6,8 @@ WARNING: This test runs in both single-node (4 GPUs) and multi-node
6 to fail.
7 """
8 import os
9 +from dataclasses import dataclass
10 +from typing import List, NamedTuple, Optional
11
12 import pytest
13
@@ -18,49 +20,256 @@ logger = init_logger("test_pipeline_parallel")
20 VLLM_MULTI_NODE = os.getenv("VLLM_MULTI_NODE", "0") == "1"
21
22
23 +class ParallelSetup(NamedTuple):
24 + tp_size: int
25 + pp_size: int
26 + eager_mode: bool
27 + chunked_prefill: bool
28 +
29 +
30 +@dataclass
31 +class PPTestSettings:
32 + parallel_setups: List[ParallelSetup]
33 + distributed_backends: List[str]
34 + trust_remote_code: bool
35 + tokenizer_mode: Optional[str]
36 +
37 + @staticmethod
38 + def detailed(
39 + *,
40 + tp_base: int = 1,
41 + pp_base: int = 2,
42 + trust_remote_code: bool = False,
43 + tokenizer_mode: Optional[str] = None,
44 + ):
45 + return PPTestSettings(
46 + parallel_setups=[
47 + ParallelSetup(tp_size=tp_base,
48 + pp_size=pp_base,
49 + eager_mode=False,
50 + chunked_prefill=False),
51 + ParallelSetup(tp_size=tp_base,
52 + pp_size=2 * pp_base,
53 + eager_mode=False,
54 + chunked_prefill=True),
55 + ParallelSetup(tp_size=tp_base,
56 + pp_size=2 * pp_base,
57 + eager_mode=True,
58 + chunked_prefill=False),
59 + ParallelSetup(tp_size=2 * tp_base,
60 + pp_size=pp_base,
61 + eager_mode=False,
62 + chunked_prefill=True),
63 + ParallelSetup(tp_size=2 * tp_base,
64 + pp_size=pp_base,
65 + eager_mode=True,
66 + chunked_prefill=False),
67 + ],
68 + distributed_backends=["mp", "ray"],
69 + trust_remote_code=trust_remote_code,
70 + tokenizer_mode=tokenizer_mode,
71 + )
72 +
73 + @staticmethod
74 + def fast(
75 + *,
76 + tp_base: int = 1,
77 + pp_base: int = 2,
78 + trust_remote_code: bool = False,
79 + tokenizer_mode: Optional[str] = None,
80 + ):
81 + return PPTestSettings(
82 + parallel_setups=[
83 + ParallelSetup(tp_size=tp_base,
84 + pp_size=pp_base,
85 + eager_mode=True,
86 + chunked_prefill=False),
87 + ],
88 + distributed_backends=["mp"],
89 + trust_remote_code=trust_remote_code,
90 + tokenizer_mode=tokenizer_mode,
91 + )
92 +
93 + def iter_params(self, model_name: str):
94 + for parallel_setup in self.parallel_setups:
95 + for distributed_backend in self.distributed_backends:
96 + yield (model_name, parallel_setup, distributed_backend,
97 + self.trust_remote_code, self.tokenizer_mode)
98 +
99 +
100 +# yapf: disable
101 +GENERATION_MODEL_SETTINGS = {
102 + # [DETAILED TESTS]
103 + "meta-llama/Meta-Llama-3-8B": PPTestSettings.detailed(),
104 + # [FAST TESTS]
105 + # Uses Llama
106 + # "BAAI/AquilaChat-7B": PPTestSettings.fast(),
107 + # TODO: Test on larger GPU
108 + # "Snowflake/snowflake-arctic-instruct": PPTestSettings.fast(trust_remote_code=True), # noqa: E501
109 + "baichuan-inc/Baichuan-7B": PPTestSettings.fast(trust_remote_code=True),
110 + "baichuan-inc/Baichuan2-13B-Chat": PPTestSettings.fast(trust_remote_code=True), # noqa: E501
111 + "bigscience/bloomz-1b1": PPTestSettings.fast(),
112 + "THUDM/chatglm3-6b": PPTestSettings.fast(trust_remote_code=True),
113 + "CohereForAI/c4ai-command-r-v01": PPTestSettings.fast(tp_base=2, trust_remote_code=True), # noqa: E501
114 + # TODO: Test on larger GPU
115 + # "databricks/dbrx-instruct": PPTestSettings.fast(),
116 + "Deci/DeciLM-7B-instruct": PPTestSettings.fast(trust_remote_code=True),
117 + "deepseek-ai/deepseek-llm-7b-chat": PPTestSettings.fast(),
118 + "deepseek-ai/DeepSeek-V2-Lite-Chat": PPTestSettings.fast(trust_remote_code=True), # noqa: E501
119 + "LGAI-EXAONE/EXAONE-3.0-7.8B-Instruct": PPTestSettings.fast(),
120 + "tiiuae/falcon-7b": PPTestSettings.fast(),
121 + "google/gemma-2b": PPTestSettings.fast(),
122 + "google/gemma-2-9b": PPTestSettings.fast(),
123 + "gpt2": PPTestSettings.fast(),
124 + "bigcode/starcoder": PPTestSettings.fast(),
125 + "EleutherAI/gpt-j-6b": PPTestSettings.fast(),
126 + "EleutherAI/pythia-12b": PPTestSettings.fast(),
127 + "ibm/PowerLM-3b": PPTestSettings.fast(),
128 + "ibm/PowerMoE-3b": PPTestSettings.fast(),
129 + # Uses Llama
130 + # "internlm/internlm-chat-7b": PPTestSettings.fast(),
131 + "internlm/internlm2-chat-7b": PPTestSettings.fast(trust_remote_code=True),
132 + "core42/jais-13b-chat": PPTestSettings.fast(),
133 + # TODO: Implement PP
134 + # "ai21labs/AI21-Jamba-1.5-Mini": PPTestSettings.fast(),
135 + "openbmb/MiniCPM-2B-sft-bf16": PPTestSettings.fast(trust_remote_code=True),
136 + "openbmb/MiniCPM3-4B": PPTestSettings.fast(trust_remote_code=True),
137 + # Uses Llama
138 + # "mistralai/Mistral-7B-Instruct-v0.1": PPTestSettings.fast(),
139 + "mistralai/Mixtral-8x7B-Instruct-v0.1": PPTestSettings.fast(tp_base=4),
140 + "mosaicml/mpt-7b": PPTestSettings.fast(),
141 + "nvidia/Minitron-8B-Base": PPTestSettings.fast(),
142 + "allenai/OLMoE-1B-7B-0924-Instruct": PPTestSettings.fast(),
143 + "allenai/OLMo-1B-hf": PPTestSettings.fast(),
144 + "facebook/opt-iml-max-1.3b": PPTestSettings.fast(),
145 + "OrionStarAI/Orion-14B-Chat": PPTestSettings.fast(trust_remote_code=True),
146 + "microsoft/phi-2": PPTestSettings.fast(),
147 + "microsoft/Phi-3-mini-4k-instruct": PPTestSettings.fast(),
148 + "microsoft/Phi-3-small-8k-instruct": PPTestSettings.fast(trust_remote_code=True), # noqa: E501
149 + # FIXME: https://github.com/vllm-project/vllm/issues/8553
150 + # "microsoft/Phi-3.5-MoE-instruct": PPTestSettings.fast(trust_remote_code=True), # noqa: E501
151 + "adept/persimmon-8b-chat": PPTestSettings.fast(),
152 + "Qwen/Qwen-7B-Chat": PPTestSettings.fast(trust_remote_code=True),
153 + "Qwen/Qwen2-beta-7B-Chat": PPTestSettings.fast(),
154 + "Qwen/Qwen1.5-MoE-A2.7B-Chat": PPTestSettings.fast(),
155 + "stabilityai/stablelm-3b-4e1t": PPTestSettings.fast(),
156 + "bigcode/starcoder2-3b": PPTestSettings.fast(),
157 + "upstage/solar-pro-preview-instruct": PPTestSettings.fast(tp_base=2),
158 + # FIXME: Cannot load tokenizer in latest transformers version
159 + # "xverse/XVERSE-7B-Chat": PPTestSettings.fast(trust_remote_code=True),
160 +}
161 +
162 +EMBEDDING_MODEL_SETTINGS = { # type: ignore[var-annotated]
163 + # [FAST TESTS]
164 + # Uses Llama
165 + # "intfloat/e5-mistral-7b-instruct": PPTestSettings.fast(),
166 +}
167 +
168 +MULTIMODAL_MODEL_SETTINGS = {
169 + # [FAST TESTS]
170 + "Salesforce/blip2-opt-2.7b": PPTestSettings.fast(),
171 + "facebook/chameleon-7b": PPTestSettings.fast(),
172 + "adept/fuyu-8b": PPTestSettings.fast(),
173 + "OpenGVLab/InternVL2-1B": PPTestSettings.fast(trust_remote_code=True),
174 + "llava-hf/llava-1.5-7b-hf": PPTestSettings.fast(),
175 + "llava-hf/llava-v1.6-mistral-7b-hf": PPTestSettings.fast(),
176 + "llava-hf/LLaVA-NeXT-Video-7B-hf": PPTestSettings.fast(),
177 + "llava-hf/llava-onevision-qwen2-0.5b-ov-hf": PPTestSettings.fast(),
178 + "openbmb/MiniCPM-Llama3-V-2_5": PPTestSettings.fast(trust_remote_code=True),
179 + # TODO: Implement PP
180 + # "meta-llama/Llama-3.2-11B-Vision-Instruct": PPTestSettings.fast(),
181 + "microsoft/Phi-3-vision-128k-instruct": PPTestSettings.fast(trust_remote_code=True), # noqa: E501
182 + "mistralai/Pixtral-12B-2409": PPTestSettings.fast(tp_base=2, tokenizer_mode="mistral"), # noqa: E501
183 + "Qwen/Qwen-VL-Chat": PPTestSettings.fast(trust_remote_code=True),
184 + "Qwen/Qwen2-VL-2B-Instruct": PPTestSettings.fast(),
185 + "fixie-ai/ultravox-v0_3": PPTestSettings.fast(),
186 +}
187 +
188 +CONDITIONAL_GENERATION_MODEL_SETTINGS = { # type: ignore[var-annotated]
189 + # [FAST TESTS]
190 + # TODO: Implement PP
191 + # "facebook/bart-base": PPTestSettings.fast(),
192 +}
193 +# yapf: enable
194 +
195 +MODEL_SETTINGS = {
196 + **GENERATION_MODEL_SETTINGS,
197 + **EMBEDDING_MODEL_SETTINGS,
198 + **MULTIMODAL_MODEL_SETTINGS,
199 +}
200 +
201 +# You can update this on your local machine to run specific tests
202 +TEST_MODELS = [
203 + "meta-llama/Meta-Llama-3-8B",
204 + "facebook/chameleon-7b",
205 + "OpenGVLab/InternVL2-1B",
206 + "microsoft/Phi-3-vision-128k-instruct",
207 + "mistralai/Pixtral-12B-2409",
208 + "fixie-ai/ultravox-v0_3",
209 +]
210 +
211 +
212 @pytest.mark.parametrize(
22 - ("TP_SIZE, PP_SIZE, EAGER_MODE, CHUNKED_PREFILL, TRUST_REMOTE_CODE, "
23 - "MODEL_NAME, DIST_BACKEND"),
213 + ("model_name", "parallel_setup", "distributed_backend",
214 + "trust_remote_code", "tokenizer_mode"),
215 [
25 - (2, 2, 0, 1, 0, "meta-llama/Meta-Llama-3-8B", "mp"),
26 - (2, 2, 1, 0, 0, "meta-llama/Meta-Llama-3-8B", "mp"),
27 - (1, 3, 0, 0, 0, "meta-llama/Meta-Llama-3-8B", "mp"),
28 - (1, 4, 0, 1, 0, "meta-llama/Meta-Llama-3-8B", "mp"),
29 - (1, 4, 1, 0, 0, "meta-llama/Meta-Llama-3-8B", "mp"),
30 - (1, 3, 0, 0, 0, "meta-llama/Meta-Llama-3-8B", "ray"),
31 - (1, 4, 0, 1, 0, "meta-llama/Meta-Llama-3-8B", "ray"),
32 - (1, 4, 1, 0, 0, "meta-llama/Meta-Llama-3-8B", "ray"),
33 - (2, 2, 1, 0, 0, "meta-llama/Meta-Llama-3-8B", "ray"),
34 - (2, 2, 0, 1, 0, "meta-llama/Meta-Llama-3-8B", "ray"),
35 - # NOTE: InternVL2 multi-node tests are flaky,
36 - # use mp backend to skip the multi-node tests
37 - (1, 2, 1, 1, 1, "OpenGVLab/InternVL2-1B", "mp"),
38 - (1, 2, 1, 1, 1, "OpenGVLab/InternVL2-2B", "mp"),
39 - (1, 2, 1, 0, 1, "OpenGVLab/InternVL2-4B", "mp"),
40 - (1, 2, 0, 1, 0, "Qwen/Qwen2-VL-2B-Instruct", "mp"),
41 - # TP only models
42 - (2, 1, 1, 0, 0, "adept/fuyu-8b", "mp"),
216 + params for model_name, settings in MODEL_SETTINGS.items()
217 + for params in settings.iter_params(model_name)
218 + if model_name in TEST_MODELS
219 ],
220 )
221 @fork_new_process_for_each_test
46 -def test_compare_tp(TP_SIZE, PP_SIZE, EAGER_MODE, CHUNKED_PREFILL,
47 - TRUST_REMOTE_CODE, MODEL_NAME, DIST_BACKEND):
48 - if VLLM_MULTI_NODE and DIST_BACKEND == "mp":
222 +def test_compare_tp(model_name: str, parallel_setup: ParallelSetup,
223 + distributed_backend: str, trust_remote_code: bool,
224 + tokenizer_mode: Optional[str], num_gpus_available):
225 + tp_size, pp_size, eager_mode, chunked_prefill = parallel_setup
226 +
227 + if num_gpus_available < tp_size:
228 + pytest.skip(f"Need at least {tp_size} GPUs to run the test")
229 + if VLLM_MULTI_NODE and distributed_backend == "mp":
230 pytest.skip("Skipping multi-node pipeline parallel test for "
231 "multiprocessing distributed backend")
232
52 - pp_args = [
233 + common_args = [
234 # use half precision for speed and memory savings in CI environment
235 "--dtype",
236 "float16",
237 "--max-model-len",
57 - "8192",
238 + "2048",
239 + "--max-num-seqs",
240 + "8",
241 + ]
242 + if chunked_prefill:
243 + common_args.append("--enable-chunked-prefill")
244 + if eager_mode:
245 + common_args.append("--enforce-eager")
246 + if trust_remote_code:
247 + common_args.append("--trust-remote-code")
248 + if tokenizer_mode:
249 + common_args.extend(["--tokenizer-mode", tokenizer_mode])
250 +
251 + if (distributed_backend == "ray" and tp_size == 2 and pp_size == 2
252 + and chunked_prefill):
253 + # Test Ray ADAG for a subset of the tests
254 + pp_env = {
255 + "VLLM_USE_RAY_COMPILED_DAG": "1",
256 + "VLLM_USE_RAY_SPMD_WORKER": "1",
257 + "VLLM_USE_RAY_COMPILED_DAG_NCCL_CHANNEL": "1",
258 + }
259 + # Temporary. Currently when zeromq + SPMD is used, it does not properly
260 + # terminate because of aDAG issue.
261 + common_args.append("--disable-frontend-multiprocessing")
262 + else:
263 + pp_env = None
264 +
265 + pp_args = [
266 + *common_args,
267 "--pipeline-parallel-size",
59 - str(PP_SIZE),
268 + str(pp_size),
269 "--tensor-parallel-size",
61 - str(TP_SIZE),
270 + str(tp_size),
271 "--distributed-executor-backend",
63 - DIST_BACKEND,
272 + distributed_backend,
273 ]
274
275 # compare without pipeline parallelism
@@ -69,41 +278,15 @@ def test_compare_tp(TP_SIZE, PP_SIZE, EAGER_MODE, CHUNKED_PREFILL,
278 # schedule all workers in a node other than the head node,
279 # which can cause the test to fail.
280 tp_args = [
72 - # use half precision for speed and memory savings in CI environment
73 - "--dtype",
74 - "float16",
75 - "--max-model-len",
76 - "8192",
281 + *common_args,
282 "--tensor-parallel-size",
78 - str(max(TP_SIZE, 2)), # We only use 2 GPUs in the CI.
283 + str(tp_size),
284 "--distributed-executor-backend",
285 "mp",
286 ]
82 - if CHUNKED_PREFILL:
83 - pp_args.append("--enable-chunked-prefill")
84 - tp_args.append("--enable-chunked-prefill")
85 - if EAGER_MODE:
86 - pp_args.append("--enforce-eager")
87 - tp_args.append("--enforce-eager")
88 - if TRUST_REMOTE_CODE:
89 - pp_args.append("--trust-remote-code")
90 - tp_args.append("--trust-remote-code")
91 - pp_env = None
92 - if (DIST_BACKEND == "ray" and TP_SIZE == 2 and PP_SIZE == 2
93 - and CHUNKED_PREFILL):
94 - # Test Ray ADAG for a subset of the tests
95 - pp_env = {
96 - "VLLM_USE_RAY_COMPILED_DAG": "1",
97 - "VLLM_USE_RAY_SPMD_WORKER": "1",
98 - "VLLM_USE_RAY_COMPILED_DAG_NCCL_CHANNEL": "1",
99 - }
100 - # Temporary. Currently when zeromq + SPMD is used, it does not properly
101 - # terminate because of aDAG issue.
102 - pp_args.append("--disable-frontend-multiprocessing")
103 - tp_args.append("--disable-frontend-multiprocessing")
287
288 try:
106 - compare_two_settings(MODEL_NAME, pp_args, tp_args, pp_env)
289 + compare_two_settings(model_name, pp_args, tp_args, pp_env)
290 except Exception:
291 if pp_env is None:
292 raise
tests/models/test_registry.py
+49 -3
@@ -1,9 +1,55 @@
1 +import warnings
2 +
3 import pytest
4 +import torch.cuda
5
6 from vllm.model_executor.models import _MODELS, ModelRegistry
7 +from vllm.platforms import current_platform
8 +
9 +from ..utils import fork_new_process_for_each_test
10
11
6 -@pytest.mark.parametrize("model_cls", _MODELS)
7 -def test_registry_imports(model_cls):
12 +@pytest.mark.parametrize("model_arch", _MODELS)
13 +def test_registry_imports(model_arch):
14 # Ensure all model classes can be imported successfully
9 - ModelRegistry.resolve_model_cls([model_cls])
15 + ModelRegistry.resolve_model_cls(model_arch)
16 +
17 +
18 +@fork_new_process_for_each_test
19 +@pytest.mark.parametrize("model_arch,is_mm,init_cuda", [
20 + ("LlamaForCausalLM", False, False),
21 + ("MllamaForConditionalGeneration", True, False),
22 + ("LlavaForConditionalGeneration", True, True),
23 +])
24 +def test_registry_is_multimodal(model_arch, is_mm, init_cuda):
25 + assert ModelRegistry.is_multimodal_model(model_arch) is is_mm
26 +
27 + if init_cuda and current_platform.is_cuda_alike():
28 + assert not torch.cuda.is_initialized()
29 +
30 + ModelRegistry.resolve_model_cls(model_arch)
31 + if not torch.cuda.is_initialized():
32 + warnings.warn(
33 + "This model no longer initializes CUDA on import. "
34 + "Please test using a different one.",
35 + stacklevel=2)
36 +
37 +
38 +@fork_new_process_for_each_test
39 +@pytest.mark.parametrize("model_arch,is_pp,init_cuda", [
40 + ("MLPSpeculatorPreTrainedModel", False, False),
41 + ("DeepseekV2ForCausalLM", True, False),
42 + ("Qwen2VLForConditionalGeneration", True, True),
43 +])
44 +def test_registry_is_pp(model_arch, is_pp, init_cuda):
45 + assert ModelRegistry.is_pp_supported_model(model_arch) is is_pp
46 +
47 + if init_cuda and current_platform.is_cuda_alike():
48 + assert not torch.cuda.is_initialized()
49 +
50 + ModelRegistry.resolve_model_cls(model_arch)
51 + if not torch.cuda.is_initialized():
52 + warnings.warn(
53 + "This model no longer initializes CUDA on import. "
54 + "Please test using a different one.",
55 + stacklevel=2)
tests/utils.py
+19 -8
@@ -14,7 +14,6 @@ import openai
14 import pytest
15 import requests
16 from openai.types.completion import Completion
17 -from transformers import AutoTokenizer
17 from typing_extensions import ParamSpec
18
19 from tests.models.utils import TextTextLogprobs
@@ -24,6 +23,7 @@ from vllm.engine.arg_utils import AsyncEngineArgs
23 from vllm.entrypoints.openai.cli_args import make_arg_parser
24 from vllm.model_executor.model_loader.loader import get_model_loader
25 from vllm.platforms import current_platform
26 +from vllm.transformers_utils.tokenizer import get_tokenizer
27 from vllm.utils import (FlexibleArgumentParser, GB_bytes,
28 cuda_device_count_stateless, get_open_port, is_hip)
29
@@ -181,15 +181,26 @@ def compare_two_settings(model: str,
181 env2: The second set of environment variables to pass to the API server.
182 """
183
184 - trust_remote_code = "--trust-remote-code"
185 - if trust_remote_code in arg1 or trust_remote_code in arg2:
186 - tokenizer = AutoTokenizer.from_pretrained(model,
187 - trust_remote_code=True)
188 - else:
189 - tokenizer = AutoTokenizer.from_pretrained(model)
184 + trust_remote_code = False
185 + for args in (arg1, arg2):
186 + if "--trust-remote-code" in args:
187 + trust_remote_code = True
188 + break
189 +
190 + tokenizer_mode = "auto"
191 + for args in (arg1, arg2):
192 + if "--tokenizer-mode" in args:
193 + tokenizer_mode = args[args.index("--tokenizer-mode") + 1]
194 + break
195 +
196 + tokenizer = get_tokenizer(
197 + model,
198 + trust_remote_code=trust_remote_code,
199 + tokenizer_mode=tokenizer_mode,
200 + )
201
202 prompt = "Hello, my name is"
192 - token_ids = tokenizer(prompt)["input_ids"]
203 + token_ids = tokenizer(prompt).input_ids
204 results = []
205 for args, env in ((arg1, env1), (arg2, env2)):
206 with RemoteOpenAIServer(model,
vllm/config.py
+19 -43
@@ -31,28 +31,7 @@ if TYPE_CHECKING:
31 logger = init_logger(__name__)
32
33 _EMBEDDING_MODEL_MAX_NUM_BATCHED_TOKENS = 32768
34 -_MULTIMODAL_MODEL_MAX_NUM_BATCHED_TOKENS = 4096
35 -
36 -_PP_SUPPORTED_MODELS = [
37 - "AquilaForCausalLM",
38 - "AquilaModel",
39 - "DeepseekV2ForCausalLM",
40 - "GPT2LMHeadModel",
41 - "InternLM2ForCausalLM",
42 - "InternLMForCausalLM",
43 - "InternVLChatModel",
44 - "JAISLMHeadModel",
45 - "LlamaForCausalLM",
46 - "LLaMAForCausalLM",
47 - "MistralForCausalLM",
48 - "MixtralForCausalLM",
49 - "NemotronForCausalLM",
50 - "Phi3ForCausalLM",
51 - "Qwen2ForCausalLM",
52 - "Qwen2MoeForCausalLM",
53 - "QWenLMHeadModel",
54 - "Qwen2VLForConditionalGeneration",
55 -]
34 +_MULTIMODAL_MODEL_MAX_NUM_BATCHED_TOKENS = 5120
35
36
37 class ModelConfig:
@@ -228,16 +207,14 @@ class ModelConfig:
207 self, limit_mm_per_prompt: Optional[Mapping[str, int]]
208 ) -> Optional["MultiModalConfig"]:
209 architectures = getattr(self.hf_config, "architectures", [])
231 - if any(
232 - ModelRegistry.is_multimodal_model(arch)
233 - for arch in architectures):
210 + if ModelRegistry.is_multimodal_model(architectures):
211 return MultiModalConfig(limit_per_prompt=limit_mm_per_prompt or {})
235 - else:
236 - if limit_mm_per_prompt:
237 - raise ValueError(
238 - "limit_mm_per_prompt is only supported for multimodal "
239 - "models.")
240 - return None
212 +
213 + if limit_mm_per_prompt:
214 + raise ValueError("`limit_mm_per_prompt` is only supported for "
215 + "multimodal models.")
216 +
217 + return None
218
219 def _verify_tokenizer_mode(self) -> None:
220 tokenizer_mode = self.tokenizer_mode.lower()
@@ -249,8 +226,7 @@ class ModelConfig:
226
227 def _verify_embedding_mode(self) -> None:
228 architectures = getattr(self.hf_config, "architectures", [])
252 - self.embedding_mode = any(
253 - ModelRegistry.is_embedding_model(arch) for arch in architectures)
229 + self.embedding_mode = ModelRegistry.is_embedding_model(architectures)
230
231 def _parse_quant_hf_config(self):
232 quant_cfg = getattr(self.hf_config, "quantization_config", None)
@@ -417,17 +393,17 @@ class ModelConfig:
393 f"({tensor_parallel_size}).")
394
395 pipeline_parallel_size = parallel_config.pipeline_parallel_size
420 - architectures = getattr(self.hf_config, "architectures", [])
421 - if not all(arch in _PP_SUPPORTED_MODELS
422 - for arch in architectures) and pipeline_parallel_size > 1:
423 - raise NotImplementedError(
424 - "Pipeline parallelism is only supported for the following "
425 - f" architectures: {_PP_SUPPORTED_MODELS}.")
396 + if pipeline_parallel_size > 1:
397 + architectures = getattr(self.hf_config, "architectures", [])
398 + if not ModelRegistry.is_pp_supported_model(architectures):
399 + raise NotImplementedError(
400 + "Pipeline parallelism is not supported for this model. "
401 + "Supported models implement the `SupportsPP` interface.")
402
427 - if pipeline_parallel_size > 1 and self.use_async_output_proc:
428 - logger.warning("Async output processor is not supported with "
429 - "pipeline parallelism currently. Disabling it.")
430 - self.use_async_output_proc = False
403 + if self.use_async_output_proc:
404 + logger.warning("Async output processor is not supported with "
405 + "pipeline parallelism currently. Disabling it.")
406 + self.use_async_output_proc = False
407
408 def get_hf_config_sliding_window(self) -> Optional[int]:
409 """Get the sliding window size, or None if disabled."""
vllm/model_executor/models/__init__.py
+125 -22
@@ -1,12 +1,18 @@
1 -import functools
1 import importlib
3 -from typing import Dict, List, Optional, Tuple, Type
2 +import string
3 +import subprocess
4 +import sys
5 +import uuid
6 +from functools import lru_cache, partial
7 +from typing import Callable, Dict, List, Optional, Tuple, Type, Union
8
9 import torch.nn as nn
10
11 from vllm.logger import init_logger
12 from vllm.utils import is_hip
13
14 +from .interfaces import supports_multimodal, supports_pp
15 +
16 logger = init_logger(__name__)
17
18 _GENERATION_MODELS = {
@@ -152,19 +158,25 @@ _ROCM_PARTIALLY_SUPPORTED_MODELS: Dict[str, str] = {
158 class ModelRegistry:
159
160 @staticmethod
155 - @functools.lru_cache(maxsize=128)
156 - def _get_model(model_arch: str):
157 - module_name, model_cls_name = _MODELS[model_arch]
158 - module = importlib.import_module(
159 - f"vllm.model_executor.models.{module_name}")
160 - return getattr(module, model_cls_name, None)
161 + def _get_module_cls_name(model_arch: str) -> Tuple[str, str]:
162 + module_relname, cls_name = _MODELS[model_arch]
163 + return f"vllm.model_executor.models.{module_relname}", cls_name
164
165 @staticmethod
163 - def _try_load_model_cls(model_arch: str) -> Optional[Type[nn.Module]]:
164 - if model_arch in _OOT_MODELS:
165 - return _OOT_MODELS[model_arch]
166 + @lru_cache(maxsize=128)
167 + def _try_get_model_stateful(model_arch: str) -> Optional[Type[nn.Module]]:
168 if model_arch not in _MODELS:
169 return None
170 +
171 + module_name, cls_name = ModelRegistry._get_module_cls_name(model_arch)
172 + module = importlib.import_module(module_name)
173 + return getattr(module, cls_name, None)
174 +
175 + @staticmethod
176 + def _try_get_model_stateless(model_arch: str) -> Optional[Type[nn.Module]]:
177 + if model_arch in _OOT_MODELS:
178 + return _OOT_MODELS[model_arch]
179 +
180 if is_hip():
181 if model_arch in _ROCM_UNSUPPORTED_MODELS:
182 raise ValueError(
@@ -175,11 +187,24 @@ class ModelRegistry:
187 "Model architecture %s is partially supported by ROCm: %s",
188 model_arch, _ROCM_PARTIALLY_SUPPORTED_MODELS[model_arch])
189
178 - return ModelRegistry._get_model(model_arch)
190 + return None
191 +
192 + @staticmethod
193 + def _try_load_model_cls(model_arch: str) -> Optional[Type[nn.Module]]:
194 + model = ModelRegistry._try_get_model_stateless(model_arch)
195 + if model is not None:
196 + return model
197 +
198 + return ModelRegistry._try_get_model_stateful(model_arch)
199
200 @staticmethod
201 def resolve_model_cls(
182 - architectures: List[str]) -> Tuple[Type[nn.Module], str]:
202 + architectures: Union[str, List[str]], ) -> Tuple[Type[nn.Module], str]:
203 + if isinstance(architectures, str):
204 + architectures = [architectures]
205 + if not architectures:
206 + logger.warning("No model architectures are specified")
207 +
208 for arch in architectures:
209 model_cls = ModelRegistry._try_load_model_cls(arch)
210 if model_cls is not None:
@@ -200,21 +225,99 @@ class ModelRegistry:
225 "Model architecture %s is already registered, and will be "
226 "overwritten by the new model class %s.", model_arch,
227 model_cls.__name__)
203 - global _OOT_MODELS
228 +
229 _OOT_MODELS[model_arch] = model_cls
230
231 @staticmethod
207 - def is_embedding_model(model_arch: str) -> bool:
208 - return model_arch in _EMBEDDING_MODELS
232 + @lru_cache(maxsize=128)
233 + def _check_stateless(
234 + func: Callable[[Type[nn.Module]], bool],
235 + model_arch: str,
236 + *,
237 + default: Optional[bool] = None,
238 + ) -> bool:
239 + """
240 + Run a boolean function against a model and return the result.
241 +
242 + If the model is not found, returns the provided default value.
243 +
244 + If the model is not already imported, the function is run inside a
245 + subprocess to avoid initializing CUDA for the main program.
246 + """
247 + model = ModelRegistry._try_get_model_stateless(model_arch)
248 + if model is not None:
249 + return func(model)
250 +
251 + if model_arch not in _MODELS and default is not None:
252 + return default
253 +
254 + module_name, cls_name = ModelRegistry._get_module_cls_name(model_arch)
255 +
256 + valid_name_characters = string.ascii_letters + string.digits + "._"
257 + if any(s not in valid_name_characters for s in module_name):
258 + raise ValueError(f"Unsafe module name detected for {model_arch}")
259 + if any(s not in valid_name_characters for s in cls_name):
260 + raise ValueError(f"Unsafe class name detected for {model_arch}")
261 + if any(s not in valid_name_characters for s in func.__module__):
262 + raise ValueError(f"Unsafe module name detected for {func}")
263 + if any(s not in valid_name_characters for s in func.__name__):
264 + raise ValueError(f"Unsafe class name detected for {func}")
265 +
266 + err_id = uuid.uuid4()
267 +
268 + stmts = ";".join([
269 + f"from {module_name} import {cls_name}",
270 + f"from {func.__module__} import {func.__name__}",
271 + f"assert {func.__name__}({cls_name}), '{err_id}'",
272 + ])
273 +
274 + result = subprocess.run([sys.executable, "-c", stmts],
275 + capture_output=True)
276 +
277 + if result.returncode != 0:
278 + err_lines = [line.decode() for line in result.stderr.splitlines()]
279 + if err_lines and err_lines[-1] != f"AssertionError: {err_id}":
280 + err_str = "\n".join(err_lines)
281 + raise RuntimeError(
282 + "An unexpected error occurred while importing the model in "
283 + f"another process. Error log:\n{err_str}")
284 +
285 + return result.returncode == 0
286 +
287 + @staticmethod
288 + def is_embedding_model(architectures: Union[str, List[str]]) -> bool:
289 + if isinstance(architectures, str):
290 + architectures = [architectures]
291 + if not architectures:
292 + logger.warning("No model architectures are specified")
293 +
294 + return any(arch in _EMBEDDING_MODELS for arch in architectures)
295 +
296 + @staticmethod
297 + def is_multimodal_model(architectures: Union[str, List[str]]) -> bool:
298 + if isinstance(architectures, str):
299 + architectures = [architectures]
300 + if not architectures:
301 + logger.warning("No model architectures are specified")
302 +
303 + is_mm = partial(ModelRegistry._check_stateless,
304 + supports_multimodal,
305 + default=False)
306 +
307 + return any(is_mm(arch) for arch in architectures)
308
309 @staticmethod
211 - def is_multimodal_model(model_arch: str) -> bool:
310 + def is_pp_supported_model(architectures: Union[str, List[str]]) -> bool:
311 + if isinstance(architectures, str):
312 + architectures = [architectures]
313 + if not architectures:
314 + logger.warning("No model architectures are specified")
315 +
316 + is_pp = partial(ModelRegistry._check_stateless,
317 + supports_pp,
318 + default=False)
319
213 - # TODO: find a way to avoid initializing CUDA prematurely to
214 - # use `supports_multimodal` to determine if a model is multimodal
215 - # model_cls = ModelRegistry._try_load_model_cls(model_arch)
216 - # from vllm.model_executor.models.interfaces import supports_multimodal
217 - return model_arch in _MULTIMODAL_MODELS
320 + return any(is_pp(arch) for arch in architectures)
321
322
323 __all__ = [
vllm/model_executor/models/arctic.py
+41 -18
@@ -1,12 +1,12 @@
1 """Inference-only Snowflake Arctic model."""
2 -from typing import Iterable, List, Optional, Tuple
2 +from typing import Iterable, List, Optional, Tuple, Union
3
4 import torch
5 from torch import nn
6
7 from vllm.attention import Attention, AttentionMetadata
8 from vllm.config import CacheConfig
9 -from vllm.distributed import (get_tensor_model_parallel_rank,
9 +from vllm.distributed import (get_pp_group, get_tensor_model_parallel_rank,
10 get_tensor_model_parallel_world_size,
11 tensor_model_parallel_all_reduce)
12 from vllm.logger import init_logger
@@ -18,8 +18,7 @@ from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
18 ReplicatedLinear,
19 RowParallelLinear)
20 from vllm.model_executor.layers.logits_processor import LogitsProcessor
21 -from vllm.model_executor.layers.quantization.base_config import (
22 - QuantizationConfig)
21 +from vllm.model_executor.layers.quantization import QuantizationConfig
22 from vllm.model_executor.layers.quantization.deepspeedfp import (
23 DeepSpeedFPConfig, DeepSpeedFPParameter)
24 from vllm.model_executor.layers.rotary_embedding import get_rope
@@ -32,6 +31,10 @@ from vllm.model_executor.utils import set_weight_attrs
31 from vllm.sequence import IntermediateTensors
32 from vllm.transformers_utils.configs.arctic import ArcticConfig
33
34 +from .interfaces import SupportsPP
35 +from .utils import (is_pp_missing_parameter,
36 + make_empty_intermediate_tensors_factory, make_layers)
37 +
38 logger = init_logger(__name__)
39
40
@@ -364,6 +367,7 @@ class ArcticModel(nn.Module):
367 config: ArcticConfig,
368 cache_config: Optional[CacheConfig] = None,
369 quant_config: Optional[QuantizationConfig] = None,
370 + prefix: str = "",
371 ) -> None:
372 super().__init__()
373 self.padding_idx = config.pad_token_id
@@ -372,15 +376,16 @@ class ArcticModel(nn.Module):
376 self.vocab_size,
377 config.hidden_size,
378 org_num_embeddings=self.vocab_size)
375 - self.layers = nn.ModuleList([
376 - ArcticDecoderLayer(config,
377 - layer_idx,
378 - cache_config,
379 - quant_config=quant_config)
380 - for layer_idx in range(config.num_hidden_layers)
381 - ])
379 + self.start_layer, self.end_layer, self.layers = make_layers(
380 + config.num_hidden_layers,
381 + lambda prefix: ArcticDecoderLayer(config, int(
382 + prefix.split(".")[-1]), cache_config, quant_config),
383 + prefix=f"{prefix}.layers")
384 self._attn_implementation = config._attn_implementation
385 self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
386 + self.make_empty_intermediate_tensors = (
387 + make_empty_intermediate_tensors_factory(["hidden_states"],
388 + config.hidden_size))
389
390 def forward(
391 self,
@@ -388,17 +393,25 @@ class ArcticModel(nn.Module):
393 positions: torch.Tensor,
394 kv_caches: List[torch.Tensor],
395 attn_metadata: AttentionMetadata,
391 - ) -> torch.Tensor:
392 - hidden_states = self.embed_tokens(input_ids)
393 - for i in range(len(self.layers)):
396 + intermediate_tensors: Optional[IntermediateTensors],
397 + ) -> Union[torch.Tensor, IntermediateTensors]:
398 + if get_pp_group().is_first_rank:
399 + hidden_states = self.embed_tokens(input_ids)
400 + else:
401 + assert intermediate_tensors is not None
402 + hidden_states = intermediate_tensors["hidden_states"]
403 + for i in range(self.start_layer, self.end_layer):
404 layer = self.layers[i]
395 - hidden_states = layer(positions, hidden_states, kv_caches[i],
405 + hidden_states = layer(positions, hidden_states,
406 + kv_caches[i - self.start_layer],
407 attn_metadata)
408 + if not get_pp_group().is_last_rank:
409 + return IntermediateTensors({"hidden_states": hidden_states})
410 hidden_states = self.norm(hidden_states)
411 return hidden_states
412
413
401 -class ArcticForCausalLM(nn.Module):
414 +class ArcticForCausalLM(nn.Module, SupportsPP):
415
416 def __init__(self,
417 config: ArcticConfig,
@@ -422,6 +435,8 @@ class ArcticForCausalLM(nn.Module):
435 self.logits_processor = LogitsProcessor(self.unpadded_vocab_size,
436 config.vocab_size)
437 self.sampler = Sampler()
438 + self.make_empty_intermediate_tensors = (
439 + self.model.make_empty_intermediate_tensors)
440
441 def forward(
442 self,
@@ -430,9 +445,9 @@ class ArcticForCausalLM(nn.Module):
445 kv_caches: List[torch.Tensor],
446 attn_metadata: AttentionMetadata,
447 intermediate_tensors: Optional[IntermediateTensors] = None,
433 - ) -> torch.Tensor:
448 + ) -> Union[torch.Tensor, IntermediateTensors]:
449 hidden_states = self.model(input_ids, positions, kv_caches,
435 - attn_metadata)
450 + attn_metadata, intermediate_tensors)
451 return hidden_states
452
453 def compute_logits(
@@ -503,6 +518,8 @@ class ArcticForCausalLM(nn.Module):
518 # Skip loading extra bias for GPTQ models.
519 if name.endswith(".bias") and name not in params_dict:
520 continue
521 + if is_pp_missing_parameter(name, self):
522 + continue
523 param = params_dict[name]
524 weight_loader = param.weight_loader
525 weight_loader(param, loaded_weight, shard_id)
@@ -512,6 +529,8 @@ class ArcticForCausalLM(nn.Module):
529 if weight_name not in name:
530 continue
531 name = name.replace(weight_name, param_name)
532 + if is_pp_missing_parameter(name, self):
533 + continue
534 param = params_dict[name]
535 weight_loader = param.weight_loader
536 weight_loader(param, loaded_weight, shard_id)
@@ -522,6 +541,8 @@ class ArcticForCausalLM(nn.Module):
541 if weight_name not in name:
542 continue
543 name = name.replace(weight_name, param_name)
544 + if is_pp_missing_parameter(name, self):
545 + continue
546 param = params_dict[name]
547 weight_loader = param.weight_loader
548 weight_loader(param,
@@ -532,6 +553,8 @@ class ArcticForCausalLM(nn.Module):
553 else:
554 if name.endswith(".bias") and name not in params_dict:
555 continue
556 + if is_pp_missing_parameter(name, self):
557 + continue
558 param = params_dict[name]
559
560 weight_loader = getattr(param, "weight_loader",
vllm/model_executor/models/baichuan.py
+44 -21
@@ -19,7 +19,7 @@
19 # limitations under the License.
20 """Inference-only BaiChuan model compatible with HuggingFace weights."""
21 import math
22 -from typing import Iterable, List, Optional, Tuple
22 +from typing import Iterable, List, Optional, Tuple, Union
23
24 import torch
25 from torch import nn
@@ -27,7 +27,7 @@ from transformers import PretrainedConfig
27
28 from vllm.attention import Attention, AttentionMetadata
29 from vllm.config import CacheConfig, LoRAConfig
30 -from vllm.distributed import (get_tensor_model_parallel_rank,
30 +from vllm.distributed import (get_pp_group, get_tensor_model_parallel_rank,
31 get_tensor_model_parallel_world_size)
32 from vllm.model_executor.layers.activation import SiluAndMul
33 from vllm.model_executor.layers.layernorm import RMSNorm
@@ -35,8 +35,7 @@ from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
35 QKVParallelLinear,
36 RowParallelLinear)
37 from vllm.model_executor.layers.logits_processor import LogitsProcessor
38 -from vllm.model_executor.layers.quantization.base_config import (
39 - QuantizationConfig)
38 +from vllm.model_executor.layers.quantization import QuantizationConfig
39 from vllm.model_executor.layers.rotary_embedding import get_rope
40 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
41 from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -45,7 +44,9 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
44 from vllm.model_executor.sampling_metadata import SamplingMetadata
45 from vllm.sequence import IntermediateTensors
46
48 -from .interfaces import SupportsLoRA
47 +from .interfaces import SupportsLoRA, SupportsPP
48 +from .utils import (is_pp_missing_parameter,
49 + make_empty_intermediate_tensors_factory, make_layers)
50
51
52 def _get_alibi_slopes(total_num_heads: int) -> torch.Tensor:
@@ -255,7 +256,8 @@ class BaiChuanModel(nn.Module):
256 config: PretrainedConfig,
257 position_embedding: str,
258 cache_config: Optional[CacheConfig] = None,
258 - quant_config: Optional[QuantizationConfig] = None):
259 + quant_config: Optional[QuantizationConfig] = None,
260 + prefix: str = ""):
261 super().__init__()
262 self.config = config
263 self.padding_idx = config.pad_token_id
@@ -265,12 +267,16 @@ class BaiChuanModel(nn.Module):
267 config.vocab_size,
268 config.hidden_size,
269 )
268 - self.layers = nn.ModuleList([
269 - BaiChuanDecoderLayer(config, position_embedding, cache_config,
270 - quant_config)
271 - for _ in range(config.num_hidden_layers)
272 - ])
270 + self.start_layer, self.end_layer, self.layers = make_layers(
271 + config.num_hidden_layers,
272 + lambda prefix: BaiChuanDecoderLayer(config, position_embedding,
273 + cache_config, quant_config),
274 + prefix=f"{prefix}.layers",
275 + )
276 self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
277 + self.make_empty_intermediate_tensors = (
278 + make_empty_intermediate_tensors_factory(
279 + ["hidden_states", "residual"], config.hidden_size))
280
281 def forward(
282 self,
@@ -278,23 +284,34 @@ class BaiChuanModel(nn.Module):
284 positions: torch.Tensor,
285 kv_caches: List[torch.Tensor],
286 attn_metadata: AttentionMetadata,
281 - ) -> torch.Tensor:
282 - hidden_states = self.embed_tokens(input_ids)
283 - residual = None
284 - for i in range(len(self.layers)):
287 + intermediate_tensors: Optional[IntermediateTensors],
288 + ) -> Union[torch.Tensor, IntermediateTensors]:
289 + if get_pp_group().is_first_rank:
290 + hidden_states = self.embed_tokens(input_ids)
291 + residual = None
292 + else:
293 + assert intermediate_tensors is not None
294 + hidden_states = intermediate_tensors["hidden_states"]
295 + residual = intermediate_tensors["residual"]
296 + for i in range(self.start_layer, self.end_layer):
297 layer = self.layers[i]
298 hidden_states, residual = layer(
299 positions,
300 hidden_states,
289 - kv_caches[i],
301 + kv_caches[i - self.start_layer],
302 attn_metadata,
303 residual,
304 )
305 + if not get_pp_group().is_last_rank:
306 + return IntermediateTensors({
307 + "hidden_states": hidden_states,
308 + "residual": residual,
309 + })
310 hidden_states, _ = self.norm(hidden_states, residual)
311 return hidden_states
312
313
297 -class BaiChuanBaseForCausalLM(nn.Module, SupportsLoRA):
314 +class BaiChuanBaseForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
315 packed_modules_mapping = {
316 "W_pack": ["W_pack"],
317 "gate_up_proj": [
@@ -335,6 +352,8 @@ class BaiChuanBaseForCausalLM(nn.Module, SupportsLoRA):
352 self.lm_head.weight = self.model.embed_tokens.weight
353 self.logits_processor = LogitsProcessor(config.vocab_size)
354 self.sampler = Sampler()
355 + self.make_empty_intermediate_tensors = (
356 + self.model.make_empty_intermediate_tensors)
357
358 def forward(
359 self,
@@ -343,9 +362,9 @@ class BaiChuanBaseForCausalLM(nn.Module, SupportsLoRA):
362 kv_caches: List[torch.Tensor],
363 attn_metadata: AttentionMetadata,
364 intermediate_tensors: Optional[IntermediateTensors] = None,
346 - ) -> torch.Tensor:
365 + ) -> Union[torch.Tensor, IntermediateTensors]:
366 hidden_states = self.model(input_ids, positions, kv_caches,
348 - attn_metadata)
367 + attn_metadata, intermediate_tensors)
368 return hidden_states
369
370 def compute_logits(
@@ -394,6 +413,8 @@ class BaiChuanBaseForCausalLM(nn.Module, SupportsLoRA):
413 # Skip loading extra bias for GPTQ models.
414 if name.endswith(".bias") and name not in params_dict:
415 continue
416 + if is_pp_missing_parameter(name, self):
417 + continue
418 param = params_dict[name]
419 weight_loader = param.weight_loader
420 weight_loader(param, loaded_weight, shard_id)
@@ -402,6 +423,8 @@ class BaiChuanBaseForCausalLM(nn.Module, SupportsLoRA):
423 # Skip loading extra bias for GPTQ models.
424 if name.endswith(".bias") and name not in params_dict:
425 continue
426 + if is_pp_missing_parameter(name, self):
427 + continue
428 param = params_dict[name]
429 weight_loader = getattr(param, "weight_loader",
430 default_weight_loader)
@@ -413,7 +436,7 @@ class BaichuanForCausalLM(BaiChuanBaseForCausalLM):
436
437 def __init__(
438 self,
416 - config,
439 + config: PretrainedConfig,
440 cache_config: Optional[CacheConfig] = None,
441 quant_config: Optional[QuantizationConfig] = None,
442 lora_config: Optional[LoRAConfig] = None,
@@ -431,7 +454,7 @@ class BaiChuanForCausalLM(BaiChuanBaseForCausalLM):
454
455 def __init__(
456 self,
434 - config,
457 + config: PretrainedConfig,
458 cache_config: Optional[CacheConfig] = None,
459 quant_config: Optional[QuantizationConfig] = None,
460 lora_config: Optional[LoRAConfig] = None,
vllm/model_executor/models/blip2.py
+39 -22
@@ -1,3 +1,4 @@
1 +from functools import cached_property
2 from typing import (Iterable, List, Literal, Mapping, Optional, Tuple,
3 TypedDict, Union)
4
@@ -11,7 +12,7 @@ from vllm.config import CacheConfig, MultiModalConfig
12 from vllm.inputs import INPUT_REGISTRY, InputContext, LLMInputs
13 from vllm.model_executor.layers.activation import get_act_fn
14 from vllm.model_executor.layers.quantization import QuantizationConfig
14 -from vllm.model_executor.layers.sampler import SamplerOutput
15 +from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
16 from vllm.model_executor.model_loader.weight_utils import default_weight_loader
17 from vllm.model_executor.sampling_metadata import SamplingMetadata
18 from vllm.multimodal import MULTIMODAL_REGISTRY
@@ -19,7 +20,7 @@ from vllm.sequence import IntermediateTensors, SequenceData
20
21 from .blip import (BlipVisionModel, dummy_image_for_blip,
22 get_max_blip_image_tokens)
22 -from .interfaces import SupportsMultiModal
23 +from .interfaces import SupportsMultiModal, SupportsPP
24 from .utils import (group_weights_with_prefix, init_vllm_registered_model,
25 merge_multimodal_embeddings)
26
@@ -475,7 +476,7 @@ def input_processor_for_blip2(ctx: InputContext, llm_inputs: LLMInputs):
476 @MULTIMODAL_REGISTRY.register_max_image_tokens(get_max_blip2_image_tokens)
477 @INPUT_REGISTRY.register_dummy_data(dummy_data_for_blip2)
478 @INPUT_REGISTRY.register_input_processor(input_processor_for_blip2)
478 -class Blip2ForConditionalGeneration(nn.Module, SupportsMultiModal):
479 +class Blip2ForConditionalGeneration(nn.Module, SupportsMultiModal, SupportsPP):
480
481 def __init__(self,
482 config: Blip2Config,
@@ -508,6 +509,16 @@ class Blip2ForConditionalGeneration(nn.Module, SupportsMultiModal):
509 self.language_model = init_vllm_registered_model(
510 config.text_config, cache_config, quant_config)
511
512 + self.make_empty_intermediate_tensors = (
513 + self.language_model.make_empty_intermediate_tensors)
514 +
515 + @cached_property
516 + def sampler(self):
517 + if hasattr(self.language_model, "sampler"):
518 + return self.language_model.sampler
519 +
520 + return Sampler()
521 +
522 def _validate_pixel_values(self, data: torch.Tensor) -> torch.Tensor:
523 h = w = self.config.vision_config.image_size
524 expected_dims = (3, h, w)
@@ -600,7 +611,7 @@ class Blip2ForConditionalGeneration(nn.Module, SupportsMultiModal):
611 attn_metadata: AttentionMetadata,
612 intermediate_tensors: Optional[IntermediateTensors] = None,
613 **kwargs: object,
603 - ) -> SamplerOutput:
614 + ) -> Union[SamplerOutput, IntermediateTensors]:
615 """Run forward pass for BLIP-2.
616
617 One key thing to understand is the `input_ids` already accounts for the
@@ -631,26 +642,32 @@ class Blip2ForConditionalGeneration(nn.Module, SupportsMultiModal):
642 See also:
643 :class:`Blip2ImageInputs`
644 """
634 - image_input = self._parse_and_validate_image_input(**kwargs)
635 -
636 - if image_input is not None:
637 - vision_embeddings = self._process_image_input(image_input)
638 - inputs_embeds = self.language_model.model.get_input_embeddings(
639 - input_ids)
640 -
641 - inputs_embeds = merge_multimodal_embeddings(
642 - input_ids, inputs_embeds, vision_embeddings,
643 - BLIP2_IMAGE_TOKEN_ID)
644 -
645 + if intermediate_tensors is not None:
646 input_ids = None
646 - else:
647 inputs_embeds = None
648 -
649 - hidden_states = self.language_model.model(input_ids,
650 - positions,
651 - kv_caches,
652 - attn_metadata,
653 - inputs_embeds=inputs_embeds)
648 + else:
649 + image_input = self._parse_and_validate_image_input(**kwargs)
650 +
651 + if image_input is not None:
652 + vision_embeddings = self._process_image_input(image_input)
653 + inputs_embeds = self.language_model.model.get_input_embeddings(
654 + input_ids)
655 +
656 + inputs_embeds = merge_multimodal_embeddings(
657 + input_ids, inputs_embeds, vision_embeddings,
658 + BLIP2_IMAGE_TOKEN_ID)
659 +
660 + input_ids = None
661 + else:
662 + inputs_embeds = None
663 +
664 + hidden_states = self.language_model.model(
665 + input_ids,
666 + positions,
667 + kv_caches,
668 + attn_metadata,
669 + intermediate_tensors=intermediate_tensors,
670 + inputs_embeds=inputs_embeds)
671
672 return hidden_states
673
vllm/model_executor/models/bloom.py
+34 -16
@@ -17,7 +17,7 @@
17 # limitations under the License.
18 """Inference-only BLOOM model compatible with HuggingFace weights."""
19 import math
20 -from typing import Iterable, List, Optional, Tuple
20 +from typing import Iterable, List, Optional, Tuple, Union
21
22 import torch
23 from torch import nn
@@ -25,15 +25,14 @@ from transformers import BloomConfig
25
26 from vllm.attention import Attention, AttentionMetadata
27 from vllm.config import CacheConfig
28 -from vllm.distributed import (get_tensor_model_parallel_rank,
28 +from vllm.distributed import (get_pp_group, get_tensor_model_parallel_rank,
29 get_tensor_model_parallel_world_size)
30 from vllm.model_executor.layers.activation import get_act_fn
31 from vllm.model_executor.layers.linear import (ColumnParallelLinear,
32 QKVParallelLinear,
33 RowParallelLinear)
34 from vllm.model_executor.layers.logits_processor import LogitsProcessor
35 -from vllm.model_executor.layers.quantization.base_config import (
36 - QuantizationConfig)
35 +from vllm.model_executor.layers.quantization import QuantizationConfig
36 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
37 from vllm.model_executor.layers.vocab_parallel_embedding import (
38 ParallelLMHead, VocabParallelEmbedding)
@@ -41,6 +40,10 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
40 from vllm.model_executor.sampling_metadata import SamplingMetadata
41 from vllm.sequence import IntermediateTensors
42
43 +from .interfaces import SupportsPP
44 +from .utils import (is_pp_missing_parameter,
45 + make_empty_intermediate_tensors_factory, make_layers)
46 +
47
48 def _get_alibi_slopes(total_num_heads: int) -> torch.Tensor:
49 closest_power_of_2 = 2**math.floor(math.log2(total_num_heads))
@@ -222,6 +225,7 @@ class BloomModel(nn.Module):
225 config: BloomConfig,
226 cache_config: Optional[CacheConfig] = None,
227 quant_config: Optional[QuantizationConfig] = None,
228 + prefix: str = "",
229 ):
230 super().__init__()
231 self.embed_dim = config.hidden_size
@@ -235,13 +239,16 @@ class BloomModel(nn.Module):
239 self.embed_dim, eps=config.layer_norm_epsilon)
240
241 # Transformer blocks
238 - self.h = nn.ModuleList([
239 - BloomBlock(config, cache_config, quant_config)
240 - for _ in range(config.num_hidden_layers)
241 - ])
242 + self.start_layer, self.end_layer, self.h = make_layers(
243 + config.num_hidden_layers,
244 + lambda prefix: BloomBlock(config, cache_config, quant_config),
245 + prefix=f"{prefix}.h")
246
247 # Final Layer Norm
248 self.ln_f = nn.LayerNorm(self.embed_dim, eps=config.layer_norm_epsilon)
249 + self.make_empty_intermediate_tensors = (
250 + make_empty_intermediate_tensors_factory(["hidden_states"],
251 + config.hidden_size))
252
253 def forward(
254 self,
@@ -249,22 +256,29 @@ class BloomModel(nn.Module):
256 position_ids: torch.Tensor,
257 kv_caches: List[torch.Tensor],
258 attn_metadata: AttentionMetadata,
252 - ) -> torch.Tensor:
253 - hidden_states = self.word_embeddings(input_ids)
254 - hidden_states = self.word_embeddings_layernorm(hidden_states)
255 - for i in range(len(self.h)):
259 + intermediate_tensors: Optional[IntermediateTensors],
260 + ) -> Union[torch.Tensor, IntermediateTensors]:
261 + if get_pp_group().is_first_rank:
262 + hidden_states = self.word_embeddings(input_ids)
263 + hidden_states = self.word_embeddings_layernorm(hidden_states)
264 + else:
265 + assert intermediate_tensors is not None
266 + hidden_states = intermediate_tensors["hidden_states"]
267 + for i in range(self.start_layer, self.end_layer):
268 layer = self.h[i]
269 hidden_states = layer(
270 position_ids,
271 hidden_states,
260 - kv_caches[i],
272 + kv_caches[i - self.start_layer],
273 attn_metadata,
274 )
275 + if not get_pp_group().is_last_rank:
276 + return IntermediateTensors({"hidden_states": hidden_states})
277 hidden_states = self.ln_f(hidden_states)
278 return hidden_states
279
280
267 -class BloomForCausalLM(nn.Module):
281 +class BloomForCausalLM(nn.Module, SupportsPP):
282
283 def __init__(
284 self,
@@ -284,6 +298,8 @@ class BloomForCausalLM(nn.Module):
298
299 self.logits_processor = LogitsProcessor(config.vocab_size)
300 self.sampler = Sampler()
301 + self.make_empty_intermediate_tensors = (
302 + self.transformer.make_empty_intermediate_tensors)
303
304 def forward(
305 self,
@@ -292,9 +308,9 @@ class BloomForCausalLM(nn.Module):
308 kv_caches: List[torch.Tensor],
309 attn_metadata: AttentionMetadata,
310 intermediate_tensors: Optional[IntermediateTensors] = None,
295 - ) -> torch.Tensor:
311 + ) -> Union[torch.Tensor, IntermediateTensors]:
312 hidden_states = self.transformer(input_ids, positions, kv_caches,
297 - attn_metadata)
313 + attn_metadata, intermediate_tensors)
314 return hidden_states
315
316 def compute_logits(
@@ -321,6 +337,8 @@ class BloomForCausalLM(nn.Module):
337 continue
338 if not name.startswith("transformer."):
339 name = "transformer." + name
340 + if is_pp_missing_parameter(name, self):
341 + continue
342 param = params_dict[name]
343
344 if "query_key_value" in name:
vllm/model_executor/models/chameleon.py
+63 -30
@@ -1,6 +1,6 @@
1 from functools import cached_property
2 from typing import (Any, Dict, Iterable, List, Literal, Mapping, Optional,
3 - Tuple, TypedDict)
3 + Tuple, TypedDict, Union)
4
5 import torch
6 import torch.nn.functional as F
@@ -10,7 +10,7 @@ from transformers import ChameleonConfig, ChameleonVQVAEConfig
10
11 from vllm.attention import Attention, AttentionMetadata
12 from vllm.config import CacheConfig, MultiModalConfig
13 -from vllm.distributed import get_tensor_model_parallel_world_size
13 +from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
14 from vllm.inputs import INPUT_REGISTRY, InputContext, LLMInputs
15 from vllm.model_executor.layers.activation import SiluAndMul
16 from vllm.model_executor.layers.layernorm import RMSNorm
@@ -33,7 +33,9 @@ from vllm.multimodal.utils import (cached_get_tokenizer,
33 from vllm.sequence import IntermediateTensors, SequenceData
34 from vllm.utils import print_warning_once
35
36 -from .interfaces import SupportsMultiModal
36 +from .interfaces import SupportsMultiModal, SupportsPP
37 +from .utils import (is_pp_missing_parameter,
38 + make_empty_intermediate_tensors_factory, make_layers)
39
40 # These configs are not part of the model config but the preprocessor
41 # and processor files, so we hardcode them in the model file for now.
@@ -822,6 +824,7 @@ class ChameleonModel(nn.Module):
824 config: ChameleonConfig,
825 cache_config: Optional[CacheConfig] = None,
826 quant_config: Optional[QuantizationConfig] = None,
827 + prefix: str = "",
828 ) -> None:
829 super().__init__()
830 self.config = config
@@ -835,14 +838,20 @@ class ChameleonModel(nn.Module):
838 config.vocabulary_map)
839 decoder_layer = ChameleonDecoderLayer if not self.config.swin_norm \
840 else ChameleonSwinDecoderLayer
838 - self.layers = nn.ModuleList([
839 - decoder_layer(config=config,
840 - cache_config=cache_config,
841 - quant_config=quant_config)
842 - for _ in range(config.num_hidden_layers)
843 - ])
841 +
842 + self.start_layer, self.end_layer, self.layers = make_layers(
843 + config.num_hidden_layers,
844 + lambda prefix: decoder_layer(config=config,
845 + cache_config=cache_config,
846 + quant_config=quant_config),
847 + prefix=f"{prefix}.layers",
848 + )
849 +
850 self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
851 self.vqmodel = ChameleonVQVAE(config.vq_config)
852 + self.make_empty_intermediate_tensors = (
853 + make_empty_intermediate_tensors_factory(
854 + ["hidden_states", "residual"], config.hidden_size))
855
856 def get_input_embeddings(self, input_ids: torch.Tensor) -> torch.Tensor:
857 return self.embed_tokens(input_ids)
@@ -865,22 +874,33 @@ class ChameleonModel(nn.Module):
874 positions: torch.Tensor,
875 kv_caches: List[torch.Tensor],
876 attn_metadata: AttentionMetadata,
877 + intermediate_tensors: Optional[IntermediateTensors],
878 inputs_embeds: Optional[torch.Tensor] = None,
869 - ) -> torch.Tensor:
870 - if inputs_embeds is not None:
871 - hidden_states = inputs_embeds
879 + ) -> Union[torch.Tensor, IntermediateTensors]:
880 + if get_pp_group().is_first_rank:
881 + if inputs_embeds is not None:
882 + hidden_states = inputs_embeds
883 + else:
884 + hidden_states = self.get_input_embeddings(input_ids)
885 + residual = None
886 else:
873 - hidden_states = self.get_input_embeddings(input_ids)
874 - residual = None
875 - for i in range(len(self.layers)):
887 + assert intermediate_tensors is not None
888 + hidden_states = intermediate_tensors["hidden_states"]
889 + residual = intermediate_tensors["residual"]
890 + for i in range(self.start_layer, self.end_layer):
891 layer = self.layers[i]
892 hidden_states, residual = layer(
893 positions,
894 hidden_states,
880 - kv_caches[i],
895 + kv_caches[i - self.start_layer],
896 attn_metadata,
897 residual,
898 )
899 + if not get_pp_group().is_last_rank:
900 + return IntermediateTensors({
901 + "hidden_states": hidden_states,
902 + "residual": residual
903 + })
904 hidden_states, _ = self.norm(hidden_states, residual)
905 return hidden_states
906
@@ -889,7 +909,8 @@ class ChameleonModel(nn.Module):
909 @MULTIMODAL_REGISTRY.register_max_image_tokens(get_max_chameleon_image_tokens)
910 @INPUT_REGISTRY.register_dummy_data(dummy_data_for_chameleon)
911 @INPUT_REGISTRY.register_input_processor(input_processor_for_chameleon)
892 -class ChameleonForConditionalGeneration(nn.Module, SupportsMultiModal):
912 +class ChameleonForConditionalGeneration(nn.Module, SupportsMultiModal,
913 + SupportsPP):
914
915 def __init__(
916 self,
@@ -914,6 +935,8 @@ class ChameleonForConditionalGeneration(nn.Module, SupportsMultiModal):
935 self.logits_processor = LogitsProcessor(self.unpadded_vocab_size,
936 config.vocab_size, logit_scale)
937 self.sampler = Sampler()
938 + self.make_empty_intermediate_tensors = (
939 + self.model.make_empty_intermediate_tensors)
940
941 def _validate_pixel_values(self, data: torch.Tensor) -> torch.Tensor:
942
@@ -956,22 +979,26 @@ class ChameleonForConditionalGeneration(nn.Module, SupportsMultiModal):
979 attn_metadata: AttentionMetadata,
980 intermediate_tensors: Optional[IntermediateTensors] = None,
981 **kwargs,
959 - ) -> torch.Tensor:
960 -
961 - image_input = self._parse_and_validate_image_input(**kwargs)
982 + ) -> Union[torch.Tensor, IntermediateTensors]:
983
963 - if image_input is not None:
964 - assert self.model.vqmodel is not None
965 - image_tokens = self.model.get_image_tokens(image_input["data"].to(
966 - self.config.torch_dtype))
967 - image_token_id = self.model.vocabulary_mapping.image_token_id
968 - special_image_mask = input_ids == image_token_id
969 - image_tokens = image_tokens.to(input_ids.device, input_ids.dtype)
970 - input_ids = input_ids.masked_scatter(special_image_mask,
971 - image_tokens)
984 + if intermediate_tensors is not None:
985 + input_ids = None
986 + else:
987 + image_input = self._parse_and_validate_image_input(**kwargs)
988 +
989 + if image_input is not None:
990 + assert self.model.vqmodel is not None
991 + image_tokens = self.model.get_image_tokens(
992 + image_input["data"].to(self.config.torch_dtype))
993 + image_token_id = self.model.vocabulary_mapping.image_token_id
994 + special_image_mask = input_ids == image_token_id
995 + image_tokens = image_tokens.to(input_ids.device,
996 + input_ids.dtype)
997 + input_ids = input_ids.masked_scatter(special_image_mask,
998 + image_tokens)
999
1000 hidden_states = self.model(input_ids, positions, kv_caches,
974 - attn_metadata)
1001 + attn_metadata, intermediate_tensors)
1002 return hidden_states
1003
1004 def compute_logits(
@@ -1039,6 +1066,8 @@ class ChameleonForConditionalGeneration(nn.Module, SupportsMultiModal):
1066 # Skip loading extra bias for GPTQ models.
1067 if name.endswith(".bias") and name not in params_dict:
1068 continue
1069 + if is_pp_missing_parameter(name, self):
1070 + continue
1071 param = params_dict[name]
1072 weight_loader = param.weight_loader
1073 weight_loader(param, loaded_weight, shard_id)
@@ -1060,11 +1089,15 @@ class ChameleonForConditionalGeneration(nn.Module, SupportsMultiModal):
1089 continue
1090 else:
1091 name = remapped_kv_scale_name
1092 + if is_pp_missing_parameter(name, self):
1093 + continue
1094 param = params_dict[name]
1095 weight_loader = getattr(param, "weight_loader",
1096 default_weight_loader)
1097 weight_loader(param, loaded_weight)
1098 if use_default_weight_loading and name in params_dict:
1099 + if is_pp_missing_parameter(name, self):
1100 + continue
1101 param = params_dict[name]
1102 weight_loader = getattr(param, "weight_loader",
1103 default_weight_loader)
vllm/model_executor/models/chatglm.py
+39 -22
@@ -2,7 +2,7 @@
2 # Adapted from
3 # https://github.com/THUDM/ChatGLM2-6B
4 """Inference-only ChatGLM model compatible with THUDM weights."""
5 -from typing import Iterable, List, Optional, Tuple
5 +from typing import Iterable, List, Optional, Tuple, Union
6
7 import torch
8 from torch import nn
@@ -10,15 +10,14 @@ from torch.nn import LayerNorm
10
11 from vllm.attention import Attention, AttentionMetadata
12 from vllm.config import CacheConfig, LoRAConfig
13 -from vllm.distributed import get_tensor_model_parallel_world_size
13 +from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
14 from vllm.model_executor.layers.activation import SiluAndMul
15 from vllm.model_executor.layers.layernorm import RMSNorm
16 from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
17 QKVParallelLinear,
18 RowParallelLinear)
19 from vllm.model_executor.layers.logits_processor import LogitsProcessor
20 -from vllm.model_executor.layers.quantization.base_config import (
21 - QuantizationConfig)
20 +from vllm.model_executor.layers.quantization import QuantizationConfig
21 from vllm.model_executor.layers.rotary_embedding import get_rope
22 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
23 from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -28,14 +27,16 @@ from vllm.model_executor.sampling_metadata import SamplingMetadata
27 from vllm.sequence import IntermediateTensors
28 from vllm.transformers_utils.configs import ChatGLMConfig
29
31 -from .interfaces import SupportsLoRA
30 +from .interfaces import SupportsLoRA, SupportsPP
31 +from .utils import (is_pp_missing_parameter,
32 + make_empty_intermediate_tensors_factory, make_layers)
33
34
35 class GLMAttention(nn.Module):
36
37 def __init__(
38 self,
38 - config,
39 + config: ChatGLMConfig,
40 cache_config: Optional[CacheConfig] = None,
41 quant_config: Optional[QuantizationConfig] = None,
42 ):
@@ -126,7 +127,7 @@ class GLMMLP(nn.Module):
127
128 def __init__(
129 self,
129 - config,
130 + config: ChatGLMConfig,
131 quant_config: Optional[QuantizationConfig] = None,
132 ):
133 super().__init__()
@@ -169,7 +170,7 @@ class GLMBlock(nn.Module):
170
171 def __init__(
172 self,
172 - config,
173 + config: ChatGLMConfig,
174 cache_config: Optional[CacheConfig] = None,
175 quant_config: Optional[QuantizationConfig] = None,
176 ):
@@ -240,9 +241,10 @@ class GLMTransformer(nn.Module):
241
242 def __init__(
243 self,
243 - config,
244 + config: ChatGLMConfig,
245 cache_config: Optional[CacheConfig] = None,
246 quant_config: Optional[QuantizationConfig] = None,
247 + prefix: str = "",
248 ):
249 super().__init__()
250 self.post_layer_norm = config.post_layer_norm
@@ -251,10 +253,11 @@ class GLMTransformer(nn.Module):
253 self.num_layers = config.num_layers
254
255 # Transformer layers.
254 - self.layers = nn.ModuleList([
255 - GLMBlock(config, cache_config, quant_config)
256 - for i in range(self.num_layers)
257 - ])
256 + self.start_layer, self.end_layer, self.layers = make_layers(
257 + self.num_layers,
258 + lambda prefix: GLMBlock(config, cache_config, quant_config),
259 + prefix=f"{prefix}.layers",
260 + )
261
262 if self.post_layer_norm:
263 layer_norm_func = RMSNorm if config.rmsnorm else LayerNorm
@@ -269,16 +272,16 @@ class GLMTransformer(nn.Module):
272 kv_caches: List[torch.Tensor],
273 attn_metadata: AttentionMetadata,
274 ) -> torch.Tensor:
272 - for i in range(self.num_layers):
275 + for i in range(self.start_layer, self.end_layer):
276 layer = self.layers[i]
277 hidden_states = layer(
278 hidden_states=hidden_states,
279 position_ids=position_ids,
277 - kv_cache=kv_caches[i],
280 + kv_cache=kv_caches[i - self.start_layer],
281 attn_metadata=attn_metadata,
282 )
283 # Final layer norm.
281 - if self.post_layer_norm:
284 + if get_pp_group().is_last_rank and self.post_layer_norm:
285 hidden_states = self.final_layernorm(hidden_states)
286
287 return hidden_states
@@ -288,7 +291,7 @@ class ChatGLMModel(nn.Module):
291
292 def __init__(
293 self,
291 - config,
294 + config: ChatGLMConfig,
295 cache_config: Optional[CacheConfig] = None,
296 quant_config: Optional[QuantizationConfig] = None,
297 ):
@@ -305,6 +308,9 @@ class ChatGLMModel(nn.Module):
308 self.output_layer = ParallelLMHead(config.padded_vocab_size,
309 config.hidden_size,
310 quant_config=quant_config)
311 + self.make_empty_intermediate_tensors = (
312 + make_empty_intermediate_tensors_factory(["hidden_states"],
313 + config.hidden_size))
314
315 def forward(
316 self,
@@ -312,8 +318,12 @@ class ChatGLMModel(nn.Module):
318 position_ids: torch.Tensor,
319 kv_caches: List[torch.Tensor],
320 attn_metadata: AttentionMetadata,
315 - ) -> torch.Tensor:
316 - inputs_embeds = self.embedding(input_ids)
321 + intermediate_tensors: Optional[IntermediateTensors],
322 + ) -> Union[torch.Tensor, IntermediateTensors]:
323 + if get_pp_group().is_first_rank:
324 + inputs_embeds = self.embedding(input_ids)
325 + else:
326 + inputs_embeds = intermediate_tensors["hidden_states"]
327
328 # Run encoder.
329 hidden_states = self.encoder(
@@ -322,10 +332,13 @@ class ChatGLMModel(nn.Module):
332 kv_caches=kv_caches,
333 attn_metadata=attn_metadata,
334 )
335 +
336 + if not get_pp_group().is_last_rank:
337 + return IntermediateTensors({"hidden_states": hidden_states})
338 return hidden_states
339
340
328 -class ChatGLMForCausalLM(nn.Module, SupportsLoRA):
341 +class ChatGLMForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
342 packed_modules_mapping = {
343 "query_key_value": ["query_key_value"],
344 "dense_h_to_4h": ["dense_h_to_4h"]
@@ -362,6 +375,8 @@ class ChatGLMForCausalLM(nn.Module, SupportsLoRA):
375 self.lm_head = self.transformer.output_layer
376 self.logits_processor = LogitsProcessor(config.padded_vocab_size)
377 self.sampler = Sampler()
378 + self.make_empty_intermediate_tensors = (
379 + self.transformer.make_empty_intermediate_tensors)
380
381 def forward(
382 self,
@@ -370,9 +385,9 @@ class ChatGLMForCausalLM(nn.Module, SupportsLoRA):
385 kv_caches: List[torch.Tensor],
386 attn_metadata: AttentionMetadata,
387 intermediate_tensors: Optional[IntermediateTensors] = None,
373 - ) -> torch.Tensor:
388 + ) -> Union[torch.Tensor, IntermediateTensors]:
389 hidden_states = self.transformer(input_ids, positions, kv_caches,
375 - attn_metadata)
390 + attn_metadata, intermediate_tensors)
391 return hidden_states
392
393 def compute_logits(
@@ -402,6 +417,8 @@ class ChatGLMForCausalLM(nn.Module, SupportsLoRA):
417 # Skip loading extra bias for GPTQ models.
418 if name.endswith(".bias") and name not in params_dict:
419 continue
420 + if is_pp_missing_parameter(name, self):
421 + continue
422 param = params_dict[name]
423 weight_loader = getattr(param, "weight_loader",
424 default_weight_loader)
vllm/model_executor/models/commandr.py
+41 -18
@@ -20,7 +20,7 @@
20
21 # This file is based on the LLama model definition file in transformers
22 """PyTorch Cohere model."""
23 -from typing import Iterable, List, Optional, Set, Tuple
23 +from typing import Iterable, List, Optional, Set, Tuple, Union
24
25 import torch
26 import torch.utils.checkpoint
@@ -29,14 +29,13 @@ from transformers import CohereConfig
29
30 from vllm.attention import Attention, AttentionMetadata
31 from vllm.config import CacheConfig, LoRAConfig
32 -from vllm.distributed import get_tensor_model_parallel_world_size
32 +from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
33 from vllm.model_executor.layers.activation import SiluAndMul
34 from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
35 QKVParallelLinear,
36 RowParallelLinear)
37 from vllm.model_executor.layers.logits_processor import LogitsProcessor
38 -from vllm.model_executor.layers.quantization.base_config import (
39 - QuantizationConfig)
38 +from vllm.model_executor.layers.quantization import QuantizationConfig
39 from vllm.model_executor.layers.rotary_embedding import get_rope
40 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
41 from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -47,7 +46,9 @@ from vllm.model_executor.sampling_metadata import SamplingMetadata
46 from vllm.model_executor.utils import set_weight_attrs
47 from vllm.sequence import IntermediateTensors
48
50 -from .interfaces import SupportsLoRA
49 +from .interfaces import SupportsLoRA, SupportsPP
50 +from .utils import (is_pp_missing_parameter,
51 + make_empty_intermediate_tensors_factory, make_layers)
52
53
54 @torch.compile
@@ -82,7 +83,7 @@ class CohereMLP(nn.Module):
83
84 def __init__(
85 self,
85 - config,
86 + config: CohereConfig,
87 quant_config: Optional[QuantizationConfig] = None,
88 ):
89 super().__init__()
@@ -256,6 +257,7 @@ class CohereModel(nn.Module):
257 cache_config: Optional[CacheConfig] = None,
258 quant_config: Optional[QuantizationConfig] = None,
259 lora_config: Optional[LoRAConfig] = None,
260 + prefix: str = "",
261 ):
262 super().__init__()
263 self.config = config
@@ -265,12 +267,16 @@ class CohereModel(nn.Module):
267 self.org_vocab_size = config.vocab_size
268 self.embed_tokens = VocabParallelEmbedding(config.vocab_size,
269 config.hidden_size)
268 - self.layers = nn.ModuleList([
269 - CohereDecoderLayer(config, cache_config, quant_config=quant_config)
270 - for _ in range(config.num_hidden_layers)
271 - ])
270 + self.start_layer, self.end_layer, self.layers = make_layers(
271 + config.num_hidden_layers,
272 + lambda prefix: CohereDecoderLayer(config, cache_config,
273 + quant_config),
274 + prefix=f"{prefix}.layers")
275 self.norm = LayerNorm(param_shape=(config.hidden_size),
276 eps=config.layer_norm_eps)
277 + self.make_empty_intermediate_tensors = (
278 + make_empty_intermediate_tensors_factory(
279 + ["hidden_states", "residual"], config.hidden_size))
280
281 def forward(
282 self,
@@ -278,23 +284,34 @@ class CohereModel(nn.Module):
284 positions: torch.Tensor,
285 kv_caches: List[torch.Tensor],
286 attn_metadata: AttentionMetadata,
281 - ) -> torch.Tensor:
282 - hidden_states = self.embed_tokens(input_ids)
283 - residual = None
284 - for i in range(len(self.layers)):
287 + intermediate_tensors: Optional[IntermediateTensors],
288 + ) -> Union[torch.Tensor, IntermediateTensors]:
289 + if get_pp_group().is_first_rank:
290 + hidden_states = self.embed_tokens(input_ids)
291 + residual = None
292 + else:
293 + assert intermediate_tensors is not None
294 + hidden_states = intermediate_tensors["hidden_states"]
295 + residual = intermediate_tensors["residual"]
296 + for i in range(self.start_layer, self.end_layer):
297 layer = self.layers[i]
298 hidden_states, residual = layer(
299 positions,
300 hidden_states,
289 - kv_caches[i],
301 + kv_caches[i - self.start_layer],
302 attn_metadata,
303 residual,
304 )
305 + if not get_pp_group().is_last_rank:
306 + return IntermediateTensors({
307 + "hidden_states": hidden_states,
308 + "residual": residual
309 + })
310 hidden_states, _ = self.norm(hidden_states, residual)
311 return hidden_states
312
313
297 -class CohereForCausalLM(nn.Module, SupportsLoRA):
314 +class CohereForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
315 packed_modules_mapping = {
316 "qkv_proj": [
317 "q_proj",
@@ -337,6 +354,8 @@ class CohereForCausalLM(nn.Module, SupportsLoRA):
354 quant_config,
355 lora_config=lora_config)
356 self.sampler = Sampler()
357 + self.make_empty_intermediate_tensors = (
358 + self.model.make_empty_intermediate_tensors)
359
360 @torch.no_grad()
361 def forward(
@@ -346,9 +365,9 @@ class CohereForCausalLM(nn.Module, SupportsLoRA):
365 kv_caches: List[torch.Tensor],
366 attn_metadata: AttentionMetadata,
367 intermediate_tensors: Optional[IntermediateTensors] = None,
349 - ) -> torch.Tensor:
368 + ) -> Union[torch.Tensor, IntermediateTensors]:
369 hidden_states = self.model(input_ids, positions, kv_caches,
351 - attn_metadata)
370 + attn_metadata, intermediate_tensors)
371 return hidden_states
372
373 def compute_logits(
@@ -393,6 +412,8 @@ class CohereForCausalLM(nn.Module, SupportsLoRA):
412 # Skip loading extra bias for GPTQ models.
413 if name.endswith(".bias") and name not in params_dict:
414 continue
415 + if is_pp_missing_parameter(name, self):
416 + continue
417 param = params_dict[name]
418 weight_loader = param.weight_loader
419 weight_loader(param, loaded_weight, shard_id)
@@ -405,6 +426,8 @@ class CohereForCausalLM(nn.Module, SupportsLoRA):
426 # Skip loading extra bias for GPTQ models.
427 if name.endswith(".bias") and name not in params_dict:
428 continue
429 + if is_pp_missing_parameter(name, self):
430 + continue
431 param = params_dict[name]
432 weight_loader = getattr(param, "weight_loader",
433 default_weight_loader)
vllm/model_executor/models/dbrx.py
+36 -15
@@ -1,20 +1,19 @@
1 # coding=utf-8
2 -from typing import Iterable, List, Optional, Tuple
2 +from typing import Iterable, List, Optional, Tuple, Union
3
4 import torch
5 import torch.nn as nn
6
7 from vllm.attention import Attention, AttentionMetadata
8 from vllm.config import CacheConfig
9 -from vllm.distributed import (get_tensor_model_parallel_rank,
9 +from vllm.distributed import (get_pp_group, get_tensor_model_parallel_rank,
10 get_tensor_model_parallel_world_size)
11 from vllm.model_executor.layers.fused_moe import FusedMoE
12 from vllm.model_executor.layers.linear import (QKVParallelLinear,
13 ReplicatedLinear,
14 RowParallelLinear)
15 from vllm.model_executor.layers.logits_processor import LogitsProcessor
16 -from vllm.model_executor.layers.quantization.base_config import (
17 - QuantizationConfig)
16 +from vllm.model_executor.layers.quantization import QuantizationConfig
17 from vllm.model_executor.layers.rotary_embedding import get_rope
18 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
19 from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -24,6 +23,10 @@ from vllm.model_executor.sampling_metadata import SamplingMetadata
23 from vllm.sequence import IntermediateTensors
24 from vllm.transformers_utils.configs.dbrx import DbrxConfig
25
26 +from .interfaces import SupportsPP
27 +from .utils import (is_pp_missing_parameter,
28 + make_empty_intermediate_tensors_factory, make_layers)
29 +
30
31 class DbrxRouter(nn.Module):
32 """A Router implementation for DBRX that returns logits for each expert
@@ -296,22 +299,27 @@ class DbrxModel(nn.Module):
299 config: DbrxConfig,
300 cache_config: Optional[CacheConfig] = None,
301 quant_config: Optional[QuantizationConfig] = None,
302 + prefix: str = "",
303 ):
304 super().__init__()
305 self.wte = VocabParallelEmbedding(
306 config.vocab_size,
307 config.d_model,
308 )
305 - self.blocks = nn.ModuleList([
306 - DbrxBlock(config, cache_config, quant_config)
307 - for _ in range(config.n_layers)
308 - ])
309 + self.start_layer, self.end_layer, self.blocks = make_layers(
310 + config.n_layers,
311 + lambda prefix: DbrxBlock(config, cache_config, quant_config),
312 + prefix=f"{prefix}.blocks",
313 + )
314 self.norm_f = nn.LayerNorm(config.d_model, eps=1e-5)
315 for module in self.modules():
316 if hasattr(module, "bias") and isinstance(module.bias,
317 nn.Parameter):
318 # Remove the bias term in Linear and LayerNorm.
319 module.register_parameter("bias", None)
320 + self.make_empty_intermediate_tensors = (
321 + make_empty_intermediate_tensors_factory(["hidden_states"],
322 + config.d_model))
323
324 def forward(
325 self,
@@ -319,21 +327,28 @@ class DbrxModel(nn.Module):
327 position_ids: torch.Tensor,
328 kv_caches: List[torch.Tensor],
329 attn_metadata: AttentionMetadata,
322 - ) -> torch.Tensor:
323 - hidden_states = self.wte(input_ids)
324 - for i in range(len(self.blocks)):
330 + intermediate_tensors: Optional[IntermediateTensors],
331 + ) -> Union[torch.Tensor, IntermediateTensors]:
332 + if get_pp_group().is_first_rank:
333 + hidden_states = self.wte(input_ids)
334 + else:
335 + assert intermediate_tensors
336 + hidden_states = intermediate_tensors["hidden_states"]
337 + for i in range(self.start_layer, self.end_layer):
338 block = self.blocks[i]
339 hidden_states = block(
340 position_ids,
341 hidden_states,
329 - kv_caches[i],
342 + kv_caches[i - self.start_layer],
343 attn_metadata,
344 )
345 + if not get_pp_group().is_last_rank:
346 + return IntermediateTensors({"hidden_states": hidden_states})
347 hidden_states = self.norm_f(hidden_states)
348 return hidden_states
349
350
336 -class DbrxForCausalLM(nn.Module):
351 +class DbrxForCausalLM(nn.Module, SupportsPP):
352
353 def __init__(
354 self,
@@ -359,6 +374,8 @@ class DbrxForCausalLM(nn.Module):
374 self.logits_processor = LogitsProcessor(self.unpadded_vocab_size,
375 config.vocab_size)
376 self.sampler = Sampler()
377 + self.make_empty_intermediate_tensors = (
378 + self.transformer.make_empty_intermediate_tensors)
379
380 def forward(
381 self,
@@ -367,9 +384,9 @@ class DbrxForCausalLM(nn.Module):
384 kv_caches: List[torch.Tensor],
385 attn_metadata: AttentionMetadata,
386 intermediate_tensors: Optional[IntermediateTensors] = None,
370 - ) -> torch.Tensor:
387 + ) -> Union[torch.Tensor, IntermediateTensors]:
388 hidden_states = self.transformer(input_ids, positions, kv_caches,
372 - attn_metadata)
389 + attn_metadata, intermediate_tensors)
390 return hidden_states
391
392 def compute_logits(
@@ -401,11 +418,15 @@ class DbrxForCausalLM(nn.Module):
418 if weight_name not in name:
419 continue
420 name = name.replace(weight_name, param_name)
421 + if is_pp_missing_parameter(name, self):
422 + continue
423 param = params_dict[name]
424 weight_loader = param.weight_loader
425 weight_loader(param, loaded_weight, weight_name)
426 break
427 else:
428 + if is_pp_missing_parameter(name, self):
429 + continue
430 param = params_dict[name]
431 weight_loader = getattr(param, "weight_loader",
432 default_weight_loader)
vllm/model_executor/models/decilm.py
+7 -2
@@ -29,11 +29,12 @@ import torch
29 from transformers import LlamaConfig
30
31 from vllm.config import CacheConfig, LoRAConfig
32 -from vllm.model_executor.layers.quantization.base_config import (
33 - QuantizationConfig)
32 +from vllm.model_executor.layers.quantization import QuantizationConfig
33 from vllm.model_executor.model_loader.weight_utils import default_weight_loader
34 from vllm.model_executor.models.llama import LlamaForCausalLM
35
36 +from .utils import is_pp_missing_parameter
37 +
38
39 class DeciLMForCausalLM(LlamaForCausalLM):
40 """
@@ -91,6 +92,8 @@ class DeciLMForCausalLM(LlamaForCausalLM):
92 # Skip loading extra bias for GPTQ models.
93 if name.endswith(".bias") and name not in params_dict:
94 continue
95 + if is_pp_missing_parameter(name, self):
96 + continue
97 param = params_dict[name]
98 weight_loader = param.weight_loader
99 weight_loader(param, loaded_weight, shard_id)
@@ -99,6 +102,8 @@ class DeciLMForCausalLM(LlamaForCausalLM):
102 # Skip loading extra bias for GPTQ models.
103 if name.endswith(".bias") and name not in params_dict:
104 continue
105 + if is_pp_missing_parameter(name, self):
106 + continue
107 param = params_dict[name]
108 weight_loader = getattr(param, "weight_loader",
109 default_weight_loader)
vllm/model_executor/models/deepseek.py
+43 -20
@@ -21,7 +21,7 @@
21 # See the License for the specific language governing permissions and
22 # limitations under the License.
23 """Inference-only Deepseek model."""
24 -from typing import Any, Dict, Iterable, List, Optional, Tuple
24 +from typing import Any, Dict, Iterable, List, Optional, Tuple, Union
25
26 import torch
27 from torch import nn
@@ -29,7 +29,7 @@ from transformers import PretrainedConfig
29
30 from vllm.attention import Attention, AttentionMetadata
31 from vllm.config import CacheConfig
32 -from vllm.distributed import (get_tensor_model_parallel_rank,
32 +from vllm.distributed import (get_pp_group, get_tensor_model_parallel_rank,
33 get_tensor_model_parallel_world_size,
34 tensor_model_parallel_all_reduce)
35 from vllm.model_executor.layers.activation import SiluAndMul
@@ -40,8 +40,7 @@ from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
40 ReplicatedLinear,
41 RowParallelLinear)
42 from vllm.model_executor.layers.logits_processor import LogitsProcessor
43 -from vllm.model_executor.layers.quantization.base_config import (
44 - QuantizationConfig)
43 +from vllm.model_executor.layers.quantization import QuantizationConfig
44 from vllm.model_executor.layers.rotary_embedding import get_rope
45 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
46 from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -50,6 +49,10 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
49 from vllm.model_executor.sampling_metadata import SamplingMetadata
50 from vllm.sequence import IntermediateTensors
51
52 +from .interfaces import SupportsPP
53 +from .utils import (is_pp_missing_parameter,
54 + make_empty_intermediate_tensors_factory, make_layers)
55 +
56
57 class DeepseekMLP(nn.Module):
58
@@ -329,6 +332,7 @@ class DeepseekModel(nn.Module):
332 config: PretrainedConfig,
333 cache_config: Optional[CacheConfig] = None,
334 quant_config: Optional[QuantizationConfig] = None,
335 + prefix: str = "",
336 ) -> None:
337 super().__init__()
338 self.padding_idx = config.pad_token_id
@@ -338,14 +342,17 @@ class DeepseekModel(nn.Module):
342 config.vocab_size,
343 config.hidden_size,
344 )
341 - self.layers = nn.ModuleList([
342 - DeepseekDecoderLayer(config,
343 - layer_idx,
344 - cache_config,
345 - quant_config=quant_config)
346 - for layer_idx in range(config.num_hidden_layers)
347 - ])
345 + self.start_layer, self.end_layer, self.layers = make_layers(
346 + config.num_hidden_layers,
347 + lambda prefix: DeepseekDecoderLayer(config,
348 + int(prefix.split(".")[-1]),
349 + cache_config,
350 + quant_config=quant_config),
351 + prefix=f"{prefix}.layers")
352 self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
353 + self.make_empty_intermediate_tensors = (
354 + make_empty_intermediate_tensors_factory(
355 + ["hidden_states", "residual"], config.hidden_size))
356
357 def forward(
358 self,
@@ -353,19 +360,29 @@ class DeepseekModel(nn.Module):
360 positions: torch.Tensor,
361 kv_caches: List[torch.Tensor],
362 attn_metadata: AttentionMetadata,
356 - ) -> torch.Tensor:
357 - hidden_states = self.embed_tokens(input_ids)
358 - residual = None
359 - for i in range(len(self.layers)):
363 + intermediate_tensors: Optional[IntermediateTensors],
364 + ) -> Union[torch.Tensor, IntermediateTensors]:
365 + if get_pp_group().is_first_rank:
366 + hidden_states = self.embed_tokens(input_ids)
367 + residual = None
368 + else:
369 + hidden_states = intermediate_tensors["hidden_states"]
370 + residual = intermediate_tensors["residual"]
371 + for i in range(self.start_layer, self.end_layer):
372 layer = self.layers[i]
373 hidden_states, residual = layer(positions, hidden_states,
362 - kv_caches[i], attn_metadata,
363 - residual)
374 + kv_caches[i - self.start_layer],
375 + attn_metadata, residual)
376 + if not get_pp_group().is_last_rank:
377 + return IntermediateTensors({
378 + "hidden_states": hidden_states,
379 + "residual": residual
380 + })
381 hidden_states, _ = self.norm(hidden_states, residual)
382 return hidden_states
383
384
368 -class DeepseekForCausalLM(nn.Module):
385 +class DeepseekForCausalLM(nn.Module, SupportsPP):
386
387 def __init__(
388 self,
@@ -384,6 +401,8 @@ class DeepseekForCausalLM(nn.Module):
401 self.lm_head.weight = self.model.embed_tokens.weight
402 self.logits_processor = LogitsProcessor(config.vocab_size)
403 self.sampler = Sampler()
404 + self.make_empty_intermediate_tensors = (
405 + self.model.make_empty_intermediate_tensors)
406
407 def forward(
408 self,
@@ -392,9 +411,9 @@ class DeepseekForCausalLM(nn.Module):
411 kv_caches: List[torch.Tensor],
412 attn_metadata: AttentionMetadata,
413 intermediate_tensors: Optional[IntermediateTensors] = None,
395 - ) -> torch.Tensor:
414 + ) -> Union[torch.Tensor, IntermediateTensors]:
415 hidden_states = self.model(input_ids, positions, kv_caches,
397 - attn_metadata)
416 + attn_metadata, intermediate_tensors)
417 return hidden_states
418
419 def compute_logits(
@@ -439,6 +458,8 @@ class DeepseekForCausalLM(nn.Module):
458 if (("mlp.experts." in name or "mlp.shared_experts." in name)
459 and name not in params_dict):
460 continue
461 + if is_pp_missing_parameter(name, self):
462 + continue
463 param = params_dict[name]
464 weight_loader = param.weight_loader
465 weight_loader(param, loaded_weight, shard_id)
@@ -451,6 +472,8 @@ class DeepseekForCausalLM(nn.Module):
472 if (("mlp.experts." in name or "mlp.shared_experts." in name)
473 and name not in params_dict):
474 continue
475 + if is_pp_missing_parameter(name, self):
476 + continue
477 param = params_dict[name]
478 weight_loader = getattr(param, "weight_loader",
479 default_weight_loader)
vllm/model_executor/models/deepseek_v2.py
+13 -7
@@ -21,7 +21,7 @@
21 # See the License for the specific language governing permissions and
22 # limitations under the License.
23 """Inference-only DeepseekV2 model."""
24 -from typing import Any, Dict, Iterable, List, Optional, Tuple
24 +from typing import Any, Dict, Iterable, List, Optional, Tuple, Union
25
26 import torch
27 from torch import nn
@@ -40,8 +40,7 @@ from vllm.model_executor.layers.linear import (ColumnParallelLinear,
40 ReplicatedLinear,
41 RowParallelLinear)
42 from vllm.model_executor.layers.logits_processor import LogitsProcessor
43 -from vllm.model_executor.layers.quantization.base_config import (
44 - QuantizationConfig)
43 +from vllm.model_executor.layers.quantization import QuantizationConfig
44 from vllm.model_executor.layers.rotary_embedding import get_rope
45 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
46 from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -50,7 +49,9 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
49 from vllm.model_executor.sampling_metadata import SamplingMetadata
50 from vllm.sequence import IntermediateTensors
51
53 -from .utils import PPMissingLayer, is_pp_missing_parameter, make_layers
52 +from .interfaces import SupportsPP
53 +from .utils import (PPMissingLayer, is_pp_missing_parameter,
54 + make_empty_intermediate_tensors_factory, make_layers)
55
56
57 class DeepseekV2MLP(nn.Module):
@@ -439,6 +440,9 @@ class DeepseekV2Model(nn.Module):
440 self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
441 else:
442 self.norm = PPMissingLayer()
443 + self.make_empty_intermediate_tensors = (
444 + make_empty_intermediate_tensors_factory(
445 + ["hidden_states", "residual"], config.hidden_size))
446
447 def forward(
448 self,
@@ -447,7 +451,7 @@ class DeepseekV2Model(nn.Module):
451 kv_caches: List[torch.Tensor],
452 attn_metadata: AttentionMetadata,
453 intermediate_tensors: Optional[IntermediateTensors],
450 - ) -> torch.Tensor:
454 + ) -> Union[torch.Tensor, IntermediateTensors]:
455 if get_pp_group().is_first_rank:
456 hidden_states = self.embed_tokens(input_ids)
457 residual = None
@@ -472,7 +476,7 @@ class DeepseekV2Model(nn.Module):
476 return hidden_states
477
478
475 -class DeepseekV2ForCausalLM(nn.Module):
479 +class DeepseekV2ForCausalLM(nn.Module, SupportsPP):
480
481 def __init__(
482 self,
@@ -492,6 +496,8 @@ class DeepseekV2ForCausalLM(nn.Module):
496 quant_config=quant_config)
497 self.logits_processor = LogitsProcessor(config.vocab_size)
498 self.sampler = Sampler()
499 + self.make_empty_intermediate_tensors = (
500 + self.model.make_empty_intermediate_tensors)
501
502 def forward(
503 self,
@@ -500,7 +506,7 @@ class DeepseekV2ForCausalLM(nn.Module):
506 kv_caches: List[torch.Tensor],
507 attn_metadata: AttentionMetadata,
508 intermediate_tensors: Optional[IntermediateTensors] = None,
503 - ) -> torch.Tensor:
509 + ) -> Union[torch.Tensor, IntermediateTensors]:
510 hidden_states = self.model(input_ids, positions, kv_caches,
511 attn_metadata, intermediate_tensors)
512 return hidden_states
vllm/model_executor/models/exaone.py
+12 -23
@@ -38,8 +38,7 @@ from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
38 QKVParallelLinear,
39 RowParallelLinear)
40 from vllm.model_executor.layers.logits_processor import LogitsProcessor
41 -from vllm.model_executor.layers.quantization.base_config import (
42 - QuantizationConfig)
41 +from vllm.model_executor.layers.quantization import QuantizationConfig
42 from vllm.model_executor.layers.quantization.compressed_tensors.utils import (
43 get_compressed_tensors_cache_scale)
44 from vllm.model_executor.layers.rotary_embedding import get_rope
@@ -53,8 +52,9 @@ from vllm.sequence import IntermediateTensors
52 from vllm.transformers_utils.configs.exaone import ExaoneConfig
53 from vllm.utils import is_hip
54
56 -from .interfaces import SupportsLoRA
57 -from .utils import PPMissingLayer, is_pp_missing_parameter, make_layers
55 +from .interfaces import SupportsLoRA, SupportsPP
56 +from .utils import (PPMissingLayer, is_pp_missing_parameter,
57 + make_empty_intermediate_tensors_factory, make_layers)
58
59
60 class ExaoneGatedMLP(nn.Module):
@@ -354,6 +354,10 @@ class ExaoneModel(nn.Module):
354 else:
355 self.ln_f = PPMissingLayer()
356
357 + self.make_empty_intermediate_tensors = (
358 + make_empty_intermediate_tensors_factory(
359 + ["hidden_states", "residual"], config.hidden_size))
360 +
361 def get_input_embeddings(self, input_ids: torch.Tensor) -> torch.Tensor:
362 return self.wte(input_ids)
363
@@ -397,7 +401,7 @@ class ExaoneModel(nn.Module):
401 return hidden_states
402
403
400 -class ExaoneForCausalLM(nn.Module, SupportsLoRA):
404 +class ExaoneForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
405 packed_modules_mapping = {
406 "qkv_proj": [
407 "q_proj",
@@ -477,6 +481,9 @@ class ExaoneForCausalLM(nn.Module, SupportsLoRA):
481 else:
482 self.lm_head = PPMissingLayer()
483
484 + self.make_empty_intermediate_tensors = (
485 + self.transformer.make_empty_intermediate_tensors)
486 +
487 def forward(
488 self,
489 input_ids: torch.Tensor,
@@ -506,24 +513,6 @@ class ExaoneForCausalLM(nn.Module, SupportsLoRA):
513 next_tokens = self.sampler(logits, sampling_metadata)
514 return next_tokens
515
509 - def make_empty_intermediate_tensors(
510 - self, batch_size: int, dtype: torch.dtype,
511 - device: torch.device) -> IntermediateTensors:
512 - return IntermediateTensors({
513 - "hidden_states":
514 - torch.zeros(
515 - (batch_size, self.config.hidden_size),
516 - dtype=dtype,
517 - device=device,
518 - ),
519 - "residual":
520 - torch.zeros(
521 - (batch_size, self.config.hidden_size),
522 - dtype=dtype,
523 - device=device,
524 - ),
525 - })
526 -
516 def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
517 stacked_params_mapping = [
518 # (param_name, shard_name, shard_id)
vllm/model_executor/models/falcon.py
+33 -19
@@ -28,7 +28,7 @@ from transformers import FalconConfig as HF_FalconConfig
28
29 from vllm.attention import Attention, AttentionMetadata
30 from vllm.config import CacheConfig
31 -from vllm.distributed import (get_tensor_model_parallel_rank,
31 +from vllm.distributed import (get_pp_group, get_tensor_model_parallel_rank,
32 get_tensor_model_parallel_world_size,
33 tensor_model_parallel_all_reduce)
34 from vllm.model_executor.layers.activation import get_act_fn
@@ -36,8 +36,7 @@ from vllm.model_executor.layers.linear import (ColumnParallelLinear,
36 QKVParallelLinear,
37 RowParallelLinear)
38 from vllm.model_executor.layers.logits_processor import LogitsProcessor
39 -from vllm.model_executor.layers.quantization.base_config import (
40 - QuantizationConfig)
39 +from vllm.model_executor.layers.quantization import QuantizationConfig
40 from vllm.model_executor.layers.rotary_embedding import get_rope
41 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
42 from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -47,6 +46,10 @@ from vllm.model_executor.sampling_metadata import SamplingMetadata
46 from vllm.sequence import IntermediateTensors
47 from vllm.transformers_utils.configs import RWConfig
48
49 +from .interfaces import SupportsPP
50 +from .utils import (is_pp_missing_parameter,
51 + make_empty_intermediate_tensors_factory, make_layers)
52 +
53 FalconConfig = Union[HF_FalconConfig, RWConfig]
54
55
@@ -333,6 +336,7 @@ class FalconModel(nn.Module):
336 config: FalconConfig,
337 cache_config: Optional[CacheConfig] = None,
338 quant_config: Optional[QuantizationConfig] = None,
339 + prefix: str = "",
340 ):
341 super().__init__()
342 self.config = config
@@ -347,35 +351,45 @@ class FalconModel(nn.Module):
351 )
352
353 # Transformer blocks
350 - self.h = nn.ModuleList([
351 - FalconDecoderLayer(config, cache_config, quant_config)
352 - for _ in range(config.num_hidden_layers)
353 - ])
354 + self.start_layer, self.end_layer, self.h = make_layers(
355 + config.num_hidden_layers,
356 + lambda prefix: FalconDecoderLayer(config, cache_config,
357 + quant_config),
358 + prefix=f"{prefix}.h")
359
360 # Final Layer Norm
361 self.ln_f = LayerNorm(self.embed_dim, eps=config.layer_norm_epsilon)
362 + self.make_empty_intermediate_tensors = (
363 + make_empty_intermediate_tensors_factory(["hidden_states"],
364 + config.hidden_size))
365
366 def forward(
367 self,
360 - input_ids: torch.LongTensor,
368 + input_ids: torch.Tensor,
369 positions: torch.Tensor,
370 kv_caches: List[torch.Tensor],
371 attn_metadata: AttentionMetadata,
364 - ) -> torch.Tensor:
365 - hidden_states = self.word_embeddings(input_ids)
366 - for i in range(len(self.h)):
372 + intermediate_tensors: Optional[IntermediateTensors],
373 + ) -> Union[torch.Tensor, IntermediateTensors]:
374 + if get_pp_group().is_first_rank:
375 + hidden_states = self.word_embeddings(input_ids)
376 + else:
377 + hidden_states = intermediate_tensors["hidden_states"]
378 + for i in range(self.start_layer, self.end_layer):
379 layer = self.h[i]
380 hidden_states = layer(
381 positions,
382 hidden_states,
371 - kv_caches[i],
383 + kv_caches[i - self.start_layer],
384 attn_metadata,
385 )
386 + if not get_pp_group().is_last_rank:
387 + return IntermediateTensors({"hidden_states": hidden_states})
388 hidden_states = self.ln_f(hidden_states)
389 return hidden_states
390
391
378 -class FalconForCausalLM(nn.Module):
392 +class FalconForCausalLM(nn.Module, SupportsPP):
393
394 def __init__(
395 self,
@@ -403,6 +417,8 @@ class FalconForCausalLM(nn.Module):
417 )
418 self.logits_processor = LogitsProcessor(config.vocab_size)
419 self.sampler = Sampler()
420 + self.make_empty_intermediate_tensors = (
421 + self.transformer.make_empty_intermediate_tensors)
422
423 def forward(
424 self,
@@ -412,12 +428,8 @@ class FalconForCausalLM(nn.Module):
428 attn_metadata: AttentionMetadata,
429 intermediate_tensors: Optional[IntermediateTensors] = None,
430 ) -> torch.Tensor:
415 - hidden_states = self.transformer(
416 - input_ids,
417 - positions,
418 - kv_caches,
419 - attn_metadata,
420 - )
431 + hidden_states = self.transformer(input_ids, positions, kv_caches,
432 + attn_metadata, intermediate_tensors)
433 return hidden_states
434
435 def compute_logits(
@@ -454,6 +466,8 @@ class FalconForCausalLM(nn.Module):
466 # Skip loading extra bias for GPTQ models.
467 if name.endswith(".bias") and name not in params_dict:
468 continue
469 + if is_pp_missing_parameter(name, self):
470 + continue
471 param = params_dict[name]
472 if "query_key_value" in name:
473 output_dim = getattr(param, "output_dim", None)
vllm/model_executor/models/fuyu.py
+34 -39
@@ -41,8 +41,9 @@ from vllm.multimodal.utils import cached_get_tokenizer
41 from vllm.sequence import (VLLM_TOKEN_ID_ARRAY_TYPE, IntermediateTensors,
42 SequenceData)
43
44 -from .interfaces import SupportsMultiModal
45 -from .utils import flatten_bn, merge_multimodal_embeddings
44 +from .interfaces import SupportsMultiModal, SupportsPP
45 +from .utils import (flatten_bn, group_weights_with_prefix,
46 + merge_multimodal_embeddings)
47
48 # Cannot find the following 2 numbers from hf config.
49 _IMAGE_TOKEN_ID = 71011
@@ -217,7 +218,7 @@ def input_mapper_for_fuyu(ctx: InputContext, data: object):
218 @MULTIMODAL_REGISTRY.register_max_image_tokens(get_max_fuyu_image_tokens)
219 @INPUT_REGISTRY.register_dummy_data(dummy_data_for_fuyu)
220 @INPUT_REGISTRY.register_input_processor(input_processor_for_fuyu)
220 -class FuyuForCausalLM(nn.Module, SupportsMultiModal):
221 +class FuyuForCausalLM(nn.Module, SupportsMultiModal, SupportsPP):
222
223 def __init__(self,
224 config: FuyuConfig,
@@ -242,6 +243,12 @@ class FuyuForCausalLM(nn.Module, SupportsMultiModal):
243 self.language_model = PersimmonForCausalLM(config.text_config,
244 cache_config=cache_config,
245 quant_config=quant_config)
246 + self.make_empty_intermediate_tensors = (
247 + self.language_model.make_empty_intermediate_tensors)
248 +
249 + @property
250 + def sampler(self):
251 + return self.language_model.sampler
252
253 def _validate_pixel_values(self, data: torch.Tensor) -> torch.Tensor:
254
@@ -297,23 +304,29 @@ class FuyuForCausalLM(nn.Module, SupportsMultiModal):
304 intermediate_tensors: Optional[IntermediateTensors] = None,
305 **kwargs: object,
306 ):
300 - image_input = self._parse_and_validate_image_input(**kwargs)
307 + if intermediate_tensors is not None:
308 + input_ids = None
309 + inputs_embeds = None
310 + else:
311 + image_input = self._parse_and_validate_image_input(**kwargs)
312
302 - if image_input is not None:
303 - vision_embeddings = self._process_image_input(image_input)
304 - inputs_embeds = self.language_model.model.embed_tokens(input_ids)
305 - inputs_embeds = merge_multimodal_embeddings(
306 - input_ids, inputs_embeds, vision_embeddings,
307 - self.image_token_id)
313 + if image_input is not None:
314 + vision_embeddings = self._process_image_input(image_input)
315 + inputs_embeds = self.language_model.model.embed_tokens(
316 + input_ids)
317 + inputs_embeds = merge_multimodal_embeddings(
318 + input_ids, inputs_embeds, vision_embeddings,
319 + self.image_token_id)
320
309 - else:
310 - inputs_embeds = None
321 + else:
322 + inputs_embeds = None
323
324 hidden_states = self.language_model(
325 input_ids=input_ids,
326 positions=positions,
327 kv_caches=kv_caches,
328 attn_metadata=attn_metadata,
329 + intermediate_tensors=intermediate_tensors,
330 inputs_embeds=inputs_embeds,
331 )
332 return hidden_states
@@ -336,34 +349,16 @@ class FuyuForCausalLM(nn.Module, SupportsMultiModal):
349 return next_tokens
350
351 def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
339 - params_dict = dict(self.named_parameters(remove_duplicate=False))
340 - for name, loaded_weight in weights:
341 - if "rotary_emb.inv_freq" in name:
342 - continue
343 - if ("rotary_emb.cos_cached" in name
344 - or "rotary_emb.sin_cached" in name):
345 - # Models trained using ColossalAI may include these tensors in
346 - # the checkpoint. Skip them.
347 - continue
348 - param = params_dict[name]
349 -
350 - if "query_key_value" in name:
351 - # copy from vllm/model_executor/models/bloom.py
352 - # NOTE: Fuyu's fused QKV's output_dim has the shape of
353 - # (num_heads * 3 * head_size), while the
354 - # required shape is (3 * num_heads * head_size).
355 - # Thus, we need weight conversion.
356 - output_dim = getattr(param, "output_dim", None)
357 - num_heads = self.config.num_attention_heads
358 - if output_dim is not None:
359 - loaded_weight_shape = loaded_weight.shape
360 - loaded_weight = loaded_weight.view(
361 - loaded_weight_shape[:output_dim] + (num_heads, 3, -1) +
362 - loaded_weight_shape[output_dim + 1:])
363 - loaded_weight = loaded_weight.transpose(
364 - output_dim, output_dim + 1)
365 - loaded_weight = loaded_weight.reshape(loaded_weight_shape)
352 + # prepare weight iterators for components
353 + weights_group = group_weights_with_prefix(weights)
354
355 + # load vision embeddings
356 + vision_params_dict = dict(self.vision_embed_tokens.named_parameters())
357 + for name, loaded_weight in weights_group["vision_embed_tokens"]:
358 + param = vision_params_dict[name]
359 weight_loader = getattr(param, "weight_loader",
360 default_weight_loader)
361 weight_loader(param, loaded_weight)
362 +
363 + # load llm backbone
364 + self.language_model.load_weights(weights_group["language_model"])
vllm/model_executor/models/gemma.py
+42 -21
@@ -15,7 +15,7 @@
15 # limitations under the License.
16 """Inference-only Gemma model compatible with HuggingFace weights."""
17 from functools import lru_cache
18 -from typing import Iterable, List, Optional, Set, Tuple
18 +from typing import Iterable, List, Optional, Set, Tuple, Union
19
20 import torch
21 from torch import nn
@@ -23,7 +23,7 @@ from transformers import GemmaConfig
23
24 from vllm.attention import Attention, AttentionMetadata
25 from vllm.config import CacheConfig, LoRAConfig
26 -from vllm.distributed import get_tensor_model_parallel_world_size
26 +from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
27 from vllm.logger import init_logger
28 from vllm.model_executor.layers.activation import GeluAndMul
29 from vllm.model_executor.layers.layernorm import GemmaRMSNorm
@@ -31,8 +31,7 @@ from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
31 QKVParallelLinear,
32 RowParallelLinear)
33 from vllm.model_executor.layers.logits_processor import LogitsProcessor
34 -from vllm.model_executor.layers.quantization.base_config import (
35 - QuantizationConfig)
34 +from vllm.model_executor.layers.quantization import QuantizationConfig
35 from vllm.model_executor.layers.rotary_embedding import get_rope
36 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
37 from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -41,7 +40,9 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
40 from vllm.model_executor.sampling_metadata import SamplingMetadata
41 from vllm.sequence import IntermediateTensors
42
44 -from .interfaces import SupportsLoRA
43 +from .interfaces import SupportsLoRA, SupportsPP
44 +from .utils import (is_pp_missing_parameter,
45 + make_empty_intermediate_tensors_factory, make_layers)
46
47 logger = init_logger(__name__)
48
@@ -245,6 +246,7 @@ class GemmaModel(nn.Module):
246 config: GemmaConfig,
247 cache_config: Optional[CacheConfig] = None,
248 quant_config: Optional[QuantizationConfig] = None,
249 + prefix: str = "",
250 ) -> None:
251 super().__init__()
252 self.config = config
@@ -253,10 +255,11 @@ class GemmaModel(nn.Module):
255 config.vocab_size,
256 config.hidden_size,
257 )
256 - self.layers = nn.ModuleList([
257 - GemmaDecoderLayer(config, cache_config, quant_config)
258 - for _ in range(config.num_hidden_layers)
259 - ])
258 + self.start_layer, self.end_layer, self.layers = make_layers(
259 + config.num_hidden_layers,
260 + lambda prefix: GemmaDecoderLayer(config, cache_config, quant_config
261 + ),
262 + prefix=f"{prefix}.layers")
263 self.norm = GemmaRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
264
265 # Normalize the embedding by sqrt(hidden_size)
@@ -265,6 +268,9 @@ class GemmaModel(nn.Module):
268 # See https://github.com/huggingface/transformers/pull/29402
269 normalizer = self.config.hidden_size**0.5
270 self.register_buffer("normalizer", torch.tensor(normalizer))
271 + self.make_empty_intermediate_tensors = (
272 + make_empty_intermediate_tensors_factory(
273 + ["hidden_states", "residual"], config.hidden_size))
274
275 def get_input_embeddings(self, input_ids: torch.Tensor) -> torch.Tensor:
276 return self.embed_tokens(input_ids)
@@ -275,29 +281,38 @@ class GemmaModel(nn.Module):
281 positions: torch.Tensor,
282 kv_caches: List[torch.Tensor],
283 attn_metadata: AttentionMetadata,
278 - intermediate_tensors: Optional[IntermediateTensors] = None,
284 + intermediate_tensors: Optional[IntermediateTensors],
285 inputs_embeds: Optional[torch.Tensor] = None,
280 - ) -> torch.Tensor:
281 - if inputs_embeds is not None:
282 - hidden_states = inputs_embeds
286 + ) -> Union[torch.Tensor, IntermediateTensors]:
287 + if get_pp_group().is_first_rank:
288 + if inputs_embeds is not None:
289 + hidden_states = inputs_embeds
290 + else:
291 + hidden_states = self.get_input_embeddings(input_ids)
292 + hidden_states *= self.normalizer
293 + residual = None
294 else:
284 - hidden_states = self.get_input_embeddings(input_ids)
285 - hidden_states *= self.normalizer
286 - residual = None
287 - for i in range(len(self.layers)):
295 + hidden_states = intermediate_tensors["hidden_states"]
296 + residual = intermediate_tensors["residual"]
297 + for i in range(self.start_layer, self.end_layer):
298 layer = self.layers[i]
299 hidden_states, residual = layer(
300 positions,
301 hidden_states,
292 - kv_caches[i],
302 + kv_caches[i - self.start_layer],
303 attn_metadata,
304 residual,
305 )
306 + if not get_pp_group().is_last_rank:
307 + return IntermediateTensors({
308 + "hidden_states": hidden_states,
309 + "residual": residual
310 + })
311 hidden_states, _ = self.norm(hidden_states, residual)
312 return hidden_states
313
314
300 -class GemmaForCausalLM(nn.Module, SupportsLoRA):
315 +class GemmaForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
316 packed_modules_mapping = {
317 "qkv_proj": [
318 "q_proj",
@@ -339,6 +354,8 @@ class GemmaForCausalLM(nn.Module, SupportsLoRA):
354 self.model = GemmaModel(config, cache_config, quant_config)
355 self.logits_processor = LogitsProcessor(config.vocab_size)
356 self.sampler = Sampler()
357 + self.make_empty_intermediate_tensors = (
358 + self.model.make_empty_intermediate_tensors)
359
360 def forward(
361 self,
@@ -347,9 +364,9 @@ class GemmaForCausalLM(nn.Module, SupportsLoRA):
364 kv_caches: List[torch.Tensor],
365 attn_metadata: AttentionMetadata,
366 intermediate_tensors: Optional[IntermediateTensors] = None,
350 - ) -> torch.Tensor:
367 + ) -> Union[torch.Tensor, IntermediateTensors]:
368 hidden_states = self.model(input_ids, positions, kv_caches,
352 - attn_metadata)
369 + attn_metadata, intermediate_tensors)
370 return hidden_states
371
372 def compute_logits(
@@ -388,6 +405,8 @@ class GemmaForCausalLM(nn.Module, SupportsLoRA):
405 # Skip loading extra bias for GPTQ models.
406 if name.endswith(".bias") and name not in params_dict:
407 continue
408 + if is_pp_missing_parameter(name, self):
409 + continue
410 param = params_dict[name]
411 weight_loader = param.weight_loader
412 weight_loader(param, loaded_weight, shard_id)
@@ -400,6 +419,8 @@ class GemmaForCausalLM(nn.Module, SupportsLoRA):
419 # Skip loading extra bias for GPTQ models.
420 if name.endswith(".bias") and name not in params_dict:
421 continue
422 + if is_pp_missing_parameter(name, self):
423 + continue
424 param = params_dict[name]
425 weight_loader = getattr(param, "weight_loader",
426 default_weight_loader)
vllm/model_executor/models/gemma2.py
+41 -18
@@ -14,7 +14,7 @@
14 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15 # See the License for the specific language governing permissions and
16 # limitations under the License.
17 -from typing import Iterable, List, Optional, Set, Tuple
17 +from typing import Iterable, List, Optional, Set, Tuple, Union
18
19 import torch
20 from torch import nn
@@ -22,7 +22,7 @@ from transformers import Gemma2Config
22
23 from vllm.attention import Attention, AttentionMetadata
24 from vllm.config import CacheConfig, LoRAConfig
25 -from vllm.distributed import get_tensor_model_parallel_world_size
25 +from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
26 from vllm.logger import init_logger
27 from vllm.model_executor.layers.activation import GeluAndMul
28 from vllm.model_executor.layers.layernorm import GemmaRMSNorm
@@ -30,8 +30,7 @@ from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
30 QKVParallelLinear,
31 RowParallelLinear)
32 from vllm.model_executor.layers.logits_processor import LogitsProcessor
33 -from vllm.model_executor.layers.quantization.base_config import (
34 - QuantizationConfig)
33 +from vllm.model_executor.layers.quantization import QuantizationConfig
34 from vllm.model_executor.layers.rotary_embedding import get_rope
35 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
36 from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -40,7 +39,9 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
39 from vllm.model_executor.sampling_metadata import SamplingMetadata
40 from vllm.sequence import IntermediateTensors
41
43 -from .interfaces import SupportsLoRA
42 +from .interfaces import SupportsLoRA, SupportsPP
43 +from .utils import (is_pp_missing_parameter,
44 + make_empty_intermediate_tensors_factory, make_layers)
45
46 logger = init_logger(__name__)
47
@@ -244,6 +245,7 @@ class Gemma2Model(nn.Module):
245 config: Gemma2Config,
246 cache_config: Optional[CacheConfig] = None,
247 quant_config: Optional[QuantizationConfig] = None,
248 + prefix: str = "",
249 ) -> None:
250 super().__init__()
251 self.config = config
@@ -252,10 +254,11 @@ class Gemma2Model(nn.Module):
254 config.vocab_size,
255 config.hidden_size,
256 )
255 - self.layers = nn.ModuleList([
256 - Gemma2DecoderLayer(layer_idx, config, cache_config, quant_config)
257 - for layer_idx in range(config.num_hidden_layers)
258 - ])
257 + self.start_layer, self.end_layer, self.layers = make_layers(
258 + config.num_hidden_layers,
259 + lambda prefix: Gemma2DecoderLayer(int(prefix.split(".")[
260 + -1]), config, cache_config, quant_config),
261 + prefix=f"{prefix}.layers")
262 self.norm = GemmaRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
263
264 # Normalize the embedding by sqrt(hidden_size)
@@ -264,6 +267,9 @@ class Gemma2Model(nn.Module):
267 # See https://github.com/huggingface/transformers/pull/29402
268 normalizer = self.config.hidden_size**0.5
269 self.register_buffer("normalizer", torch.tensor(normalizer))
270 + self.make_empty_intermediate_tensors = (
271 + make_empty_intermediate_tensors_factory(
272 + ["hidden_states", "residual"], config.hidden_size))
273
274 def forward(
275 self,
@@ -271,25 +277,36 @@ class Gemma2Model(nn.Module):
277 positions: torch.Tensor,
278 kv_caches: List[torch.Tensor],
279 attn_metadata: AttentionMetadata,
274 - ) -> torch.Tensor:
275 - hidden_states = self.embed_tokens(input_ids)
276 - hidden_states *= self.normalizer
280 + intermediate_tensors: Optional[IntermediateTensors],
281 + ) -> Union[torch.Tensor, IntermediateTensors]:
282 + if get_pp_group().is_first_rank:
283 + hidden_states = self.embed_tokens(input_ids)
284 + hidden_states *= self.normalizer
285
278 - residual = None
279 - for i in range(len(self.layers)):
286 + residual = None
287 + else:
288 + assert intermediate_tensors is not None
289 + hidden_states = intermediate_tensors["hidden_states"]
290 + residual = intermediate_tensors["residual"]
291 + for i in range(self.start_layer, self.end_layer):
292 layer = self.layers[i]
293 hidden_states, residual = layer(
294 positions,
295 hidden_states,
284 - kv_caches[i],
296 + kv_caches[i - self.start_layer],
297 attn_metadata,
298 residual,
299 )
300 + if not get_pp_group().is_last_rank:
301 + return IntermediateTensors({
302 + "hidden_states": hidden_states,
303 + "residual": residual
304 + })
305 hidden_states, _ = self.norm(hidden_states, residual)
306 return hidden_states
307
308
292 -class Gemma2ForCausalLM(nn.Module, SupportsLoRA):
309 +class Gemma2ForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
310 packed_modules_mapping = {
311 "qkv_proj": [
312 "q_proj",
@@ -338,6 +355,8 @@ class Gemma2ForCausalLM(nn.Module, SupportsLoRA):
355 self.logits_processor = LogitsProcessor(
356 config.vocab_size, soft_cap=config.final_logit_softcapping)
357 self.sampler = Sampler()
358 + self.make_empty_intermediate_tensors = (
359 + self.model.make_empty_intermediate_tensors)
360
361 def forward(
362 self,
@@ -346,9 +365,9 @@ class Gemma2ForCausalLM(nn.Module, SupportsLoRA):
365 kv_caches: List[torch.Tensor],
366 attn_metadata: AttentionMetadata,
367 intermediate_tensors: Optional[IntermediateTensors] = None,
349 - ) -> torch.Tensor:
368 + ) -> Union[torch.Tensor, IntermediateTensors]:
369 hidden_states = self.model(input_ids, positions, kv_caches,
351 - attn_metadata)
370 + attn_metadata, intermediate_tensors)
371 return hidden_states
372
373 def compute_logits(
@@ -387,6 +406,8 @@ class Gemma2ForCausalLM(nn.Module, SupportsLoRA):
406 # Skip loading extra bias for GPTQ models.
407 if name.endswith(".bias") and name not in params_dict:
408 continue
409 + if is_pp_missing_parameter(name, self):
410 + continue
411 param = params_dict[name]
412 weight_loader = param.weight_loader
413 weight_loader(param, loaded_weight, shard_id)
@@ -399,6 +420,8 @@ class Gemma2ForCausalLM(nn.Module, SupportsLoRA):
420 # Skip loading extra bias for GPTQ models.
421 if name.endswith(".bias") and name not in params_dict:
422 continue
423 + if is_pp_missing_parameter(name, self):
424 + continue
425 param = params_dict[name]
426 weight_loader = getattr(param, "weight_loader",
427 default_weight_loader)
vllm/model_executor/models/gpt2.py
+11 -15
@@ -32,8 +32,7 @@ from vllm.model_executor.layers.linear import (ColumnParallelLinear,
32 QKVParallelLinear,
33 RowParallelLinear)
34 from vllm.model_executor.layers.logits_processor import LogitsProcessor
35 -from vllm.model_executor.layers.quantization.base_config import (
36 - QuantizationConfig)
35 +from vllm.model_executor.layers.quantization import QuantizationConfig
36 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
37 from vllm.model_executor.layers.vocab_parallel_embedding import (
38 ParallelLMHead, VocabParallelEmbedding)
@@ -41,7 +40,9 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
40 from vllm.model_executor.sampling_metadata import SamplingMetadata
41 from vllm.sequence import IntermediateTensors
42
44 -from .utils import is_pp_missing_parameter, make_layers
43 +from .interfaces import SupportsPP
44 +from .utils import (is_pp_missing_parameter,
45 + make_empty_intermediate_tensors_factory, make_layers)
46
47
48 class GPT2Attention(nn.Module):
@@ -204,6 +205,9 @@ class GPT2Model(nn.Module):
205 config, cache_config, quant_config, prefix=prefix),
206 prefix=f"{prefix}.h")
207 self.ln_f = nn.LayerNorm(self.embed_dim, eps=config.layer_norm_epsilon)
208 + self.make_empty_intermediate_tensors = (
209 + make_empty_intermediate_tensors_factory(["hidden_states"],
210 + config.n_embd))
211
212 def forward(
213 self,
@@ -234,7 +238,7 @@ class GPT2Model(nn.Module):
238 return hidden_states
239
240
237 -class GPT2LMHeadModel(nn.Module):
241 +class GPT2LMHeadModel(nn.Module, SupportsPP):
242
243 def __init__(
244 self,
@@ -256,6 +260,8 @@ class GPT2LMHeadModel(nn.Module):
260 self.config.hidden_size)
261 self.logits_processor = LogitsProcessor(config.vocab_size)
262 self.sampler = Sampler()
263 + self.make_empty_intermediate_tensors = (
264 + self.transformer.make_empty_intermediate_tensors)
265
266 def forward(
267 self,
@@ -264,7 +270,7 @@ class GPT2LMHeadModel(nn.Module):
270 kv_caches: List[torch.Tensor],
271 attn_metadata: AttentionMetadata,
272 intermediate_tensors: Optional[IntermediateTensors] = None,
267 - ) -> torch.Tensor:
273 + ) -> Union[torch.Tensor, IntermediateTensors]:
274 hidden_states = self.transformer(input_ids, positions, kv_caches,
275 attn_metadata, intermediate_tensors)
276 return hidden_states
@@ -286,16 +292,6 @@ class GPT2LMHeadModel(nn.Module):
292 next_tokens = self.sampler(logits, sampling_metadata)
293 return next_tokens
294
289 - def make_empty_intermediate_tensors(
290 - self, batch_size: int, dtype: torch.dtype,
291 - device: torch.device) -> IntermediateTensors:
292 - return IntermediateTensors({
293 - "hidden_states":
294 - torch.zeros((batch_size, self.config.hidden_size),
295 - dtype=dtype,
296 - device=device),
297 - })
298 -
295 def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
296 params_dict = dict(self.named_parameters(remove_duplicate=False))
297 for name, loaded_weight in weights:
vllm/model_executor/models/gpt_bigcode.py
+36 -18
@@ -18,7 +18,7 @@
18 # See the License for the specific language governing permissions and
19 # limitations under the License.
20 """Inference-only GPTBigCode model compatible with HuggingFace weights."""
21 -from typing import Iterable, List, Optional, Tuple
21 +from typing import Iterable, List, Optional, Tuple, Union
22
23 import torch
24 from torch import nn
@@ -26,14 +26,13 @@ from transformers import GPTBigCodeConfig
26
27 from vllm.attention import Attention, AttentionMetadata
28 from vllm.config import CacheConfig, LoRAConfig
29 -from vllm.distributed import get_tensor_model_parallel_world_size
29 +from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
30 from vllm.model_executor.layers.activation import get_act_fn
31 from vllm.model_executor.layers.linear import (ColumnParallelLinear,
32 QKVParallelLinear,
33 RowParallelLinear)
34 from vllm.model_executor.layers.logits_processor import LogitsProcessor
35 -from vllm.model_executor.layers.quantization.base_config import (
36 - QuantizationConfig)
35 +from vllm.model_executor.layers.quantization import QuantizationConfig
36 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
37 from vllm.model_executor.layers.vocab_parallel_embedding import (
38 ParallelLMHead, VocabParallelEmbedding)
@@ -41,7 +40,9 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
40 from vllm.model_executor.sampling_metadata import SamplingMetadata
41 from vllm.sequence import IntermediateTensors
42
44 -from .interfaces import SupportsLoRA
43 +from .interfaces import SupportsLoRA, SupportsPP
44 +from .utils import (is_pp_missing_parameter,
45 + make_empty_intermediate_tensors_factory, make_layers)
46
47
48 class GPTBigCodeAttention(nn.Module):
@@ -194,6 +195,7 @@ class GPTBigCodeModel(nn.Module):
195 cache_config: Optional[CacheConfig] = None,
196 quant_config: Optional[QuantizationConfig] = None,
197 lora_config: Optional[LoRAConfig] = None,
198 + prefix: str = "",
199 ):
200 super().__init__()
201 self.config = config
@@ -207,11 +209,15 @@ class GPTBigCodeModel(nn.Module):
209 self.embed_dim,
210 org_num_embeddings=config.vocab_size)
211 self.wpe = nn.Embedding(config.max_position_embeddings, self.embed_dim)
210 - self.h = nn.ModuleList([
211 - GPTBigCodeBlock(config, cache_config, quant_config)
212 - for _ in range(config.num_hidden_layers)
213 - ])
212 + self.start_layer, self.end_layer, self.h = make_layers(
213 + config.num_hidden_layers,
214 + lambda prefix: GPTBigCodeBlock(config, cache_config, quant_config),
215 + prefix=f"{prefix}.h",
216 + )
217 self.ln_f = nn.LayerNorm(self.embed_dim, eps=config.layer_norm_epsilon)
218 + self.make_empty_intermediate_tensors = (
219 + make_empty_intermediate_tensors_factory(["hidden_states"],
220 + config.n_embd))
221
222 def forward(
223 self,
@@ -219,20 +225,28 @@ class GPTBigCodeModel(nn.Module):
225 position_ids: torch.Tensor,
226 kv_caches: List[torch.Tensor],
227 attn_metadata: AttentionMetadata,
222 - ) -> torch.Tensor:
223 - inputs_embeds = self.wte(input_ids)
224 - position_embeds = self.wpe(position_ids)
225 - hidden_states = inputs_embeds + position_embeds
228 + intermediate_tensors: Optional[IntermediateTensors],
229 + ) -> Union[torch.Tensor, IntermediateTensors]:
230 + if get_pp_group().is_first_rank:
231 + inputs_embeds = self.wte(input_ids)
232 + position_embeds = self.wpe(position_ids)
233 + hidden_states = inputs_embeds + position_embeds
234 + else:
235 + hidden_states = intermediate_tensors["hidden_states"]
236
227 - for i in range(len(self.h)):
237 + for i in range(self.start_layer, self.end_layer):
238 layer = self.h[i]
229 - hidden_states = layer(hidden_states, kv_caches[i], attn_metadata)
239 + hidden_states = layer(hidden_states,
240 + kv_caches[i - self.start_layer],
241 + attn_metadata)
242
243 + if not get_pp_group().is_last_rank:
244 + return IntermediateTensors({"hidden_states": hidden_states})
245 hidden_states = self.ln_f(hidden_states)
246 return hidden_states
247
248
235 -class GPTBigCodeForCausalLM(nn.Module, SupportsLoRA):
249 +class GPTBigCodeForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
250 packed_modules_mapping = {"c_attn": ["c_attn"]}
251
252 supported_lora_modules = ["c_fc", "c_proj", "wte", "c_attn"]
@@ -272,6 +286,8 @@ class GPTBigCodeForCausalLM(nn.Module, SupportsLoRA):
286 self.logits_processor = LogitsProcessor(self.unpadded_vocab_size,
287 config.vocab_size)
288 self.sampler = Sampler()
289 + self.make_empty_intermediate_tensors = (
290 + self.transformer.make_empty_intermediate_tensors)
291
292 def forward(
293 self,
@@ -280,9 +296,9 @@ class GPTBigCodeForCausalLM(nn.Module, SupportsLoRA):
296 kv_caches: List[torch.Tensor],
297 attn_metadata: AttentionMetadata,
298 intermediate_tensors: Optional[IntermediateTensors] = None,
283 - ) -> torch.Tensor:
299 + ) -> Union[torch.Tensor, IntermediateTensors]:
300 hidden_states = self.transformer(input_ids, positions, kv_caches,
285 - attn_metadata)
301 + attn_metadata, intermediate_tensors)
302 return hidden_states
303
304 def compute_logits(
@@ -311,6 +327,8 @@ class GPTBigCodeForCausalLM(nn.Module, SupportsLoRA):
327 # Skip attention mask.
328 # NOTE: "c_attn.bias" should not be skipped.
329 continue
330 + if is_pp_missing_parameter(name, self):
331 + continue
332 param = params_dict[name]
333 weight_loader = getattr(param, "weight_loader",
334 default_weight_loader)
vllm/model_executor/models/gpt_j.py
+35 -15
@@ -16,7 +16,7 @@
16 # See the License for the specific language governing permissions and
17 # limitations under the License.
18 """Inference-only GPT-J model compatible with HuggingFace weights."""
19 -from typing import Iterable, List, Optional, Tuple
19 +from typing import Iterable, List, Optional, Tuple, Union
20
21 import torch
22 from torch import nn
@@ -24,14 +24,13 @@ from transformers import GPTJConfig
24
25 from vllm.attention import Attention, AttentionMetadata
26 from vllm.config import CacheConfig
27 -from vllm.distributed import get_tensor_model_parallel_world_size
27 +from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
28 from vllm.model_executor.layers.activation import get_act_fn
29 from vllm.model_executor.layers.linear import (ColumnParallelLinear,
30 QKVParallelLinear,
31 RowParallelLinear)
32 from vllm.model_executor.layers.logits_processor import LogitsProcessor
33 -from vllm.model_executor.layers.quantization.base_config import (
34 - QuantizationConfig)
33 +from vllm.model_executor.layers.quantization import QuantizationConfig
34 from vllm.model_executor.layers.rotary_embedding import get_rope
35 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
36 from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -40,6 +39,10 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
39 from vllm.model_executor.sampling_metadata import SamplingMetadata
40 from vllm.sequence import IntermediateTensors
41
42 +from .interfaces import SupportsPP
43 +from .utils import (is_pp_missing_parameter,
44 + make_empty_intermediate_tensors_factory, make_layers)
45 +
46
47 class GPTJAttention(nn.Module):
48
@@ -178,6 +181,7 @@ class GPTJModel(nn.Module):
181 config: GPTJConfig,
182 cache_config: Optional[CacheConfig] = None,
183 quant_config: Optional[QuantizationConfig] = None,
184 + prefix: str = "",
185 ):
186 super().__init__()
187 self.config = config
@@ -186,11 +190,15 @@ class GPTJModel(nn.Module):
190 config.vocab_size,
191 self.embed_dim,
192 )
189 - self.h = nn.ModuleList([
190 - GPTJBlock(config, cache_config, quant_config)
191 - for _ in range(config.n_layer)
192 - ])
193 + self.start_layer, self.end_layer, self.h = make_layers(
194 + config.n_layer,
195 + lambda prefix: GPTJBlock(config, cache_config, quant_config),
196 + prefix=f"{prefix}.h",
197 + )
198 self.ln_f = nn.LayerNorm(self.embed_dim, eps=config.layer_norm_epsilon)
199 + self.make_empty_intermediate_tensors = (
200 + make_empty_intermediate_tensors_factory(["hidden_states"],
201 + config.n_embd))
202
203 def forward(
204 self,
@@ -198,21 +206,27 @@ class GPTJModel(nn.Module):
206 position_ids: torch.Tensor,
207 kv_caches: List[torch.Tensor],
208 attn_metadata: AttentionMetadata,
201 - ) -> torch.Tensor:
202 - hidden_states = self.wte(input_ids)
203 - for i in range(len(self.h)):
209 + intermediate_tensors: Optional[IntermediateTensors],
210 + ) -> Union[torch.Tensor, IntermediateTensors]:
211 + if get_pp_group().is_first_rank:
212 + hidden_states = self.wte(input_ids)
213 + else:
214 + hidden_states = intermediate_tensors["hidden_states"]
215 + for i in range(self.start_layer, self.end_layer):
216 layer = self.h[i]
217 hidden_states = layer(
218 position_ids,
219 hidden_states,
208 - kv_caches[i],
220 + kv_caches[i - self.start_layer],
221 attn_metadata,
222 )
223 + if not get_pp_group().is_last_rank:
224 + return IntermediateTensors({"hidden_states": hidden_states})
225 hidden_states = self.ln_f(hidden_states)
226 return hidden_states
227
228
215 -class GPTJForCausalLM(nn.Module):
229 +class GPTJForCausalLM(nn.Module, SupportsPP):
230
231 def __init__(
232 self,
@@ -233,6 +247,8 @@ class GPTJForCausalLM(nn.Module):
247 )
248 self.logits_processor = LogitsProcessor(config.vocab_size)
249 self.sampler = Sampler()
250 + self.make_empty_intermediate_tensors = (
251 + self.transformer.make_empty_intermediate_tensors)
252
253 def forward(
254 self,
@@ -241,9 +257,9 @@ class GPTJForCausalLM(nn.Module):
257 kv_caches: List[torch.Tensor],
258 attn_metadata: AttentionMetadata,
259 intermediate_tensors: Optional[IntermediateTensors] = None,
244 - ) -> torch.Tensor:
260 + ) -> Union[torch.Tensor, IntermediateTensors]:
261 hidden_states = self.transformer(input_ids, positions, kv_caches,
246 - attn_metadata)
262 + attn_metadata, intermediate_tensors)
263 return hidden_states
264
265 def compute_logits(
@@ -283,6 +299,8 @@ class GPTJForCausalLM(nn.Module):
299 # Skip loading extra bias for GPTQ models.
300 if name.endswith(".bias") and name not in params_dict:
301 continue
302 + if is_pp_missing_parameter(name, self):
303 + continue
304 param = params_dict[name]
305 weight_loader = param.weight_loader
306 weight_loader(param, loaded_weight, shard_id)
@@ -291,6 +309,8 @@ class GPTJForCausalLM(nn.Module):
309 # Skip loading extra bias for GPTQ models.
310 if name.endswith(".bias") and name not in params_dict:
311 continue
312 + if is_pp_missing_parameter(name, self):
313 + continue
314 param = params_dict[name]
315 weight_loader = getattr(param, "weight_loader",
316 default_weight_loader)
vllm/model_executor/models/gpt_neox.py
+33 -15
@@ -16,7 +16,7 @@
16 # See the License for the specific language governing permissions and
17 # limitations under the License.
18 """Inference-only GPT-NeoX model compatible with HuggingFace weights."""
19 -from typing import Iterable, List, Optional, Tuple
19 +from typing import Iterable, List, Optional, Tuple, Union
20
21 import torch
22 from torch import nn
@@ -24,14 +24,13 @@ from transformers import GPTNeoXConfig
24
25 from vllm.attention import Attention, AttentionMetadata
26 from vllm.config import CacheConfig
27 -from vllm.distributed import get_tensor_model_parallel_world_size
27 +from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
28 from vllm.model_executor.layers.activation import get_act_fn
29 from vllm.model_executor.layers.linear import (ColumnParallelLinear,
30 QKVParallelLinear,
31 RowParallelLinear)
32 from vllm.model_executor.layers.logits_processor import LogitsProcessor
33 -from vllm.model_executor.layers.quantization.base_config import (
34 - QuantizationConfig)
33 +from vllm.model_executor.layers.quantization import QuantizationConfig
34 from vllm.model_executor.layers.rotary_embedding import get_rope
35 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
36 from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -40,6 +39,10 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
39 from vllm.model_executor.sampling_metadata import SamplingMetadata
40 from vllm.sequence import IntermediateTensors
41
42 +from .interfaces import SupportsPP
43 +from .utils import (is_pp_missing_parameter,
44 + make_empty_intermediate_tensors_factory, make_layers)
45 +
46
47 class GPTNeoXAttention(nn.Module):
48
@@ -191,6 +194,7 @@ class GPTNeoXModel(nn.Module):
194 config: GPTNeoXConfig,
195 cache_config: Optional[CacheConfig] = None,
196 quant_config: Optional[QuantizationConfig] = None,
197 + prefix: str = "",
198 ):
199 super().__init__()
200 self.config = config
@@ -199,12 +203,16 @@ class GPTNeoXModel(nn.Module):
203 config.vocab_size,
204 config.hidden_size,
205 )
202 - self.layers = nn.ModuleList([
203 - GPTNeoXLayer(config, cache_config, quant_config)
204 - for _ in range(config.num_hidden_layers)
205 - ])
206 + self.start_layer, self.end_layer, self.layers = make_layers(
207 + config.num_hidden_layers,
208 + lambda prefix: GPTNeoXLayer(config, cache_config, quant_config),
209 + prefix=f"{prefix}.layers",
210 + )
211 self.final_layer_norm = nn.LayerNorm(config.hidden_size,
212 eps=config.layer_norm_eps)
213 + self.make_empty_intermediate_tensors = (
214 + make_empty_intermediate_tensors_factory(["hidden_states"],
215 + config.hidden_size))
216
217 def forward(
218 self,
@@ -212,21 +220,27 @@ class GPTNeoXModel(nn.Module):
220 position_ids: torch.Tensor,
221 kv_caches: List[torch.Tensor],
222 attn_metadata: AttentionMetadata,
215 - ) -> torch.Tensor:
216 - hidden_states = self.embed_in(input_ids)
217 - for i in range(len(self.layers)):
223 + intermediate_tensors: Optional[IntermediateTensors],
224 + ) -> Union[torch.Tensor, IntermediateTensors]:
225 + if get_pp_group().is_first_rank:
226 + hidden_states = self.embed_in(input_ids)
227 + else:
228 + hidden_states = intermediate_tensors["hidden_states"]
229 + for i in range(self.start_layer, self.end_layer):
230 layer = self.layers[i]
231 hidden_states = layer(
232 position_ids,
233 hidden_states,
222 - kv_caches[i],
234 + kv_caches[i - self.start_layer],
235 attn_metadata,
236 )
237 + if not get_pp_group().is_last_rank:
238 + return IntermediateTensors({"hidden_states": hidden_states})
239 hidden_states = self.final_layer_norm(hidden_states)
240 return hidden_states
241
242
229 -class GPTNeoXForCausalLM(nn.Module):
243 +class GPTNeoXForCausalLM(nn.Module, SupportsPP):
244
245 def __init__(
246 self,
@@ -247,6 +261,8 @@ class GPTNeoXForCausalLM(nn.Module):
261 self.embed_out.weight = self.gpt_neox.embed_in.weight
262 self.logits_processor = LogitsProcessor(config.vocab_size)
263 self.sampler = Sampler()
264 + self.make_empty_intermediate_tensors = (
265 + self.gpt_neox.make_empty_intermediate_tensors)
266
267 def forward(
268 self,
@@ -255,9 +271,9 @@ class GPTNeoXForCausalLM(nn.Module):
271 kv_caches: List[torch.Tensor],
272 attn_metadata: AttentionMetadata,
273 intermediate_tensors: Optional[IntermediateTensors] = None,
258 - ) -> torch.Tensor:
274 + ) -> Union[torch.Tensor, IntermediateTensors]:
275 hidden_states = self.gpt_neox(input_ids, positions, kv_caches,
260 - attn_metadata)
276 + attn_metadata, intermediate_tensors)
277 return hidden_states
278
279 def compute_logits(
@@ -288,6 +304,8 @@ class GPTNeoXForCausalLM(nn.Module):
304 # Models trained using OpenRLHF may include
305 # these tensors in the checkpoint. Skip them.
306 continue
307 + if is_pp_missing_parameter(name, self):
308 + continue
309 param = params_dict[name]
310
311 if "query_key_value" in name:
vllm/model_executor/models/granite.py
+4 -4
@@ -51,7 +51,7 @@ from vllm.model_executor.sampling_metadata import SamplingMetadata
51 from vllm.sequence import IntermediateTensors
52 from vllm.utils import is_hip
53
54 -from .interfaces import SupportsLoRA
54 +from .interfaces import SupportsLoRA, SupportsPP
55 from .utils import PPMissingLayer, is_pp_missing_parameter, make_layers
56
57
@@ -311,13 +311,13 @@ class GraniteModel(nn.Module):
311 else:
312 hidden_states = self.get_input_embeddings(input_ids)
313 residual = None
314 +
315 + hidden_states *= self.config.embedding_multiplier
316 else:
317 assert intermediate_tensors is not None
318 hidden_states = intermediate_tensors["hidden_states"]
319 residual = intermediate_tensors["residual"]
320
319 - hidden_states *= self.config.embedding_multiplier
320 -
321 for i in range(self.start_layer, self.end_layer):
322 layer = self.layers[i]
323 hidden_states = layer(
@@ -337,7 +337,7 @@ class GraniteModel(nn.Module):
337 return hidden_states
338
339
340 -class GraniteForCausalLM(nn.Module, SupportsLoRA):
340 +class GraniteForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
341 packed_modules_mapping = {
342 "qkv_proj": [
343 "q_proj",
vllm/model_executor/models/granitemoe.py
+2 -2
@@ -46,7 +46,7 @@ from vllm.model_executor.sampling_metadata import SamplingMetadata
46 from vllm.sequence import IntermediateTensors
47
48 from . import mixtral
49 -from .interfaces import SupportsLoRA
49 +from .interfaces import SupportsLoRA, SupportsPP
50 from .utils import make_layers
51
52
@@ -307,7 +307,7 @@ class GraniteMoeModel(nn.Module):
307 return hidden_states
308
309
310 -class GraniteMoeForCausalLM(nn.Module, SupportsLoRA):
310 +class GraniteMoeForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
311 fall_back_to_pt_during_load = False
312
313 packed_modules_mapping = {
vllm/model_executor/models/interfaces.py
+141 -9
@@ -1,11 +1,17 @@
1 -from typing import (ClassVar, Dict, List, Literal, Optional, Protocol, Type,
2 - Union, overload, runtime_checkable)
1 +import inspect
2 +from typing import (TYPE_CHECKING, ClassVar, Dict, List, Literal, Optional,
3 + Protocol, Type, Union, overload, runtime_checkable)
4
5 +import torch
6 from typing_extensions import TypeIs
7
6 -from vllm.config import LoRAConfig, MultiModalConfig, SchedulerConfig
8 from vllm.logger import init_logger
9
10 +if TYPE_CHECKING:
11 + from vllm.attention import AttentionMetadata
12 + from vllm.config import LoRAConfig, MultiModalConfig, SchedulerConfig
13 + from vllm.sequence import IntermediateTensors
14 +
15 logger = init_logger(__name__)
16
17
@@ -22,7 +28,7 @@ class SupportsMultiModal(Protocol):
28 MRO of your model class.
29 """
30
25 - def __init__(self, *, multimodal_config: MultiModalConfig) -> None:
31 + def __init__(self, *, multimodal_config: "MultiModalConfig") -> None:
32 ...
33
34
@@ -32,7 +38,7 @@ class SupportsMultiModal(Protocol):
38 class _SupportsMultiModalType(Protocol):
39 supports_multimodal: Literal[True]
40
35 - def __call__(self, *, multimodal_config: MultiModalConfig) -> None:
41 + def __call__(self, *, multimodal_config: "MultiModalConfig") -> None:
42 ...
43
44
@@ -75,7 +81,7 @@ class SupportsLoRA(Protocol):
81 embedding_padding_modules: ClassVar[List[str]]
82
83 # lora_config is None when LoRA is not enabled
78 - def __init__(self, *, lora_config: Optional[LoRAConfig] = None) -> None:
84 + def __init__(self, *, lora_config: Optional["LoRAConfig"] = None) -> None:
85 ...
86
87
@@ -90,7 +96,7 @@ class _SupportsLoRAType(Protocol):
96 embedding_modules: Dict[str, str]
97 embedding_padding_modules: List[str]
98
93 - def __call__(self, *, lora_config: Optional[LoRAConfig] = None) -> None:
99 + def __call__(self, *, lora_config: Optional["LoRAConfig"] = None) -> None:
100 ...
101
102
@@ -145,6 +151,132 @@ def _supports_lora(
151 return isinstance(model, SupportsLoRA)
152
153
154 +@runtime_checkable
155 +class SupportsPP(Protocol):
156 + """The interface required for all models that support pipeline parallel."""
157 +
158 + supports_pp: ClassVar[Literal[True]] = True
159 + """
160 + A flag that indicates this model supports pipeline parallel.
161 +
162 + Note:
163 + There is no need to redefine this flag if this class is in the
164 + MRO of your model class.
165 + """
166 +
167 + def make_empty_intermediate_tensors(
168 + self,
169 + batch_size: int,
170 + dtype: torch.dtype,
171 + device: torch.device,
172 + ) -> "IntermediateTensors":
173 + """Called when PP rank > 0 for profiling purposes."""
174 + ...
175 +
176 + def forward(
177 + self,
178 + input_ids: torch.Tensor,
179 + position_ids: torch.Tensor,
180 + kv_caches: List[torch.Tensor],
181 + attn_metadata: "AttentionMetadata",
182 + intermediate_tensors: Optional["IntermediateTensors"],
183 + ) -> Union[torch.Tensor, "IntermediateTensors"]:
184 + """
185 + Accept :class:`IntermediateTensors` when PP rank > 0.
186 +
187 + Return :class:`IntermediateTensors` only for the last PP rank.
188 + """
189 + ...
190 +
191 +
192 +# We can't use runtime_checkable with ClassVar for issubclass checks
193 +# so we need to treat the class as an instance and use isinstance instead
194 +@runtime_checkable
195 +class _SupportsPPType(Protocol):
196 + supports_pp: Literal[True]
197 +
198 + def make_empty_intermediate_tensors(
199 + self,
200 + batch_size: int,
201 + dtype: torch.dtype,
202 + device: torch.device,
203 + ) -> "IntermediateTensors":
204 + ...
205 +
206 + def forward(
207 + self,
208 + input_ids: torch.Tensor,
209 + position_ids: torch.Tensor,
210 + kv_caches: List[torch.Tensor],
211 + attn_metadata: "AttentionMetadata",
212 + intermediate_tensors: Optional["IntermediateTensors"],
213 + ) -> Union[torch.Tensor, "IntermediateTensors"]:
214 + ...
215 +
216 +
217 +@overload
218 +def supports_pp(model: Type[object]) -> TypeIs[Type[SupportsPP]]:
219 + ...
220 +
221 +
222 +@overload
223 +def supports_pp(model: object) -> TypeIs[SupportsPP]:
224 + ...
225 +
226 +
227 +def supports_pp(
228 + model: Union[Type[object], object],
229 +) -> Union[bool, TypeIs[Type[SupportsPP]], TypeIs[SupportsPP]]:
230 + supports_attributes = _supports_pp_attributes(model)
231 + supports_inspect = _supports_pp_inspect(model)
232 +
233 + if supports_attributes and not supports_inspect:
234 + logger.warning(
235 + "The model (%s) sets `supports_pp=True`, but does not accept "
236 + "`intermediate_tensors` in its `forward` method", model)
237 +
238 + if not supports_attributes:
239 + pp_attrs = ("make_empty_intermediate_tensors", )
240 + missing_attrs = tuple(attr for attr in pp_attrs
241 + if not hasattr(model, attr))
242 +
243 + if getattr(model, "supports_pp", False):
244 + if missing_attrs:
245 + logger.warning(
246 + "The model (%s) sets `supports_pp=True`, "
247 + "but is missing PP-specific attributes: %s",
248 + model,
249 + missing_attrs,
250 + )
251 + else:
252 + if not missing_attrs:
253 + logger.warning(
254 + "The model (%s) contains all PP-specific attributes, "
255 + "but does not set `supports_pp=True`.", model)
256 +
257 + return supports_attributes and supports_inspect
258 +
259 +
260 +def _supports_pp_attributes(
261 + model: Union[Type[object], object],
262 +) -> Union[bool, TypeIs[Type[SupportsPP]], TypeIs[SupportsPP]]:
263 + if isinstance(model, type):
264 + return isinstance(model, _SupportsPPType)
265 +
266 + return isinstance(model, SupportsPP)
267 +
268 +
269 +def _supports_pp_inspect(
270 + model: Union[Type[object], object],
271 +) -> Union[bool, TypeIs[Type[SupportsPP]], TypeIs[SupportsPP]]:
272 + model_forward = getattr(model, "forward", None)
273 + if not callable(model_forward):
274 + return False
275 +
276 + forward_params = inspect.signature(model_forward).parameters
277 + return "intermediate_tensors" in forward_params
278 +
279 +
280 @runtime_checkable
281 class HasInnerState(Protocol):
282 """The interface required for all models that has inner state."""
@@ -158,7 +290,7 @@ class HasInnerState(Protocol):
290
291 def __init__(self,
292 *,
161 - scheduler_config: Optional[SchedulerConfig] = None) -> None:
293 + scheduler_config: Optional["SchedulerConfig"] = None) -> None:
294 ...
295
296
@@ -168,7 +300,7 @@ class _HasInnerStateType(Protocol):
300
301 def __init__(self,
302 *,
171 - scheduler_config: Optional[SchedulerConfig] = None) -> None:
303 + scheduler_config: Optional["SchedulerConfig"] = None) -> None:
304 ...
305
306
vllm/model_executor/models/internlm2.py
+5 -5
@@ -18,8 +18,7 @@ from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
18 QKVParallelLinear,
19 RowParallelLinear)
20 from vllm.model_executor.layers.logits_processor import LogitsProcessor
21 -from vllm.model_executor.layers.quantization.base_config import (
22 - QuantizationConfig)
21 +from vllm.model_executor.layers.quantization import QuantizationConfig
22 from vllm.model_executor.layers.rotary_embedding import get_rope
23 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
24 from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -28,6 +27,7 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
27 from vllm.model_executor.sampling_metadata import SamplingMetadata
28 from vllm.sequence import IntermediateTensors
29
30 +from .interfaces import SupportsPP
31 from .utils import (is_pp_missing_parameter,
32 make_empty_intermediate_tensors_factory, make_layers)
33
@@ -266,7 +266,7 @@ class InternLM2Model(nn.Module):
266 positions: torch.Tensor,
267 kv_caches: List[torch.Tensor],
268 attn_metadata: AttentionMetadata,
269 - intermediate_tensors: IntermediateTensors = None,
269 + intermediate_tensors: Optional[IntermediateTensors] = None,
270 inputs_embeds: Optional[torch.Tensor] = None,
271 ) -> Union[torch.Tensor, IntermediateTensors]:
272 if get_pp_group().is_first_rank:
@@ -297,7 +297,7 @@ class InternLM2Model(nn.Module):
297 return hidden_states
298
299
300 -class InternLM2ForCausalLM(nn.Module):
300 +class InternLM2ForCausalLM(nn.Module, SupportsPP):
301
302 def __init__(
303 self,
@@ -325,7 +325,7 @@ class InternLM2ForCausalLM(nn.Module):
325 positions: torch.Tensor,
326 kv_caches: List[torch.Tensor],
327 attn_metadata: AttentionMetadata,
328 - intermediate_tensors: IntermediateTensors,
328 + intermediate_tensors: Optional[IntermediateTensors],
329 ) -> torch.Tensor:
330 hidden_states = self.model(input_ids, positions, kv_caches,
331 attn_metadata, intermediate_tensors)
vllm/model_executor/models/internvl.py
+27 -22
@@ -5,9 +5,9 @@
5 # Licensed under The MIT License [see LICENSE for details]
6 # --------------------------------------------------------
7 import re
8 -from functools import partial
9 -from typing import (Any, Dict, Iterable, List, Literal, Mapping, Optional,
10 - Tuple, TypedDict, Union)
8 +from functools import cached_property, partial
9 +from typing import (Iterable, List, Literal, Mapping, Optional, Tuple,
10 + TypedDict, Union)
11
12 import torch
13 import torch.nn as nn
@@ -17,7 +17,6 @@ from transformers import PretrainedConfig
17
18 from vllm.attention import AttentionMetadata
19 from vllm.config import CacheConfig, MultiModalConfig
20 -from vllm.distributed import get_pp_group
20 from vllm.inputs import INPUT_REGISTRY, InputContext, LLMInputs
21 from vllm.model_executor.layers.quantization import QuantizationConfig
22 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
@@ -32,7 +31,7 @@ from vllm.utils import is_list_of
31
32 from .clip import (dummy_image_for_clip, dummy_seq_data_for_clip,
33 get_clip_num_patches)
35 -from .interfaces import SupportsMultiModal
34 +from .interfaces import SupportsMultiModal, SupportsPP
35 from .utils import (flatten_bn, group_weights_with_prefix,
36 init_vllm_registered_model, merge_multimodal_embeddings)
37
@@ -123,7 +122,7 @@ def calculate_num_blocks(orig_width: int, orig_height: int, min_num: int,
122 return blocks, target_width, target_height
123
124
126 -def calculate_num_blocks_wrapper(hf_config: Dict[str, Any],
125 +def calculate_num_blocks_wrapper(hf_config: PretrainedConfig,
126 max_dynamic_patch: Optional[int] = None):
127 if max_dynamic_patch is None:
128 max_dynamic_patch = hf_config.max_dynamic_patch
@@ -183,7 +182,7 @@ def image_to_pixel_values(image: Image.Image, input_size: int, min_num: int,
182 return pixel_values
183
184
186 -def image_to_pixel_values_wrapper(hf_config: Dict[str, Any],
185 +def image_to_pixel_values_wrapper(hf_config: PretrainedConfig,
186 max_dynamic_patch: Optional[int] = None):
187 image_size = hf_config.vision_config.image_size
188 min_num = hf_config.min_dynamic_patch
@@ -197,7 +196,7 @@ def image_to_pixel_values_wrapper(hf_config: Dict[str, Any],
196 use_thumbnail=use_thumbnail)
197
198
200 -def get_internvl_num_patches(hf_config: Dict[str, Any]):
199 +def get_internvl_num_patches(hf_config: PretrainedConfig):
200 vision_config = hf_config.vision_config
201 downsample_ratio = hf_config.downsample_ratio
202 image_size = vision_config.image_size
@@ -362,7 +361,7 @@ def dummy_data_for_internvl(ctx: InputContext,
361 @MULTIMODAL_REGISTRY.register_max_image_tokens(get_max_internvl_image_tokens)
362 @INPUT_REGISTRY.register_dummy_data(dummy_data_for_internvl)
363 @INPUT_REGISTRY.register_input_processor(input_processor_for_internvl)
365 -class InternVLChatModel(nn.Module, SupportsMultiModal):
364 +class InternVLChatModel(nn.Module, SupportsMultiModal, SupportsPP):
365
366 def __init__(self,
367 config: PretrainedConfig,
@@ -408,10 +407,12 @@ class InternVLChatModel(nn.Module, SupportsMultiModal):
407 self.make_empty_intermediate_tensors = (
408 self.language_model.make_empty_intermediate_tensors)
409
410 + @cached_property
411 + def sampler(self):
412 if hasattr(self.language_model, "sampler"):
412 - self.sampler = self.language_model.sampler
413 - else:
414 - self.sampler = Sampler()
413 + return self.language_model.sampler
414 +
415 + return Sampler()
416
417 def pixel_shuffle(self, x, scale_factor=0.5):
418 n, w, h, c = x.size()
@@ -515,18 +516,22 @@ class InternVLChatModel(nn.Module, SupportsMultiModal):
516 attn_metadata: AttentionMetadata,
517 intermediate_tensors: Optional[IntermediateTensors] = None,
518 **kwargs: object,
518 - ) -> SamplerOutput:
519 - image_input = self._parse_and_validate_image_input(**kwargs)
520 - if image_input is not None and get_pp_group().is_first_rank:
521 - inputs_embeds = self.language_model.model.get_input_embeddings(
522 - input_ids)
523 - vision_embeddings = self._process_image_input(image_input)
524 - inputs_embeds = merge_multimodal_embeddings(
525 - input_ids, inputs_embeds, vision_embeddings,
526 - self.img_context_token_id)
519 + ) -> Union[SamplerOutput, IntermediateTensors]:
520 + if intermediate_tensors is not None:
521 input_ids = None
528 - else:
522 inputs_embeds = None
523 + else:
524 + image_input = self._parse_and_validate_image_input(**kwargs)
525 + if image_input is not None:
526 + inputs_embeds = self.language_model.model.get_input_embeddings(
527 + input_ids)
528 + vision_embeddings = self._process_image_input(image_input)
529 + inputs_embeds = merge_multimodal_embeddings(
530 + input_ids, inputs_embeds, vision_embeddings,
531 + self.img_context_token_id)
532 + input_ids = None
533 + else:
534 + inputs_embeds = None
535
536 hidden_states = self.language_model.model(input_ids,
537 positions,
vllm/model_executor/models/jais.py
+10 -14
@@ -33,8 +33,7 @@ from vllm.model_executor.layers.linear import (ColumnParallelLinear,
33 QKVParallelLinear,
34 RowParallelLinear)
35 from vllm.model_executor.layers.logits_processor import LogitsProcessor
36 -from vllm.model_executor.layers.quantization.base_config import (
37 - QuantizationConfig)
36 +from vllm.model_executor.layers.quantization import QuantizationConfig
37 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
38 from vllm.model_executor.layers.vocab_parallel_embedding import (
39 ParallelLMHead, VocabParallelEmbedding)
@@ -43,7 +42,9 @@ from vllm.model_executor.sampling_metadata import SamplingMetadata
42 from vllm.sequence import IntermediateTensors
43 from vllm.transformers_utils.configs import JAISConfig
44
46 -from .utils import is_pp_missing_parameter, make_layers
45 +from .interfaces import SupportsPP
46 +from .utils import (is_pp_missing_parameter,
47 + make_empty_intermediate_tensors_factory, make_layers)
48
49
50 class SwiGLUActivation(nn.Module):
@@ -244,6 +245,9 @@ class JAISModel(nn.Module):
245 )
246
247 self.ln_f = nn.LayerNorm(self.embed_dim, eps=config.layer_norm_epsilon)
248 + self.make_empty_intermediate_tensors = (
249 + make_empty_intermediate_tensors_factory(["hidden_states"],
250 + config.n_embd))
251
252 def forward(
253 self,
@@ -279,7 +283,7 @@ class JAISModel(nn.Module):
283 return hidden_states
284
285
282 -class JAISLMHeadModel(nn.Module):
286 +class JAISLMHeadModel(nn.Module, SupportsPP):
287
288 def __init__(
289 self,
@@ -304,6 +308,8 @@ class JAISLMHeadModel(nn.Module):
308 self.logits_processor = LogitsProcessor(vocab_size=config.vocab_size,
309 scale=self.output_logits_scale)
310 self.sampler = Sampler()
311 + self.make_empty_intermediate_tensors = (
312 + self.transformer.make_empty_intermediate_tensors)
313
314 def forward(
315 self,
@@ -326,16 +332,6 @@ class JAISLMHeadModel(nn.Module):
332 sampling_metadata)
333 return logits
334
329 - def make_empty_intermediate_tensors(
330 - self, batch_size: int, dtype: torch.dtype,
331 - device: torch.device) -> IntermediateTensors:
332 - return IntermediateTensors({
333 - "hidden_states":
334 - torch.zeros((batch_size, self.config.hidden_size),
335 - dtype=dtype,
336 - device=device),
337 - })
338 -
335 def sample(
336 self,
337 logits: torch.Tensor,
vllm/model_executor/models/llama.py
+49 -65
@@ -37,8 +37,7 @@ from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
37 QKVParallelLinear,
38 RowParallelLinear)
39 from vllm.model_executor.layers.logits_processor import LogitsProcessor
40 -from vllm.model_executor.layers.quantization.base_config import (
41 - QuantizationConfig)
40 +from vllm.model_executor.layers.quantization import QuantizationConfig
41 from vllm.model_executor.layers.quantization.compressed_tensors.utils import (
42 get_compressed_tensors_cache_scale)
43 from vllm.model_executor.layers.rotary_embedding import get_rope
@@ -51,8 +50,9 @@ from vllm.model_executor.sampling_metadata import SamplingMetadata
50 from vllm.sequence import IntermediateTensors
51 from vllm.utils import is_hip
52
54 -from .interfaces import SupportsLoRA
55 -from .utils import PPMissingLayer, is_pp_missing_parameter, make_layers
53 +from .interfaces import SupportsLoRA, SupportsPP
54 +from .utils import (PPMissingLayer, is_pp_missing_parameter,
55 + make_empty_intermediate_tensors_factory, make_layers)
56
57
58 class LlamaMLP(nn.Module):
@@ -72,12 +72,15 @@ class LlamaMLP(nn.Module):
72 output_sizes=[intermediate_size] * 2,
73 bias=bias,
74 quant_config=quant_config,
75 - prefix=f"{prefix}.gate_up_proj")
76 - self.down_proj = RowParallelLinear(input_size=intermediate_size,
77 - output_size=hidden_size,
78 - bias=bias,
79 - quant_config=quant_config,
80 - prefix=f"{prefix}.down_proj")
75 + prefix=f"{prefix}.gate_up_proj",
76 + )
77 + self.down_proj = RowParallelLinear(
78 + input_size=intermediate_size,
79 + output_size=hidden_size,
80 + bias=bias,
81 + quant_config=quant_config,
82 + prefix=f"{prefix}.down_proj",
83 + )
84 if hidden_act != "silu":
85 raise ValueError(f"Unsupported activation: {hidden_act}. "
86 "Only silu is supported for now.")
@@ -161,12 +164,14 @@ class LlamaAttention(nn.Module):
164 rope_scaling=rope_scaling,
165 is_neox_style=is_neox_style,
166 )
164 - self.attn = Attention(self.num_heads,
165 - self.head_dim,
166 - self.scaling,
167 - num_kv_heads=self.num_kv_heads,
168 - cache_config=cache_config,
169 - quant_config=quant_config)
167 + self.attn = Attention(
168 + self.num_heads,
169 + self.head_dim,
170 + self.scaling,
171 + num_kv_heads=self.num_kv_heads,
172 + cache_config=cache_config,
173 + quant_config=quant_config,
174 + )
175
176 def forward(
177 self,
@@ -248,12 +253,10 @@ class LlamaDecoderLayer(nn.Module):
253 else:
254 hidden_states, residual = self.input_layernorm(
255 hidden_states, residual)
251 - hidden_states = self.self_attn(
252 - positions=positions,
253 - hidden_states=hidden_states,
254 - kv_cache=kv_cache,
255 - attn_metadata=attn_metadata,
256 - )
256 + hidden_states = self.self_attn(positions=positions,
257 + hidden_states=hidden_states,
258 + kv_cache=kv_cache,
259 + attn_metadata=attn_metadata)
260
261 # Fully Connected
262 hidden_states, residual = self.post_attention_layernorm(
@@ -295,12 +298,17 @@ class LlamaModel(nn.Module):
298 cache_config=cache_config,
299 quant_config=quant_config,
300 prefix=prefix),
298 - prefix=f"{prefix}.layers")
301 + prefix=f"{prefix}.layers",
302 + )
303 if get_pp_group().is_last_rank:
304 self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
305 else:
306 self.norm = PPMissingLayer()
307
308 + self.make_empty_intermediate_tensors = (
309 + make_empty_intermediate_tensors_factory(
310 + ["hidden_states", "residual"], config.hidden_size))
311 +
312 def get_input_embeddings(self, input_ids: torch.Tensor) -> torch.Tensor:
313 return self.embed_tokens(input_ids)
314
@@ -326,13 +334,9 @@ class LlamaModel(nn.Module):
334
335 for i in range(self.start_layer, self.end_layer):
336 layer = self.layers[i]
329 - hidden_states, residual = layer(
330 - positions,
331 - hidden_states,
332 - kv_caches[i - self.start_layer],
333 - attn_metadata,
334 - residual,
335 - )
337 + hidden_states, residual = layer(positions, hidden_states,
338 + kv_caches[i - self.start_layer],
339 + attn_metadata, residual)
340
341 if not get_pp_group().is_last_rank:
342 return IntermediateTensors({
@@ -344,17 +348,10 @@ class LlamaModel(nn.Module):
348 return hidden_states
349
350
347 -class LlamaForCausalLM(nn.Module, SupportsLoRA):
351 +class LlamaForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
352 packed_modules_mapping = {
349 - "qkv_proj": [
350 - "q_proj",
351 - "k_proj",
352 - "v_proj",
353 - ],
354 - "gate_up_proj": [
355 - "gate_proj",
356 - "up_proj",
357 - ],
353 + "qkv_proj": ["q_proj", "k_proj", "v_proj"],
354 + "gate_up_proj": ["gate_proj", "up_proj"]
355 }
356
357 # LoRA specific attributes
@@ -364,7 +361,7 @@ class LlamaForCausalLM(nn.Module, SupportsLoRA):
361 ]
362 embedding_modules = {
363 "embed_tokens": "input_embeddings",
367 - "lm_head": "output_embeddings",
364 + "lm_head": "output_embeddings"
365 }
366 embedding_padding_modules = ["lm_head"]
367 bitsandbytes_stacked_params_mapping = {
@@ -420,10 +417,12 @@ class LlamaForCausalLM(nn.Module, SupportsLoRA):
417 self.unpadded_vocab_size,
418 config.hidden_size,
419 org_num_embeddings=config.vocab_size,
423 - padding_size=DEFAULT_VOCAB_PADDING_SIZE
424 - # We need bigger padding if using lora for kernel
425 - # compatibility
426 - if not lora_config else lora_config.lora_vocab_padding_size,
420 + padding_size=(
421 + DEFAULT_VOCAB_PADDING_SIZE
422 + # We need bigger padding if using lora for kernel
423 + # compatibility
424 + if not lora_config else
425 + lora_config.lora_vocab_padding_size),
426 quant_config=quant_config,
427 )
428 if config.tie_word_embeddings:
@@ -436,6 +435,8 @@ class LlamaForCausalLM(nn.Module, SupportsLoRA):
435 self.sampler = Sampler()
436 else:
437 self.lm_head = PPMissingLayer()
438 + self.make_empty_intermediate_tensors = (
439 + self.model.make_empty_intermediate_tensors)
440
441 def forward(
442 self,
@@ -458,28 +459,11 @@ class LlamaForCausalLM(nn.Module, SupportsLoRA):
459 sampling_metadata)
460 return logits
461
461 - def sample(
462 - self,
463 - logits: torch.Tensor,
464 - sampling_metadata: SamplingMetadata,
465 - ) -> Optional[SamplerOutput]:
462 + def sample(self, logits: torch.Tensor,
463 + sampling_metadata: SamplingMetadata) -> Optional[SamplerOutput]:
464 next_tokens = self.sampler(logits, sampling_metadata)
465 return next_tokens
466
469 - def make_empty_intermediate_tensors(
470 - self, batch_size: int, dtype: torch.dtype,
471 - device: torch.device) -> IntermediateTensors:
472 - return IntermediateTensors({
473 - "hidden_states":
474 - torch.zeros((batch_size, self.config.hidden_size),
475 - dtype=dtype,
476 - device=device),
477 - "residual":
478 - torch.zeros((batch_size, self.config.hidden_size),
479 - dtype=dtype,
480 - device=device),
481 - })
482 -
467 def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
468 stacked_params_mapping = [
469 # (param_name, shard_name, shard_id)
@@ -513,7 +497,7 @@ class LlamaForCausalLM(nn.Module, SupportsLoRA):
497 loaded_weight = loaded_weight[0]
498 weight_loader(param, loaded_weight)
499 continue
516 - for (param_name, weight_name, shard_id) in stacked_params_mapping:
500 + for param_name, weight_name, shard_id in stacked_params_mapping:
501 if weight_name not in name:
502 continue
503 name = name.replace(weight_name, param_name)
vllm/model_executor/models/llama_embedding.py
+16 -5
@@ -1,4 +1,4 @@
1 -from typing import Iterable, List, Optional, Tuple
1 +from typing import Iterable, List, Optional, Tuple, Union
2
3 import torch
4 from torch import nn
@@ -8,10 +8,13 @@ from vllm.model_executor.layers.pooler import Pooler, PoolingType
8 from vllm.model_executor.model_loader.weight_utils import default_weight_loader
9 from vllm.model_executor.models.llama import LlamaModel
10 from vllm.model_executor.pooling_metadata import PoolingMetadata
11 -from vllm.sequence import PoolerOutput
11 +from vllm.sequence import IntermediateTensors, PoolerOutput
12
13 +from .interfaces import SupportsPP
14 +from .utils import is_pp_missing_parameter
15
14 -class LlamaEmbeddingModel(nn.Module):
16 +
17 +class LlamaEmbeddingModel(nn.Module, SupportsPP):
18 """A model that uses Llama with additional embedding functionalities.
19
20 This class encapsulates the LlamaModel and provides an interface for
@@ -29,6 +32,8 @@ class LlamaEmbeddingModel(nn.Module):
32 super().__init__()
33 self.model = LlamaModel(**kwargs)
34 self._pooler = Pooler(pooling_type=PoolingType.LAST, normalize=True)
35 + self.make_empty_intermediate_tensors = (
36 + self.model.make_empty_intermediate_tensors)
37
38 def forward(
39 self,
@@ -36,10 +41,12 @@ class LlamaEmbeddingModel(nn.Module):
41 positions: torch.Tensor,
42 kv_caches: List[torch.Tensor],
43 attn_metadata: AttentionMetadata,
44 + intermediate_tensors: Optional[IntermediateTensors] = None,
45 inputs_embeds: Optional[torch.Tensor] = None,
40 - ) -> torch.Tensor:
46 + ) -> Union[torch.Tensor, IntermediateTensors]:
47 return self.model.forward(input_ids, positions, kv_caches,
42 - attn_metadata, inputs_embeds)
48 + attn_metadata, intermediate_tensors,
49 + inputs_embeds)
50
51 def pooler(
52 self,
@@ -73,6 +80,8 @@ class LlamaEmbeddingModel(nn.Module):
80 # Skip loading extra bias for GPTQ models.
81 if name.endswith(".bias") and name not in params_dict:
82 continue
83 + if is_pp_missing_parameter(name, self):
84 + continue
85 param = params_dict[name]
86 weight_loader = param.weight_loader
87 weight_loader(param, loaded_weight, shard_id)
@@ -81,6 +90,8 @@ class LlamaEmbeddingModel(nn.Module):
90 # Skip loading extra bias for GPTQ models.
91 if name.endswith(".bias") and name not in params_dict:
92 continue
93 + if is_pp_missing_parameter(name, self):
94 + continue
95 param = params_dict[name]
96 weight_loader = getattr(param, "weight_loader",
97 default_weight_loader)
vllm/model_executor/models/llava.py
+31 -16
@@ -1,3 +1,4 @@
1 +from functools import cached_property
2 from typing import (Iterable, List, Literal, Mapping, Optional, Tuple,
3 TypedDict, Union)
4
@@ -11,7 +12,7 @@ from vllm.config import CacheConfig, MultiModalConfig
12 from vllm.inputs import INPUT_REGISTRY, InputContext, LLMInputs
13 from vllm.model_executor.layers.activation import get_act_fn
14 from vllm.model_executor.layers.quantization import QuantizationConfig
14 -from vllm.model_executor.layers.sampler import SamplerOutput
15 +from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
16 from vllm.model_executor.model_loader.weight_utils import default_weight_loader
17 from vllm.model_executor.sampling_metadata import SamplingMetadata
18 from vllm.multimodal import MULTIMODAL_REGISTRY
@@ -21,7 +22,7 @@ from vllm.utils import is_list_of
22 from .clip import (CLIPVisionModel, dummy_image_for_clip,
23 dummy_seq_data_for_clip, get_max_clip_image_tokens,
24 input_processor_for_clip)
24 -from .interfaces import SupportsMultiModal
25 +from .interfaces import SupportsMultiModal, SupportsPP
26 from .siglip import (SiglipVisionModel, dummy_image_for_siglip,
27 dummy_seq_data_for_siglip, get_max_siglip_image_tokens,
28 input_processor_for_siglip)
@@ -198,7 +199,7 @@ def _init_vision_tower(hf_config: LlavaConfig):
199 @MULTIMODAL_REGISTRY.register_max_image_tokens(get_max_llava_image_tokens)
200 @INPUT_REGISTRY.register_dummy_data(dummy_data_for_llava)
201 @INPUT_REGISTRY.register_input_processor(input_processor_for_llava)
201 -class LlavaForConditionalGeneration(nn.Module, SupportsMultiModal):
202 +class LlavaForConditionalGeneration(nn.Module, SupportsMultiModal, SupportsPP):
203
204 def __init__(self,
205 config: LlavaConfig,
@@ -220,6 +221,16 @@ class LlavaForConditionalGeneration(nn.Module, SupportsMultiModal):
221 self.language_model = init_vllm_registered_model(
222 config.text_config, cache_config, quant_config)
223
224 + self.make_empty_intermediate_tensors = (
225 + self.language_model.make_empty_intermediate_tensors)
226 +
227 + @cached_property
228 + def sampler(self):
229 + if hasattr(self.language_model, "sampler"):
230 + return self.language_model.sampler
231 +
232 + return Sampler()
233 +
234 def _validate_pixel_values(self, data: torch.Tensor) -> torch.Tensor:
235 h = w = self.config.vision_config.image_size
236 expected_dims = (3, h, w)
@@ -315,7 +326,7 @@ class LlavaForConditionalGeneration(nn.Module, SupportsMultiModal):
326 attn_metadata: AttentionMetadata,
327 intermediate_tensors: Optional[IntermediateTensors] = None,
328 **kwargs: object,
318 - ) -> SamplerOutput:
329 + ) -> Union[torch.Tensor, IntermediateTensors]:
330 """Run forward pass for LLaVA-1.5.
331
332 One key thing to understand is the `input_ids` already accounts for the
@@ -351,26 +362,30 @@ class LlavaForConditionalGeneration(nn.Module, SupportsMultiModal):
362 See also:
363 :class:`LlavaImageInputs`
364 """
354 - image_input = self._parse_and_validate_image_input(**kwargs)
365 + if intermediate_tensors is not None:
366 + input_ids = None
367 + inputs_embeds = None
368 + else:
369 + image_input = self._parse_and_validate_image_input(**kwargs)
370
356 - if image_input is not None:
357 - vision_embeddings = self._process_image_input(image_input)
358 - inputs_embeds = self.language_model.model.get_input_embeddings(
359 - input_ids)
371 + if image_input is not None:
372 + vision_embeddings = self._process_image_input(image_input)
373 + inputs_embeds = self.language_model.model.get_input_embeddings(
374 + input_ids)
375
361 - inputs_embeds = merge_multimodal_embeddings(
362 - input_ids, inputs_embeds, vision_embeddings,
363 - self.config.image_token_index)
376 + inputs_embeds = merge_multimodal_embeddings(
377 + input_ids, inputs_embeds, vision_embeddings,
378 + self.config.image_token_index)
379
365 - input_ids = None
366 - else:
367 - inputs_embeds = None
380 + input_ids = None
381 + else:
382 + inputs_embeds = None
383
384 hidden_states = self.language_model.model(input_ids,
385 positions,
386 kv_caches,
387 attn_metadata,
373 - None,
388 + intermediate_tensors,
389 inputs_embeds=inputs_embeds)
390
391 return hidden_states
vllm/model_executor/models/llava_next.py
+33 -18
@@ -1,3 +1,4 @@
1 +from functools import cached_property
2 from typing import (Iterable, List, Literal, Mapping, Optional, Tuple,
3 TypedDict, Union)
4
@@ -13,7 +14,7 @@ from vllm.attention import AttentionMetadata
14 from vllm.config import CacheConfig, MultiModalConfig
15 from vllm.inputs import INPUT_REGISTRY, InputContext, LLMInputs
16 from vllm.model_executor.layers.quantization import QuantizationConfig
16 -from vllm.model_executor.layers.sampler import SamplerOutput
17 +from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
18 from vllm.model_executor.model_loader.weight_utils import default_weight_loader
19 from vllm.model_executor.sampling_metadata import SamplingMetadata
20 from vllm.multimodal import MULTIMODAL_REGISTRY
@@ -23,7 +24,7 @@ from vllm.utils import is_list_of
24 from .clip import (CLIPVisionModel, dummy_image_for_clip,
25 dummy_seq_data_for_clip, get_clip_image_feature_size,
26 get_clip_patch_grid_length, input_processor_for_clip)
26 -from .interfaces import SupportsMultiModal
27 +from .interfaces import SupportsMultiModal, SupportsPP
28 from .llava import LlavaMultiModalProjector
29 from .siglip import (SiglipVisionModel, dummy_image_for_siglip,
30 dummy_seq_data_for_siglip, get_siglip_image_feature_size,
@@ -286,7 +287,8 @@ def _init_vision_tower(hf_config: LlavaNextConfig):
287 @MULTIMODAL_REGISTRY.register_max_image_tokens(get_max_llava_next_image_tokens)
288 @INPUT_REGISTRY.register_dummy_data(dummy_data_for_llava_next)
289 @INPUT_REGISTRY.register_input_processor(input_processor_for_llava_next)
289 -class LlavaNextForConditionalGeneration(nn.Module, SupportsMultiModal):
290 +class LlavaNextForConditionalGeneration(nn.Module, SupportsMultiModal,
291 + SupportsPP):
292
293 def __init__(self,
294 config: LlavaNextConfig,
@@ -300,6 +302,8 @@ class LlavaNextForConditionalGeneration(nn.Module, SupportsMultiModal):
302
303 # TODO: Optionally initializes this for supporting embeddings.
304 self.vision_tower = _init_vision_tower(config)
305 + self.image_newline = nn.Parameter(
306 + torch.empty(config.text_config.hidden_size))
307 self.multi_modal_projector = LlavaMultiModalProjector(
308 vision_hidden_size=config.vision_config.hidden_size,
309 text_hidden_size=config.text_config.hidden_size,
@@ -308,8 +312,15 @@ class LlavaNextForConditionalGeneration(nn.Module, SupportsMultiModal):
312 self.language_model = init_vllm_registered_model(
313 config.text_config, cache_config, quant_config)
314
311 - self.image_newline = nn.Parameter(
312 - torch.empty(config.text_config.hidden_size))
315 + self.make_empty_intermediate_tensors = (
316 + self.language_model.make_empty_intermediate_tensors)
317 +
318 + @cached_property
319 + def sampler(self):
320 + if hasattr(self.language_model, "sampler"):
321 + return self.language_model.sampler
322 +
323 + return Sampler()
324
325 def _validate_image_sizes(self, data: torch.Tensor) -> torch.Tensor:
326 expected_dims = (2, )
@@ -542,7 +553,7 @@ class LlavaNextForConditionalGeneration(nn.Module, SupportsMultiModal):
553 attn_metadata: AttentionMetadata,
554 intermediate_tensors: Optional[IntermediateTensors] = None,
555 **kwargs: object,
545 - ) -> SamplerOutput:
556 + ) -> Union[torch.Tensor, IntermediateTensors]:
557 """Run forward pass for LlaVA-NeXT.
558
559 One key thing to understand is the `input_ids` already accounts for the
@@ -587,26 +598,30 @@ class LlavaNextForConditionalGeneration(nn.Module, SupportsMultiModal):
598 See also:
599 :class:`LlavaNextImageInputs`
600 """
590 - image_input = self._parse_and_validate_image_input(**kwargs)
601 + if intermediate_tensors is not None:
602 + input_ids = None
603 + inputs_embeds = None
604 + else:
605 + image_input = self._parse_and_validate_image_input(**kwargs)
606
592 - if image_input is not None:
593 - vision_embeddings = self._process_image_input(image_input)
594 - inputs_embeds = self.language_model.model.get_input_embeddings(
595 - input_ids)
607 + if image_input is not None:
608 + vision_embeddings = self._process_image_input(image_input)
609 + inputs_embeds = self.language_model.model.get_input_embeddings(
610 + input_ids)
611
597 - inputs_embeds = merge_multimodal_embeddings(
598 - input_ids, inputs_embeds, vision_embeddings,
599 - self.config.image_token_index)
612 + inputs_embeds = merge_multimodal_embeddings(
613 + input_ids, inputs_embeds, vision_embeddings,
614 + self.config.image_token_index)
615
601 - input_ids = None
602 - else:
603 - inputs_embeds = None
616 + input_ids = None
617 + else:
618 + inputs_embeds = None
619
620 hidden_states = self.language_model.model(input_ids,
621 positions,
622 kv_caches,
623 attn_metadata,
609 - None,
624 + intermediate_tensors,
625 inputs_embeds=inputs_embeds)
626
627 return hidden_states
vllm/model_executor/models/llava_next_video.py
+34 -21
@@ -1,4 +1,5 @@
1 import math
2 +from functools import cached_property
3 from typing import (Iterable, List, Literal, Mapping, Optional, Tuple,
4 TypedDict, Union)
5
@@ -12,9 +13,8 @@ from vllm.attention import AttentionMetadata
13 from vllm.config import CacheConfig, MultiModalConfig
14 from vllm.inputs import INPUT_REGISTRY, InputContext, LLMInputs
15 from vllm.model_executor.layers.activation import get_act_fn
15 -from vllm.model_executor.layers.quantization.base_config import (
16 - QuantizationConfig)
17 -from vllm.model_executor.layers.sampler import SamplerOutput
16 +from vllm.model_executor.layers.quantization import QuantizationConfig
17 +from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
18 from vllm.model_executor.model_loader.weight_utils import default_weight_loader
19 from vllm.model_executor.models.clip import CLIPVisionModel
20 from vllm.model_executor.sampling_metadata import SamplingMetadata
@@ -25,7 +25,7 @@ from vllm.sequence import IntermediateTensors
25 from vllm.utils import is_list_of
26
27 from .clip import dummy_image_for_clip, dummy_seq_data_for_clip
28 -from .interfaces import SupportsMultiModal
28 +from .interfaces import SupportsMultiModal, SupportsPP
29 from .siglip import (SiglipVisionModel, dummy_image_for_siglip,
30 dummy_seq_data_for_siglip)
31 from .utils import (group_weights_with_prefix, init_vllm_registered_model,
@@ -267,7 +267,8 @@ class LlavaNextMultiModalProjector(nn.Module):
267 "video", get_max_llava_next_video_tokens)
268 @INPUT_REGISTRY.register_dummy_data(dummy_data_for_llava_next_video)
269 @INPUT_REGISTRY.register_input_processor(input_processor_for_llava_next_video)
270 -class LlavaNextVideoForConditionalGeneration(nn.Module, SupportsMultiModal):
270 +class LlavaNextVideoForConditionalGeneration(nn.Module, SupportsMultiModal,
271 + SupportsPP):
272
273 def __init__(self,
274 config: LlavaNextVideoConfig,
@@ -281,13 +282,23 @@ class LlavaNextVideoForConditionalGeneration(nn.Module, SupportsMultiModal):
282
283 # Initialize the vision tower only up to the required feature layer
284 self.vision_tower = _init_vision_tower(config)
285 + self.vision_resampler = LlavaNextVideoPooler(config)
286 self.multi_modal_projector = LlavaNextMultiModalProjector(
287 vision_hidden_size=config.vision_config.hidden_size,
288 text_hidden_size=config.text_config.hidden_size,
289 projector_hidden_act=config.projector_hidden_act)
290 self.language_model = init_vllm_registered_model(
291 config.text_config, cache_config, quant_config)
290 - self.vision_resampler = LlavaNextVideoPooler(config)
292 +
293 + self.make_empty_intermediate_tensors = (
294 + self.language_model.model.make_empty_intermediate_tensors)
295 +
296 + @cached_property
297 + def sampler(self):
298 + if hasattr(self.language_model, "sampler"):
299 + return self.language_model.sampler
300 +
301 + return Sampler()
302
303 def _validate_video_pixel_values(
304 self, data: Union[torch.Tensor, List[torch.Tensor]]
@@ -397,34 +408,36 @@ class LlavaNextVideoForConditionalGeneration(nn.Module, SupportsMultiModal):
408 attn_metadata: AttentionMetadata,
409 intermediate_tensors: Optional[IntermediateTensors] = None,
410 **kwargs: object,
400 - ) -> SamplerOutput:
411 + ) -> Union[torch.Tensor, IntermediateTensors]:
412 """Run forward pass for LlaVA-NeXT-Video.
413 Args:
414 input_ids: Flattened (concatenated) input_ids corresponding to a
415 batch.
416 pixel_values_videos: Pixels in each frames for each input videos.
417 """
407 - video_input = self._parse_and_validate_video_input(**kwargs)
408 -
409 - # merge video embeddings into input embeddings
410 - if video_input is not None:
411 - video_embeddings = self._process_video_pixels(video_input)
412 - inputs_embeds = self.language_model \
413 - .model.get_input_embeddings(input_ids)
414 -
415 - inputs_embeds = merge_multimodal_embeddings(
416 - input_ids, inputs_embeds, video_embeddings,
417 - self.config.video_token_index)
418 -
418 + if intermediate_tensors is not None:
419 input_ids = None
420 - else:
420 inputs_embeds = None
421 + else:
422 + video_input = self._parse_and_validate_video_input(**kwargs)
423 + if video_input is not None:
424 + video_embeddings = self._process_video_pixels(video_input)
425 + inputs_embeds = self.language_model \
426 + .model.get_input_embeddings(input_ids)
427 +
428 + inputs_embeds = merge_multimodal_embeddings(
429 + input_ids, inputs_embeds, video_embeddings,
430 + self.config.video_token_index)
431 +
432 + input_ids = None
433 + else:
434 + inputs_embeds = None
435
436 hidden_states = self.language_model.model(input_ids,
437 positions,
438 kv_caches,
439 attn_metadata,
427 - None,
440 + intermediate_tensors,
441 inputs_embeds=inputs_embeds)
442
443 return hidden_states
vllm/model_executor/models/llava_onevision.py
+39 -25
@@ -1,4 +1,5 @@
1 import math
2 +from functools import cached_property
3 from typing import (Iterable, List, Literal, Mapping, Optional, Tuple,
4 TypedDict, Union)
5
@@ -17,9 +18,8 @@ from vllm.config import CacheConfig, MultiModalConfig
18 from vllm.inputs import INPUT_REGISTRY, InputContext, LLMInputs
19 from vllm.logger import init_logger
20 from vllm.model_executor.layers.activation import get_act_fn
20 -from vllm.model_executor.layers.quantization.base_config import (
21 - QuantizationConfig)
22 -from vllm.model_executor.layers.sampler import SamplerOutput
21 +from vllm.model_executor.layers.quantization import QuantizationConfig
22 +from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
23 from vllm.model_executor.model_loader.weight_utils import default_weight_loader
24 from vllm.model_executor.sampling_metadata import SamplingMetadata
25 from vllm.multimodal import MULTIMODAL_REGISTRY
@@ -31,7 +31,7 @@ from vllm.utils import is_list_of
31 from .clip import (CLIPVisionModel, dummy_seq_data_for_clip,
32 dummy_video_for_clip, get_clip_image_feature_size,
33 get_clip_patch_grid_length, input_processor_for_clip)
34 -from .interfaces import SupportsMultiModal
34 +from .interfaces import SupportsMultiModal, SupportsPP
35 from .siglip import (SiglipVisionModel, dummy_seq_data_for_siglip,
36 dummy_video_for_siglip, get_siglip_image_feature_size,
37 get_siglip_patch_grid_length, input_processor_for_siglip)
@@ -414,7 +414,8 @@ class LlavaOnevisionMultiModalProjector(nn.Module):
414 "video", get_max_llava_onevision_video_tokens)
415 @INPUT_REGISTRY.register_dummy_data(dummy_data_for_llava_onevision)
416 @INPUT_REGISTRY.register_input_processor(input_processor_for_llava_onevision)
417 -class LlavaOnevisionForConditionalGeneration(nn.Module, SupportsMultiModal):
417 +class LlavaOnevisionForConditionalGeneration(nn.Module, SupportsMultiModal,
418 + SupportsPP):
419
420 def __init__(self,
421 config: LlavaOnevisionConfig,
@@ -434,6 +435,16 @@ class LlavaOnevisionForConditionalGeneration(nn.Module, SupportsMultiModal):
435 self.image_newline = nn.Parameter(
436 torch.empty(config.text_config.hidden_size))
437
438 + self.make_empty_intermediate_tensors = (
439 + self.language_model.model.make_empty_intermediate_tensors)
440 +
441 + @cached_property
442 + def sampler(self):
443 + if hasattr(self.language_model, "sampler"):
444 + return self.language_model.sampler
445 +
446 + return Sampler()
447 +
448 def _validate_image_sizes(self, data: torch.Tensor) -> torch.Tensor:
449 expected_dims = (2, )
450
@@ -805,39 +816,42 @@ class LlavaOnevisionForConditionalGeneration(nn.Module, SupportsMultiModal):
816 attn_metadata: AttentionMetadata,
817 intermediate_tensors: Optional[IntermediateTensors] = None,
818 **kwargs: object,
808 - ) -> SamplerOutput:
819 + ) -> Union[torch.Tensor, IntermediateTensors]:
820 """Run forward pass for LlaVA-Onevision.
821 Args:
822 input_ids: Flattened (concatenated) input_ids corresponding to a
823 batch.
824 pixel_values_videos: Pixels in each frames for each input videos.
825 """
815 - modalities = self._parse_and_validate_multimodal_inputs(**kwargs)
816 - # merge video embeddings into input embeddings
817 - if modalities:
818 - inputs_embeds = self.language_model.model.get_input_embeddings(
819 - input_ids)
820 - if "images" in modalities:
821 - image_input = modalities["images"]
822 - vision_embeddings = self._process_image_input(image_input)
823 - inputs_embeds = merge_multimodal_embeddings(
824 - input_ids, inputs_embeds, vision_embeddings,
825 - self.config.image_token_index)
826 - if "videos" in modalities:
827 - video_input = modalities["videos"]
828 - video_embeddings = self._process_video_pixels(video_input)
829 - inputs_embeds = merge_multimodal_embeddings(
830 - input_ids, inputs_embeds, video_embeddings,
831 - self.config.video_token_index)
826 + if intermediate_tensors is not None:
827 input_ids = None
833 - else:
828 inputs_embeds = None
829 + else:
830 + modalities = self._parse_and_validate_multimodal_inputs(**kwargs)
831 + if modalities:
832 + inputs_embeds = self.language_model.model.get_input_embeddings(
833 + input_ids)
834 + if "images" in modalities:
835 + image_input = modalities["images"]
836 + vision_embeddings = self._process_image_input(image_input)
837 + inputs_embeds = merge_multimodal_embeddings(
838 + input_ids, inputs_embeds, vision_embeddings,
839 + self.config.image_token_index)
840 + if "videos" in modalities:
841 + video_input = modalities["videos"]
842 + video_embeddings = self._process_video_pixels(video_input)
843 + inputs_embeds = merge_multimodal_embeddings(
844 + input_ids, inputs_embeds, video_embeddings,
845 + self.config.video_token_index)
846 + input_ids = None
847 + else:
848 + inputs_embeds = None
849
850 hidden_states = self.language_model.model(input_ids,
851 positions,
852 kv_caches,
853 attn_metadata,
840 - None,
854 + intermediate_tensors,
855 inputs_embeds=inputs_embeds)
856
857 return hidden_states
vllm/model_executor/models/minicpm.py
+51 -23
@@ -22,7 +22,7 @@
22 # limitations under the License.
23 """Inference-only MiniCPM model compatible with HuggingFace weights."""
24 import math
25 -from typing import Any, Dict, Iterable, List, Optional, Tuple
25 +from typing import Any, Dict, Iterable, List, Optional, Tuple, Union
26
27 import torch
28 from torch import nn
@@ -30,7 +30,7 @@ from transformers import PretrainedConfig
30
31 from vllm.attention import Attention, AttentionMetadata
32 from vllm.config import CacheConfig, LoRAConfig
33 -from vllm.distributed import (get_tensor_model_parallel_rank,
33 +from vllm.distributed import (get_pp_group, get_tensor_model_parallel_rank,
34 get_tensor_model_parallel_world_size,
35 tensor_model_parallel_all_reduce)
36 from vllm.model_executor.layers.activation import SiluAndMul
@@ -41,8 +41,7 @@ from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
41 ReplicatedLinear,
42 RowParallelLinear)
43 from vllm.model_executor.layers.logits_processor import LogitsProcessor
44 -from vllm.model_executor.layers.quantization.base_config import (
45 - QuantizationConfig)
44 +from vllm.model_executor.layers.quantization import QuantizationConfig
45 from vllm.model_executor.layers.rotary_embedding import get_rope
46 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
47 from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -52,7 +51,9 @@ from vllm.model_executor.sampling_metadata import SamplingMetadata
51 from vllm.model_executor.utils import set_weight_attrs
52 from vllm.sequence import IntermediateTensors
53
55 -from .interfaces import SupportsLoRA
54 +from .interfaces import SupportsLoRA, SupportsPP
55 +from .utils import (is_pp_missing_parameter,
56 + make_empty_intermediate_tensors_factory, make_layers)
57
58
59 class MiniCPMMoE(nn.Module):
@@ -264,7 +265,7 @@ class MiniCPMDecoderLayer(nn.Module):
265
266 def __init__(
267 self,
267 - config,
268 + config: PretrainedConfig,
269 cache_config: Optional[CacheConfig] = None,
270 quant_config: Optional[QuantizationConfig] = None,
271 ) -> None:
@@ -346,10 +347,11 @@ class MiniCPMModel(nn.Module):
347
348 def __init__(
349 self,
349 - config,
350 + config: PretrainedConfig,
351 cache_config: Optional[CacheConfig] = None,
352 quant_config: Optional[QuantizationConfig] = None,
353 lora_config: Optional[LoRAConfig] = None,
354 + prefix: str = "",
355 ) -> None:
356 super().__init__()
357 self.config = config
@@ -365,15 +367,24 @@ class MiniCPMModel(nn.Module):
367 config.hidden_size,
368 org_num_embeddings=config.vocab_size,
369 )
368 - self._init_layers()
370 + self._init_layers(prefix, config, cache_config, quant_config)
371 self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
372 + self.make_empty_intermediate_tensors = (
373 + make_empty_intermediate_tensors_factory(
374 + ["hidden_states", "residual"], self.config.hidden_size))
375
371 - def _init_layers(self):
372 - self.layers = nn.ModuleList([
373 - MiniCPMDecoderLayer(self.config, self.cache_config,
374 - self.quant_config)
375 - for _ in range(self.config.num_hidden_layers)
376 - ])
376 + def _init_layers(
377 + self,
378 + prefix: str,
379 + config: PretrainedConfig,
380 + cache_config: Optional[CacheConfig],
381 + quant_config: Optional[QuantizationConfig],
382 + ):
383 + self.start_layer, self.end_layer, self.layers = make_layers(
384 + config.num_hidden_layers,
385 + lambda prefix: MiniCPMDecoderLayer(config, cache_config,
386 + quant_config),
387 + prefix=f"{prefix}.layers")
388
389 def get_input_embeddings(self, input_ids: torch.Tensor) -> torch.Tensor:
390 embedding = self.embed_tokens(input_ids)
@@ -387,27 +398,36 @@ class MiniCPMModel(nn.Module):
398 attn_metadata: AttentionMetadata,
399 intermediate_tensors: Optional[IntermediateTensors] = None,
400 inputs_embeds: Optional[torch.Tensor] = None,
390 - ) -> torch.Tensor:
391 - if inputs_embeds is not None:
392 - hidden_states = inputs_embeds
401 + ) -> Union[torch.Tensor, IntermediateTensors]:
402 + if get_pp_group().is_first_rank:
403 + if inputs_embeds is not None:
404 + hidden_states = inputs_embeds
405 + else:
406 + hidden_states = self.get_input_embeddings(input_ids)
407 + residual = None
408 else:
394 - hidden_states = self.get_input_embeddings(input_ids)
395 - residual = None
409 + hidden_states = intermediate_tensors["hidden_states"]
410 + residual = intermediate_tensors["residual"]
411
397 - for i in range(len(self.layers)):
412 + for i in range(self.start_layer, self.end_layer):
413 layer = self.layers[i]
414 hidden_states, residual = layer(
415 positions,
416 hidden_states,
402 - kv_caches[i],
417 + kv_caches[i - self.start_layer],
418 attn_metadata,
419 residual,
420 )
421 + if not get_pp_group().is_last_rank:
422 + return IntermediateTensors({
423 + "hidden_states": hidden_states,
424 + "residual": residual
425 + })
426 hidden_states = self.norm(hidden_states)
427 return hidden_states
428
429
410 -class MiniCPMForCausalLM(nn.Module, SupportsLoRA):
430 +class MiniCPMForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
431 packed_modules_mapping = {
432 "qkv_proj": [
433 "q_proj",
@@ -470,6 +490,8 @@ class MiniCPMForCausalLM(nn.Module, SupportsLoRA):
490 self.logits_processor = LogitsProcessor(unpadded_vocab_size,
491 config.vocab_size)
492 self.sampler = Sampler()
493 + self.make_empty_intermediate_tensors = (
494 + self.model.make_empty_intermediate_tensors)
495
496 def _init_model(self):
497 self.model = MiniCPMModel(config=self.config,
@@ -484,7 +506,7 @@ class MiniCPMForCausalLM(nn.Module, SupportsLoRA):
506 kv_caches: List[torch.Tensor],
507 attn_metadata: AttentionMetadata,
508 intermediate_tensors: Optional[IntermediateTensors] = None,
487 - ) -> torch.Tensor:
509 + ) -> Union[torch.Tensor, IntermediateTensors]:
510 hidden_states = self.model(input_ids, positions, kv_caches,
511 attn_metadata, intermediate_tensors)
512 return hidden_states
@@ -548,6 +570,8 @@ class MiniCPMForCausalLM(nn.Module, SupportsLoRA):
570 # Skip loading extra bias for GPTQ models.
571 if name.endswith(".bias") and name not in params_dict:
572 continue
573 + if is_pp_missing_parameter(name, self):
574 + continue
575 param = params_dict[name]
576 weight_loader = param.weight_loader
577 weight_loader(param, loaded_weight, shard_id)
@@ -557,6 +581,8 @@ class MiniCPMForCausalLM(nn.Module, SupportsLoRA):
581 if weight_name not in name:
582 continue
583 name = name.replace(weight_name, param_name)
584 + if is_pp_missing_parameter(name, self):
585 + continue
586 param = params_dict[name]
587 weight_loader = param.weight_loader
588 weight_loader(param,
@@ -568,6 +594,8 @@ class MiniCPMForCausalLM(nn.Module, SupportsLoRA):
594 # Skip loading extra bias for GPTQ models.
595 if name.endswith(".bias") and name not in params_dict:
596 continue
597 + if is_pp_missing_parameter(name, self):
598 + continue
599 param = params_dict[name]
600 weight_loader = getattr(param, "weight_loader",
601 default_weight_loader)
vllm/model_executor/models/minicpm3.py
+17 -9
@@ -26,6 +26,7 @@ from typing import Any, Dict, Optional
26
27 import torch
28 from torch import nn
29 +from transformers import PretrainedConfig
30
31 from vllm.attention import Attention, AttentionMetadata
32 from vllm.config import CacheConfig
@@ -34,19 +35,20 @@ from vllm.model_executor.layers.layernorm import RMSNorm
35 from vllm.model_executor.layers.linear import (ColumnParallelLinear,
36 ReplicatedLinear,
37 RowParallelLinear)
37 -from vllm.model_executor.layers.quantization.base_config import (
38 - QuantizationConfig)
38 +from vllm.model_executor.layers.quantization import QuantizationConfig
39 from vllm.model_executor.layers.rotary_embedding import get_rope
40 from vllm.model_executor.models.minicpm import (MiniCPMDecoderLayer,
41 MiniCPMForCausalLM,
42 MiniCPMModel)
43
44 +from .utils import make_layers
45 +
46
47 class MiniCPM3Attention(nn.Module):
48
49 def __init__(
50 self,
49 - config,
51 + config: PretrainedConfig,
52 hidden_size: int,
53 num_heads: int,
54 qk_nope_head_dim: int,
@@ -199,12 +201,18 @@ class MiniCPM3DecoderLayer(MiniCPMDecoderLayer):
201
202 class MiniCPM3Model(MiniCPMModel):
203
202 - def _init_layers(self):
203 - self.layers = nn.ModuleList([
204 - MiniCPM3DecoderLayer(self.config, self.cache_config,
205 - self.quant_config)
206 - for _ in range(self.config.num_hidden_layers)
207 - ])
204 + def _init_layers(
205 + self,
206 + prefix: str,
207 + config: PretrainedConfig,
208 + cache_config: Optional[CacheConfig],
209 + quant_config: Optional[QuantizationConfig],
210 + ):
211 + self.start_layer, self.end_layer, self.layers = make_layers(
212 + config.num_hidden_layers,
213 + lambda prefix: MiniCPM3DecoderLayer(config, cache_config,
214 + quant_config),
215 + prefix=f"{prefix}.layers")
216
217
218 class MiniCPM3ForCausalLM(MiniCPMForCausalLM):
vllm/model_executor/models/minicpmv.py
+16 -5
@@ -45,7 +45,6 @@ from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
45 from vllm.model_executor.layers.vocab_parallel_embedding import ParallelLMHead
46 from vllm.model_executor.model_loader.utils import set_default_torch_dtype
47 from vllm.model_executor.model_loader.weight_utils import default_weight_loader
48 -from vllm.model_executor.models.interfaces import SupportsMultiModal
48 from vllm.model_executor.models.llama import LlamaModel
49 from vllm.model_executor.models.minicpm import MiniCPMModel
50 from vllm.model_executor.models.module_mapping import MultiModelKeys
@@ -59,7 +58,8 @@ from vllm.multimodal.utils import cached_get_tokenizer
58 from vllm.sequence import IntermediateTensors, SequenceData
59
60 from .idefics2_vision_model import Idefics2VisionTransformer
62 -from .interfaces import SupportsLoRA
61 +from .interfaces import SupportsLoRA, SupportsMultiModal, SupportsPP
62 +from .utils import is_pp_missing_parameter
63
64 _KEYS_TO_MODIFY_MAPPING = {
65 "llm.lm_head": "lm_head",
@@ -337,7 +337,7 @@ def input_mapper_for_minicpmv(ctx: InputContext, data: object):
337 return MultiModalInputs(batch_data)
338
339
340 -class MiniCPMVBaseModel(nn.Module, SupportsMultiModal):
340 +class MiniCPMVBaseModel(nn.Module, SupportsMultiModal, SupportsPP):
341 """
342 The abstract class of MiniCPMV can only be inherited, but cannot be
343 instantiated.
@@ -374,6 +374,9 @@ class MiniCPMVBaseModel(nn.Module, SupportsMultiModal):
374 self.logits_processor = LogitsProcessor(config.vocab_size)
375 self.sampler = Sampler()
376
377 + self.make_empty_intermediate_tensors = (
378 + self.llm.make_empty_intermediate_tensors)
379 +
380 def get_embedding(
381 self,
382 input_ids: torch.Tensor,
@@ -498,9 +501,12 @@ class MiniCPMVBaseModel(nn.Module, SupportsMultiModal):
501 intermediate_tensors: Optional[IntermediateTensors] = None,
502 **kwargs: Any,
503 ) -> torch.Tensor:
501 - image_inputs = self._parse_and_validate_inputs(input_ids, **kwargs)
504 + if intermediate_tensors is not None:
505 + vlm_embeddings = None
506 + else:
507 + image_inputs = self._parse_and_validate_inputs(input_ids, **kwargs)
508
503 - vlm_embeddings, _ = self.get_embedding(input_ids, image_inputs)
509 + vlm_embeddings, _ = self.get_embedding(input_ids, image_inputs)
510
511 output = self.llm(
512 input_ids=None,
@@ -557,6 +563,9 @@ class MiniCPMVBaseModel(nn.Module, SupportsMultiModal):
563 for param_name, weight_name, shard_id in stacked_params_mapping:
564 if weight_name not in name:
565 continue
566 + if is_pp_missing_parameter(
567 + name.replace(weight_name, param_name), self):
568 + continue
569 param = params_dict[name.replace(weight_name, param_name)]
570 weight_loader = param.weight_loader
571 weight_loader(param, loaded_weight, shard_id)
@@ -564,6 +573,8 @@ class MiniCPMVBaseModel(nn.Module, SupportsMultiModal):
573 else:
574 use_default_weight_loading = True
575 if use_default_weight_loading:
576 + if is_pp_missing_parameter(name, self):
577 + continue
578 param = params_dict[name]
579 weight_loader = getattr(param, "weight_loader",
580 default_weight_loader)
vllm/model_executor/models/mixtral.py
+13 -22
@@ -21,7 +21,7 @@
21 # See the License for the specific language governing permissions and
22 # limitations under the License.
23 """Inference-only Mixtral model."""
24 -from typing import Iterable, List, Optional, Tuple
24 +from typing import Iterable, List, Optional, Tuple, Union
25
26 import torch
27 from torch import nn
@@ -36,8 +36,7 @@ from vllm.model_executor.layers.linear import (QKVParallelLinear,
36 ReplicatedLinear,
37 RowParallelLinear)
38 from vllm.model_executor.layers.logits_processor import LogitsProcessor
39 -from vllm.model_executor.layers.quantization.base_config import (
40 - QuantizationConfig)
39 +from vllm.model_executor.layers.quantization import QuantizationConfig
40 from vllm.model_executor.layers.rotary_embedding import get_rope
41 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
42 from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -47,8 +46,9 @@ from vllm.model_executor.model_loader.weight_utils import (
46 from vllm.model_executor.sampling_metadata import SamplingMetadata
47 from vllm.sequence import IntermediateTensors
48
50 -from .interfaces import SupportsLoRA
51 -from .utils import is_pp_missing_parameter, make_layers
49 +from .interfaces import SupportsLoRA, SupportsPP
50 +from .utils import (is_pp_missing_parameter,
51 + make_empty_intermediate_tensors_factory, make_layers)
52
53
54 class MixtralMoE(nn.Module):
@@ -276,6 +276,9 @@ class MixtralModel(nn.Module):
276 prefix=f"{prefix}.layers")
277
278 self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
279 + self.make_empty_intermediate_tensors = (
280 + make_empty_intermediate_tensors_factory(
281 + ["hidden_states", "residual"], config.hidden_size))
282
283 def forward(
284 self,
@@ -284,7 +287,7 @@ class MixtralModel(nn.Module):
287 kv_caches: List[torch.Tensor],
288 attn_metadata: AttentionMetadata,
289 intermediate_tensors: Optional[IntermediateTensors],
287 - ) -> torch.Tensor:
290 + ) -> Union[torch.Tensor, IntermediateTensors]:
291 if get_pp_group().is_first_rank:
292 hidden_states = self.embed_tokens(input_ids)
293 residual = None
@@ -306,7 +309,7 @@ class MixtralModel(nn.Module):
309 return hidden_states
310
311
309 -class MixtralForCausalLM(nn.Module, SupportsLoRA):
312 +class MixtralForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
313 fall_back_to_pt_during_load = False
314
315 packed_modules_mapping = {
@@ -365,6 +368,8 @@ class MixtralForCausalLM(nn.Module, SupportsLoRA):
368 self.logits_processor = LogitsProcessor(self.unpadded_vocab_size,
369 config.vocab_size)
370 self.sampler = Sampler()
371 + self.make_empty_intermediate_tensors = (
372 + self.model.make_empty_intermediate_tensors)
373
374 def forward(
375 self,
@@ -373,7 +378,7 @@ class MixtralForCausalLM(nn.Module, SupportsLoRA):
378 kv_caches: List[torch.Tensor],
379 attn_metadata: AttentionMetadata,
380 intermediate_tensors: Optional[IntermediateTensors] = None,
376 - ) -> torch.Tensor:
381 + ) -> Union[torch.Tensor, IntermediateTensors]:
382 hidden_states = self.model(input_ids, positions, kv_caches,
383 attn_metadata, intermediate_tensors)
384 return hidden_states
@@ -387,20 +392,6 @@ class MixtralForCausalLM(nn.Module, SupportsLoRA):
392 sampling_metadata)
393 return logits
394
390 - def make_empty_intermediate_tensors(
391 - self, batch_size: int, dtype: torch.dtype,
392 - device: torch.device) -> IntermediateTensors:
393 - return IntermediateTensors({
394 - "hidden_states":
395 - torch.zeros((batch_size, self.config.hidden_size),
396 - dtype=dtype,
397 - device=device),
398 - "residual":
399 - torch.zeros((batch_size, self.config.hidden_size),
400 - dtype=dtype,
401 - device=device),
402 - })
403 -
395 def sample(
396 self,
397 logits: Optional[torch.Tensor],
vllm/model_executor/models/mixtral_quant.py
+42 -19
@@ -21,7 +21,7 @@
21 # See the License for the specific language governing permissions and
22 # limitations under the License.
23 """Inference-only Mixtral model."""
24 -from typing import Iterable, List, Optional, Tuple
24 +from typing import Iterable, List, Optional, Tuple, Union
25
26 import numpy as np
27 import torch
@@ -31,7 +31,7 @@ from transformers import MixtralConfig
31
32 from vllm.attention import Attention, AttentionMetadata
33 from vllm.config import CacheConfig
34 -from vllm.distributed import (get_tensor_model_parallel_rank,
34 +from vllm.distributed import (get_pp_group, get_tensor_model_parallel_rank,
35 get_tensor_model_parallel_world_size,
36 tensor_model_parallel_all_reduce)
37 from vllm.model_executor.layers.layernorm import RMSNorm
@@ -39,8 +39,7 @@ from vllm.model_executor.layers.linear import (QKVParallelLinear,
39 ReplicatedLinear,
40 RowParallelLinear)
41 from vllm.model_executor.layers.logits_processor import LogitsProcessor
42 -from vllm.model_executor.layers.quantization.base_config import (
43 - QuantizationConfig)
42 +from vllm.model_executor.layers.quantization import QuantizationConfig
43 from vllm.model_executor.layers.rotary_embedding import get_rope
44 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
45 from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -49,6 +48,10 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
48 from vllm.model_executor.sampling_metadata import SamplingMetadata
49 from vllm.sequence import IntermediateTensors
50
51 +from .interfaces import SupportsPP
52 +from .utils import (is_pp_missing_parameter,
53 + make_empty_intermediate_tensors_factory, make_layers)
54 +
55
56 class MixtralMLP(nn.Module):
57
@@ -296,6 +299,7 @@ class MixtralModel(nn.Module):
299 config: MixtralConfig,
300 cache_config: Optional[CacheConfig] = None,
301 quant_config: Optional[QuantizationConfig] = None,
302 + prefix: str = "",
303 ) -> None:
304 super().__init__()
305 self.padding_idx = config.pad_token_id
@@ -305,13 +309,15 @@ class MixtralModel(nn.Module):
309 config.vocab_size,
310 config.hidden_size,
311 )
308 - self.layers = nn.ModuleList([
309 - MixtralDecoderLayer(config,
310 - cache_config,
311 - quant_config=quant_config)
312 - for _ in range(config.num_hidden_layers)
313 - ])
312 + self.start_layer, self.end_layer, self.layers = make_layers(
313 + config.num_hidden_layers,
314 + lambda prefix: MixtralDecoderLayer(
315 + config, cache_config, quant_config=quant_config),
316 + prefix=f"{prefix}.layers")
317 self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
318 + self.make_empty_intermediate_tensors = (
319 + make_empty_intermediate_tensors_factory(
320 + ["hidden_states", "residual"], config.hidden_size))
321
322 def forward(
323 self,
@@ -319,19 +325,30 @@ class MixtralModel(nn.Module):
325 positions: torch.Tensor,
326 kv_caches: List[torch.Tensor],
327 attn_metadata: AttentionMetadata,
322 - ) -> torch.Tensor:
323 - hidden_states = self.embed_tokens(input_ids)
324 - residual = None
325 - for i in range(len(self.layers)):
328 + intermediate_tensors: Optional[IntermediateTensors],
329 + ) -> Union[torch.Tensor, IntermediateTensors]:
330 + if get_pp_group().is_first_rank:
331 + hidden_states = self.embed_tokens(input_ids)
332 + residual = None
333 + else:
334 + assert intermediate_tensors is not None
335 + hidden_states = intermediate_tensors["hidden_states"]
336 + residual = intermediate_tensors["residual"]
337 + for i in range(self.start_layer, self.end_layer):
338 layer = self.layers[i]
339 hidden_states, residual = layer(positions, hidden_states,
328 - kv_caches[i], attn_metadata,
329 - residual)
340 + kv_caches[i - self.start_layer],
341 + attn_metadata, residual)
342 + if not get_pp_group().is_last_rank:
343 + return IntermediateTensors({
344 + "hidden_states": hidden_states,
345 + "residual": residual
346 + })
347 hidden_states, _ = self.norm(hidden_states, residual)
348 return hidden_states
349
350
334 -class MixtralForCausalLM(nn.Module):
351 +class MixtralForCausalLM(nn.Module, SupportsPP):
352 fall_back_to_pt_during_load = False
353
354 def __init__(
@@ -351,6 +368,8 @@ class MixtralForCausalLM(nn.Module):
368 self.lm_head.weight = self.model.embed_tokens.weight
369 self.logits_processor = LogitsProcessor(config.vocab_size)
370 self.sampler = Sampler()
371 + self.make_empty_intermediate_tensors = (
372 + self.model.make_empty_intermediate_tensors)
373
374 def forward(
375 self,
@@ -359,9 +378,9 @@ class MixtralForCausalLM(nn.Module):
378 kv_caches: List[torch.Tensor],
379 attn_metadata: AttentionMetadata,
380 intermediate_tensors: Optional[IntermediateTensors] = None,
362 - ) -> torch.Tensor:
381 + ) -> Union[torch.Tensor, IntermediateTensors]:
382 hidden_states = self.model(input_ids, positions, kv_caches,
364 - attn_metadata)
383 + attn_metadata, intermediate_tensors)
384 return hidden_states
385
386 def compute_logits(
@@ -400,6 +419,8 @@ class MixtralForCausalLM(nn.Module):
419 # Skip loading extra bias for GPTQ models.
420 if name.endswith(".bias") and name not in params_dict:
421 continue
422 + if is_pp_missing_parameter(name, self):
423 + continue
424 param = params_dict[name]
425 weight_loader = param.weight_loader
426 weight_loader(param, loaded_weight, shard_id)
@@ -412,6 +433,8 @@ class MixtralForCausalLM(nn.Module):
433 if ("block_sparse_moe.experts." in name
434 and name not in params_dict):
435 continue
436 + if is_pp_missing_parameter(name, self):
437 + continue
438 param = params_dict[name]
439 weight_loader = getattr(param, "weight_loader",
440 default_weight_loader)
vllm/model_executor/models/mpt.py
+34 -15
@@ -1,22 +1,21 @@
1 # coding=utf-8
2 # Adapted from https://huggingface.co/mosaicml/mpt-7b/tree/main
3 import math
4 -from typing import Iterable, List, Optional, Tuple
4 +from typing import Iterable, List, Optional, Tuple, Union
5
6 import torch
7 import torch.nn as nn
8
9 from vllm.attention import Attention, AttentionMetadata
10 from vllm.config import CacheConfig
11 -from vllm.distributed import (get_tensor_model_parallel_rank,
11 +from vllm.distributed import (get_pp_group, get_tensor_model_parallel_rank,
12 get_tensor_model_parallel_world_size)
13 from vllm.model_executor.layers.activation import get_act_fn
14 from vllm.model_executor.layers.linear import (ColumnParallelLinear,
15 QKVParallelLinear,
16 RowParallelLinear)
17 from vllm.model_executor.layers.logits_processor import LogitsProcessor
18 -from vllm.model_executor.layers.quantization.base_config import (
19 - QuantizationConfig)
18 +from vllm.model_executor.layers.quantization import QuantizationConfig
19 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
20 from vllm.model_executor.layers.vocab_parallel_embedding import (
21 VocabParallelEmbedding)
@@ -25,6 +24,10 @@ from vllm.model_executor.sampling_metadata import SamplingMetadata
24 from vllm.sequence import IntermediateTensors
25 from vllm.transformers_utils.configs.mpt import MPTConfig
26
27 +from .interfaces import SupportsPP
28 +from .utils import (is_pp_missing_parameter,
29 + make_empty_intermediate_tensors_factory, make_layers)
30 +
31
32 def _get_alibi_slopes(
33 total_num_heads: int,
@@ -208,6 +211,7 @@ class MPTModel(nn.Module):
211 config: MPTConfig,
212 cache_config: Optional[CacheConfig] = None,
213 quant_config: Optional[QuantizationConfig] = None,
214 + prefix: str = "",
215 ):
216 super().__init__()
217 assert config.embedding_fraction == 1.0
@@ -217,10 +221,10 @@ class MPTModel(nn.Module):
221 config.vocab_size,
222 config.d_model,
223 )
220 - self.blocks = nn.ModuleList([
221 - MPTBlock(config, cache_config, quant_config)
222 - for _ in range(config.n_layers)
223 - ])
224 + self.start_layer, self.end_layer, self.blocks = make_layers(
225 + config.n_layers,
226 + lambda prefix: MPTBlock(config, cache_config, quant_config),
227 + prefix=f"{prefix}.blocks")
228 self.norm_f = nn.LayerNorm(config.d_model)
229 if config.no_bias:
230 for module in self.modules():
@@ -228,6 +232,9 @@ class MPTModel(nn.Module):
232 module.bias, nn.Parameter):
233 # Remove the bias term in Linear and LayerNorm.
234 module.register_parameter("bias", None)
235 + self.make_empty_intermediate_tensors = (
236 + make_empty_intermediate_tensors_factory(["hidden_states"],
237 + config.d_model))
238
239 def forward(
240 self,
@@ -235,21 +242,29 @@ class MPTModel(nn.Module):
242 position_ids: torch.Tensor,
243 kv_caches: List[torch.Tensor],
244 attn_metadata: AttentionMetadata,
238 - ) -> torch.Tensor:
239 - hidden_states = self.wte(input_ids)
240 - for i in range(len(self.blocks)):
245 + intermediate_tensors: Optional[IntermediateTensors],
246 + ) -> Union[torch.Tensor, IntermediateTensors]:
247 + if get_pp_group().is_first_rank:
248 + hidden_states = self.wte(input_ids)
249 + else:
250 + assert intermediate_tensors is not None
251 + hidden_states = intermediate_tensors["hidden_states"]
252 +
253 + for i in range(self.start_layer, self.end_layer):
254 block = self.blocks[i]
255 hidden_states = block(
256 position_ids,
257 hidden_states,
245 - kv_caches[i],
258 + kv_caches[i - self.start_layer],
259 attn_metadata,
260 )
261 + if not get_pp_group().is_last_rank:
262 + return IntermediateTensors({"hidden_states": hidden_states})
263 hidden_states = self.norm_f(hidden_states)
264 return hidden_states
265
266
252 -class MPTForCausalLM(nn.Module):
267 +class MPTForCausalLM(nn.Module, SupportsPP):
268
269 def __init__(
270 self,
@@ -266,6 +281,8 @@ class MPTForCausalLM(nn.Module):
281 self.lm_head = self.transformer.wte
282 self.logits_processor = LogitsProcessor(config.vocab_size)
283 self.sampler = Sampler()
284 + self.make_empty_intermediate_tensors = (
285 + self.transformer.make_empty_intermediate_tensors)
286
287 def forward(
288 self,
@@ -274,9 +291,9 @@ class MPTForCausalLM(nn.Module):
291 kv_caches: List[torch.Tensor],
292 attn_metadata: AttentionMetadata,
293 intermediate_tensors: Optional[IntermediateTensors] = None,
277 - ) -> torch.Tensor:
294 + ) -> Union[torch.Tensor, IntermediateTensors]:
295 hidden_states = self.transformer(input_ids, positions, kv_caches,
279 - attn_metadata)
296 + attn_metadata, intermediate_tensors)
297 return hidden_states
298
299 def compute_logits(
@@ -302,6 +319,8 @@ class MPTForCausalLM(nn.Module):
319 # Skip loading extra bias for GPTQ models.
320 if name.endswith(".bias") and name not in params_dict:
321 continue
322 + if is_pp_missing_parameter(name, self):
323 + continue
324 param = params_dict[name]
325 weight_loader = getattr(param, "weight_loader",
326 default_weight_loader)
vllm/model_executor/models/nemotron.py
+10 -19
@@ -34,8 +34,7 @@ from vllm.model_executor.layers.linear import (ColumnParallelLinear,
34 QKVParallelLinear,
35 RowParallelLinear)
36 from vllm.model_executor.layers.logits_processor import LogitsProcessor
37 -from vllm.model_executor.layers.quantization.base_config import (
38 - QuantizationConfig)
37 +from vllm.model_executor.layers.quantization import QuantizationConfig
38 from vllm.model_executor.layers.rotary_embedding import get_rope
39 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
40 from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -46,8 +45,9 @@ from vllm.model_executor.sampling_metadata import SamplingMetadata
45 from vllm.sequence import IntermediateTensors
46 from vllm.transformers_utils.configs import NemotronConfig
47
49 -from .interfaces import SupportsLoRA
50 -from .utils import PPMissingLayer, is_pp_missing_parameter, make_layers
48 +from .interfaces import SupportsLoRA, SupportsPP
49 +from .utils import (PPMissingLayer, is_pp_missing_parameter,
50 + make_empty_intermediate_tensors_factory, make_layers)
51
52 # The architecture is pretty similar to Llama, with these changes:
53 # - There is no gate_proj, just up_proj
@@ -328,6 +328,9 @@ class NemotronModel(nn.Module):
328 eps=config.norm_eps)
329 else:
330 self.norm = PPMissingLayer()
331 + self.make_empty_intermediate_tensors = (
332 + make_empty_intermediate_tensors_factory(
333 + ["hidden_states", "residual"], config.hidden_size))
334
335 def get_input_embeddings(self, input_ids: torch.Tensor) -> torch.Tensor:
336 return self.embed_tokens(input_ids)
@@ -372,7 +375,7 @@ class NemotronModel(nn.Module):
375 return hidden_states
376
377
375 -class NemotronForCausalLM(nn.Module, SupportsLoRA):
378 +class NemotronForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
379 packed_modules_mapping = {
380 "qkv_proj": [
381 "q_proj",
@@ -440,6 +443,8 @@ class NemotronForCausalLM(nn.Module, SupportsLoRA):
443 self.sampler = Sampler()
444 else:
445 self.lm_head = PPMissingLayer()
446 + self.make_empty_intermediate_tensors = (
447 + self.model.make_empty_intermediate_tensors)
448
449 def forward(
450 self,
@@ -470,20 +475,6 @@ class NemotronForCausalLM(nn.Module, SupportsLoRA):
475 next_tokens = self.sampler(logits, sampling_metadata)
476 return next_tokens
477
473 - def make_empty_intermediate_tensors(
474 - self, batch_size: int, dtype: torch.dtype,
475 - device: torch.device) -> IntermediateTensors:
476 - return IntermediateTensors({
477 - "hidden_states":
478 - torch.zeros((batch_size, self.config.hidden_size),
479 - dtype=dtype,
480 - device=device),
481 - "residual":
482 - torch.zeros((batch_size, self.config.hidden_size),
483 - dtype=dtype,
484 - device=device),
485 - })
486 -
478 def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
479 stacked_params_mapping = [
480 # (param_name, shard_name, shard_id)
vllm/model_executor/models/olmo.py
+42 -20
@@ -21,7 +21,7 @@
21 # See the License for the specific language governing permissions and
22 # limitations under the License.
23 """Inference-only OLMo model compatible with HuggingFace weights."""
24 -from typing import Iterable, List, Optional, Tuple
24 +from typing import Iterable, List, Optional, Tuple, Union
25
26 import torch
27 from torch import nn
@@ -29,14 +29,13 @@ from transformers import OlmoConfig
29
30 from vllm.attention import Attention, AttentionMetadata
31 from vllm.config import CacheConfig
32 -from vllm.distributed import get_tensor_model_parallel_world_size
32 +from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
33 from vllm.model_executor.layers.activation import SiluAndMul
34 from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
35 QKVParallelLinear,
36 RowParallelLinear)
37 from vllm.model_executor.layers.logits_processor import LogitsProcessor
38 -from vllm.model_executor.layers.quantization.base_config import (
39 - QuantizationConfig)
38 +from vllm.model_executor.layers.quantization import QuantizationConfig
39 from vllm.model_executor.layers.rotary_embedding import get_rope
40 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
41 from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -45,6 +44,10 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
44 from vllm.model_executor.sampling_metadata import SamplingMetadata
45 from vllm.sequence import IntermediateTensors
46
47 +from .interfaces import SupportsPP
48 +from .utils import (is_pp_missing_parameter,
49 + make_empty_intermediate_tensors_factory, make_layers)
50 +
51
52 class OlmoAttention(nn.Module):
53 """
@@ -223,19 +226,24 @@ class OlmoModel(nn.Module):
226 def __init__(self,
227 config: OlmoConfig,
228 cache_config: Optional[CacheConfig] = None,
226 - quant_config: Optional[QuantizationConfig] = None):
229 + quant_config: Optional[QuantizationConfig] = None,
230 + prefix: str = ""):
231 super().__init__()
232 self.config = config
233
234 self.embed_tokens = VocabParallelEmbedding(config.vocab_size,
235 config.hidden_size)
232 - self.layers = nn.ModuleList([
233 - OlmoDecoderLayer(config, cache_config, quant_config)
234 - for layer_idx in range(config.num_hidden_layers)
235 - ])
236 + self.start_layer, self.end_layer, self.layers = make_layers(
237 + config.num_hidden_layers,
238 + lambda prefix: OlmoDecoderLayer(config, cache_config, quant_config
239 + ),
240 + prefix=f"{prefix}.layers")
241 self.norm = nn.LayerNorm(config.hidden_size,
242 elementwise_affine=False,
243 bias=False)
244 + self.make_empty_intermediate_tensors = (
245 + make_empty_intermediate_tensors_factory(["hidden_states"],
246 + config.hidden_size))
247
248 def forward(
249 self,
@@ -243,34 +251,41 @@ class OlmoModel(nn.Module):
251 positions: torch.Tensor,
252 kv_caches: List[torch.Tensor],
253 attn_metadata: AttentionMetadata,
246 - ) -> torch.Tensor:
254 + intermediate_tensors: Optional[IntermediateTensors],
255 + ) -> Union[torch.Tensor, IntermediateTensors]:
256 """
257 :param input_ids: A tensor of shape `(batch_size, seq_len)`.
258 """
250 - # Get embeddings of input.
251 - # shape: (batch_size, seq_len, d_model)
252 - inputs_embeds = self.embed_tokens(input_ids)
259 + if get_pp_group().is_first_rank:
260 + # Get embeddings of input.
261 + # shape: (batch_size, seq_len, d_model)
262 + inputs_embeds = self.embed_tokens(input_ids)
263
254 - # embed positions
255 - hidden_states = inputs_embeds
264 + # embed positions
265 + hidden_states = inputs_embeds
266 + else:
267 + assert intermediate_tensors is not None
268 + hidden_states = intermediate_tensors["hidden_states"]
269
270 # Apply blocks one-by-one.
258 - for layer_idx, decoder_layer in enumerate(self.layers):
271 + for i in range(self.start_layer, self.end_layer):
272 # shape: (batch_size, seq_len, d_model)
260 - hidden_states = decoder_layer(
273 + hidden_states = self.layers[i](
274 positions,
275 hidden_states,
263 - kv_caches[layer_idx],
276 + kv_caches[i - self.start_layer],
277 attn_metadata,
278 )
279
280 + if not get_pp_group().is_last_rank:
281 + return IntermediateTensors({"hidden_states": hidden_states})
282 # Apply final layer norm.
283 # shape: (batch_size, seq_len or 1, d_model)
284 hidden_states = self.norm(hidden_states)
285 return hidden_states
286
287
273 -class OlmoForCausalLM(nn.Module):
288 +class OlmoForCausalLM(nn.Module, SupportsPP):
289 """
290 Extremely barebones HF model wrapper.
291 """
@@ -294,6 +309,8 @@ class OlmoForCausalLM(nn.Module):
309 )
310 self.logits_processor = LogitsProcessor(config.vocab_size)
311 self.sampler = Sampler()
312 + self.make_empty_intermediate_tensors = (
313 + self.model.make_empty_intermediate_tensors)
314
315 def forward(
316 self,
@@ -302,12 +319,13 @@ class OlmoForCausalLM(nn.Module):
319 kv_caches: List[torch.Tensor],
320 attn_metadata: AttentionMetadata,
321 intermediate_tensors: Optional[IntermediateTensors] = None,
305 - ) -> torch.Tensor:
322 + ) -> Union[torch.Tensor, IntermediateTensors]:
323 hidden_states = self.model(
324 input_ids=input_ids,
325 positions=positions,
326 kv_caches=kv_caches,
327 attn_metadata=attn_metadata,
328 + intermediate_tensors=intermediate_tensors,
329 )
330 return hidden_states
331
@@ -358,6 +376,8 @@ class OlmoForCausalLM(nn.Module):
376 # Skip loading extra bias for GPTQ models.
377 if name.endswith(".bias") and name not in params_dict:
378 continue
379 + if is_pp_missing_parameter(name, self):
380 + continue
381 param = params_dict[name]
382 weight_loader = param.weight_loader
383 weight_loader(param, loaded_weight, shard_id)
@@ -366,6 +386,8 @@ class OlmoForCausalLM(nn.Module):
386 # Skip loading extra bias for GPTQ models.
387 if name.endswith(".bias") and name not in params_dict:
388 continue
389 + if is_pp_missing_parameter(name, self):
390 + continue
391 param = params_dict[name]
392 weight_loader = getattr(param, "weight_loader",
393 default_weight_loader)
vllm/model_executor/models/olmoe.py
+57 -21
@@ -10,7 +10,7 @@
10 # See the License for the specific language governing permissions and
11 # limitations under the License.
12 """Inference-only OLMoE model compatible with HuggingFace weights."""
13 -from typing import Any, Dict, Iterable, List, Optional, Tuple
13 +from typing import Any, Dict, Iterable, List, Optional, Tuple, Union
14
15 import torch
16 from torch import nn
@@ -18,15 +18,14 @@ from transformers import PretrainedConfig
18
19 from vllm.attention import Attention, AttentionMetadata
20 from vllm.config import CacheConfig
21 -from vllm.distributed import get_tensor_model_parallel_world_size
21 +from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
22 from vllm.model_executor.layers.fused_moe import FusedMoE
23 from vllm.model_executor.layers.layernorm import RMSNorm
24 from vllm.model_executor.layers.linear import (QKVParallelLinear,
25 ReplicatedLinear,
26 RowParallelLinear)
27 from vllm.model_executor.layers.logits_processor import LogitsProcessor
28 -from vllm.model_executor.layers.quantization.base_config import (
29 - QuantizationConfig)
28 +from vllm.model_executor.layers.quantization import QuantizationConfig
29 from vllm.model_executor.layers.rotary_embedding import get_rope
30 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
31 from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -36,6 +35,10 @@ from vllm.model_executor.sampling_metadata import SamplingMetadata
35 from vllm.sequence import IntermediateTensors
36 from vllm.utils import print_warning_once
37
38 +from .interfaces import SupportsPP
39 +from .utils import (is_pp_missing_parameter,
40 + make_empty_intermediate_tensors_factory, make_layers)
41 +
42
43 class OlmoeMoE(nn.Module):
44 """A tensor-parallel MoE implementation for Olmoe that shards each expert
@@ -243,6 +246,7 @@ class OlmoeModel(nn.Module):
246 config: PretrainedConfig,
247 cache_config: Optional[CacheConfig] = None,
248 quant_config: Optional[QuantizationConfig] = None,
249 + prefix: str = "",
250 ) -> None:
251 super().__init__()
252 self.padding_idx = config.pad_token_id
@@ -252,34 +256,54 @@ class OlmoeModel(nn.Module):
256 config.vocab_size,
257 config.hidden_size,
258 )
255 - self.layers = nn.ModuleList([
256 - OlmoeDecoderLayer(config,
257 - layer_idx,
258 - cache_config,
259 - quant_config=quant_config)
260 - for layer_idx in range(config.num_hidden_layers)
261 - ])
259 + self.start_layer, self.end_layer, self.layers = make_layers(
260 + config.num_hidden_layers,
261 + lambda prefix: OlmoeDecoderLayer(config, int(
262 + prefix.split(".")[-1]), cache_config, quant_config),
263 + prefix=f"{prefix}.layers")
264 self.norm = RMSNorm(config.hidden_size, eps=1e-5)
265
266 + self.make_empty_intermediate_tensors = (
267 + make_empty_intermediate_tensors_factory(
268 + ["hidden_states", "residual"], config.hidden_size))
269 +
270 def forward(
271 self,
272 input_ids: torch.Tensor,
273 positions: torch.Tensor,
274 kv_caches: List[torch.Tensor],
275 attn_metadata: AttentionMetadata,
270 - ) -> torch.Tensor:
271 - hidden_states = self.embed_tokens(input_ids)
272 - residual = None
273 - for i in range(len(self.layers)):
276 + intermediate_tensors: Optional[IntermediateTensors],
277 + ) -> Union[torch.Tensor, IntermediateTensors]:
278 + if get_pp_group().is_first_rank:
279 + hidden_states = self.embed_tokens(input_ids)
280 + residual = None
281 + else:
282 + assert intermediate_tensors is not None
283 + hidden_states = intermediate_tensors["hidden_states"]
284 + residual = intermediate_tensors["residual"]
285 +
286 + for i in range(self.start_layer, self.end_layer):
287 layer = self.layers[i]
275 - hidden_states, residual = layer(positions, hidden_states,
276 - kv_caches[i], attn_metadata,
277 - residual)
288 + hidden_states, residual = layer(
289 + positions,
290 + hidden_states,
291 + kv_caches[i - self.start_layer],
292 + attn_metadata,
293 + residual,
294 + )
295 +
296 + if not get_pp_group().is_last_rank:
297 + return IntermediateTensors({
298 + "hidden_states": hidden_states,
299 + "residual": residual
300 + })
301 +
302 hidden_states, _ = self.norm(hidden_states, residual)
303 return hidden_states
304
305
282 -class OlmoeForCausalLM(nn.Module):
306 +class OlmoeForCausalLM(nn.Module, SupportsPP):
307
308 fall_back_to_pt_during_load = False
309
@@ -299,6 +323,9 @@ class OlmoeForCausalLM(nn.Module):
323 self.logits_processor = LogitsProcessor(config.vocab_size)
324 self.sampler = Sampler()
325
326 + self.make_empty_intermediate_tensors = (
327 + self.model.make_empty_intermediate_tensors)
328 +
329 def forward(
330 self,
331 input_ids: torch.Tensor,
@@ -306,9 +333,9 @@ class OlmoeForCausalLM(nn.Module):
333 kv_caches: List[torch.Tensor],
334 attn_metadata: AttentionMetadata,
335 intermediate_tensors: Optional[IntermediateTensors] = None,
309 - ) -> torch.Tensor:
336 + ) -> Union[torch.Tensor, IntermediateTensors]:
337 hidden_states = self.model(input_ids, positions, kv_caches,
311 - attn_metadata)
338 + attn_metadata, intermediate_tensors)
339 return hidden_states
340
341 def compute_logits(self, hidden_states: torch.Tensor,
@@ -363,6 +390,9 @@ class OlmoeForCausalLM(nn.Module):
390 # Skip loading extra bias for GPTQ models.
391 if name.endswith(".bias") and name not in params_dict:
392 continue
393 + # Skip layers on other devices.
394 + if is_pp_missing_parameter(name, self):
395 + continue
396 if name not in params_dict:
397 continue
398
@@ -376,6 +406,9 @@ class OlmoeForCausalLM(nn.Module):
406 if weight_name not in name:
407 continue
408 name = name.replace(weight_name, param_name)
409 + # Skip layers on other devices.
410 + if is_pp_missing_parameter(name, self):
411 + continue
412 param = params_dict[name]
413 weight_loader = param.weight_loader
414 weight_loader(param,
@@ -388,6 +421,9 @@ class OlmoeForCausalLM(nn.Module):
421 # Skip loading extra bias for GPTQ models.
422 if name.endswith(".bias") and name not in params_dict:
423 continue
424 + # Skip layers on other devices.
425 + if is_pp_missing_parameter(name, self):
426 + continue
427 # Remapping the name of FP8 kv-scale.
428 if name.endswith("kv_scale"):
429 remapped_kv_scale_name = name.replace(
vllm/model_executor/models/opt.py
+47 -23
@@ -17,7 +17,7 @@
17 # See the License for the specific language governing permissions and
18 # limitations under the License.
19 """Inference-only OPT model compatible with HuggingFace weights."""
20 -from typing import Iterable, List, Optional, Tuple
20 +from typing import Iterable, List, Optional, Tuple, Union
21
22 import torch
23 from torch import nn
@@ -25,15 +25,14 @@ from transformers import OPTConfig
25
26 from vllm.attention import Attention, AttentionMetadata
27 from vllm.config import CacheConfig
28 -from vllm.distributed import get_tensor_model_parallel_world_size
28 +from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
29 from vllm.model_executor.layers.activation import get_act_fn
30 from vllm.model_executor.layers.linear import (ColumnParallelLinear,
31 QKVParallelLinear,
32 ReplicatedLinear,
33 RowParallelLinear)
34 from vllm.model_executor.layers.logits_processor import LogitsProcessor
35 -from vllm.model_executor.layers.quantization.base_config import (
36 - QuantizationConfig)
35 +from vllm.model_executor.layers.quantization import QuantizationConfig
36 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
37 from vllm.model_executor.layers.vocab_parallel_embedding import (
38 ParallelLMHead, VocabParallelEmbedding)
@@ -41,6 +40,10 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
40 from vllm.model_executor.sampling_metadata import SamplingMetadata
41 from vllm.sequence import IntermediateTensors
42
43 +from .interfaces import SupportsPP
44 +from .utils import (is_pp_missing_parameter,
45 + make_empty_intermediate_tensors_factory, make_layers)
46 +
47
48 class OPTLearnedPositionalEmbedding(nn.Embedding):
49
@@ -189,6 +192,7 @@ class OPTDecoder(nn.Module):
192 config: OPTConfig,
193 cache_config: Optional[CacheConfig] = None,
194 quant_config: Optional[QuantizationConfig] = None,
195 + prefix: str = "",
196 ):
197 super().__init__()
198 self.config = config
@@ -232,10 +236,10 @@ class OPTDecoder(nn.Module):
236 else:
237 self.final_layer_norm = None
238
235 - self.layers = nn.ModuleList([
236 - OPTDecoderLayer(config, cache_config, quant_config)
237 - for _ in range(config.num_hidden_layers)
238 - ])
239 + self.start_layer, self.end_layer, self.layers = make_layers(
240 + config.num_hidden_layers,
241 + lambda prefix: OPTDecoderLayer(config, cache_config, quant_config),
242 + prefix=f"{prefix}.layers")
243
244 def get_input_embeddings(self, input_ids: torch.Tensor) -> torch.Tensor:
245 return self.embed_tokens(input_ids)
@@ -246,19 +250,28 @@ class OPTDecoder(nn.Module):
250 positions: torch.Tensor,
251 kv_caches: List[torch.Tensor],
252 attn_metadata: AttentionMetadata,
253 + intermediate_tensors: Optional[IntermediateTensors],
254 inputs_embeds: Optional[torch.Tensor] = None,
250 - ) -> torch.Tensor:
251 - if inputs_embeds is None:
252 - inputs_embeds = self.get_input_embeddings(input_ids)
253 - pos_embeds = self.embed_positions(positions)
254 - if self.project_in is not None:
255 - inputs_embeds, _ = self.project_in(inputs_embeds)
256 - hidden_states = inputs_embeds + pos_embeds
257 -
258 - for i in range(len(self.layers)):
255 + ) -> Union[torch.Tensor, IntermediateTensors]:
256 + if get_pp_group().is_first_rank:
257 + if inputs_embeds is None:
258 + inputs_embeds = self.get_input_embeddings(input_ids)
259 + pos_embeds = self.embed_positions(positions)
260 + if self.project_in is not None:
261 + inputs_embeds, _ = self.project_in(inputs_embeds)
262 + hidden_states = inputs_embeds + pos_embeds
263 + else:
264 + assert intermediate_tensors is not None
265 + hidden_states = intermediate_tensors["hidden_states"]
266 +
267 + for i in range(self.start_layer, self.end_layer):
268 layer = self.layers[i]
260 - hidden_states = layer(hidden_states, kv_caches[i], attn_metadata)
269 + hidden_states = layer(hidden_states,
270 + kv_caches[i - self.start_layer],
271 + attn_metadata)
272
273 + if not get_pp_group().is_last_rank:
274 + return IntermediateTensors({"hidden_states": hidden_states})
275 if self.final_layer_norm is not None:
276 hidden_states = self.final_layer_norm(hidden_states)
277 if self.project_out is not None:
@@ -276,6 +289,9 @@ class OPTModel(nn.Module):
289 ):
290 super().__init__()
291 self.decoder = OPTDecoder(config, cache_config, quant_config)
292 + self.make_empty_intermediate_tensors = (
293 + make_empty_intermediate_tensors_factory(["hidden_states"],
294 + config.hidden_size))
295
296 def get_input_embeddings(self, input_ids: torch.Tensor) -> torch.Tensor:
297 return self.decoder.get_input_embeddings(input_ids)
@@ -286,20 +302,22 @@ class OPTModel(nn.Module):
302 positions: torch.Tensor,
303 kv_caches: List[torch.Tensor],
304 attn_metadata: AttentionMetadata,
305 + intermediate_tensors: Optional[IntermediateTensors],
306 inputs_embeds: Optional[torch.Tensor] = None,
290 - ) -> torch.Tensor:
307 + ) -> Union[torch.Tensor, IntermediateTensors]:
308 return self.decoder(input_ids,
309 positions,
310 kv_caches,
311 attn_metadata,
312 + intermediate_tensors,
313 inputs_embeds=inputs_embeds)
314
315
298 -class OPTForCausalLM(nn.Module):
316 +class OPTForCausalLM(nn.Module, SupportsPP):
317
318 def __init__(
319 self,
302 - config,
320 + config: OPTConfig,
321 cache_config: Optional[CacheConfig] = None,
322 quant_config: Optional[QuantizationConfig] = None,
323 ):
@@ -314,6 +332,8 @@ class OPTForCausalLM(nn.Module):
332 config.word_embed_proj_dim)
333 self.logits_processor = LogitsProcessor(config.vocab_size)
334 self.sampler = Sampler()
335 + self.make_empty_intermediate_tensors = (
336 + self.model.make_empty_intermediate_tensors)
337
338 def forward(
339 self,
@@ -322,9 +342,9 @@ class OPTForCausalLM(nn.Module):
342 kv_caches: List[torch.Tensor],
343 attn_metadata: AttentionMetadata,
344 intermediate_tensors: Optional[IntermediateTensors] = None,
325 - ) -> torch.Tensor:
345 + ) -> Union[torch.Tensor, IntermediateTensors]:
346 hidden_states = self.model(input_ids, positions, kv_caches,
327 - attn_metadata)
347 + attn_metadata, intermediate_tensors)
348 return hidden_states
349
350 def compute_logits(
@@ -365,6 +385,8 @@ class OPTForCausalLM(nn.Module):
385 # Skip loading extra bias for GPTQ models.
386 if name.endswith(".bias") and name not in params_dict:
387 continue
388 + if is_pp_missing_parameter(name, self):
389 + continue
390 param = params_dict[name]
391 weight_loader = param.weight_loader
392 weight_loader(param, loaded_weight, shard_id)
@@ -373,6 +395,8 @@ class OPTForCausalLM(nn.Module):
395 # Skip loading extra bias for GPTQ models.
396 if name.endswith(".bias") and name not in params_dict:
397 continue
398 + if is_pp_missing_parameter(name, self):
399 + continue
400 param = params_dict[name]
401 weight_loader = getattr(param, "weight_loader",
402 default_weight_loader)
vllm/model_executor/models/orion.py
+44 -16
@@ -4,7 +4,7 @@
4 # Copyright (c) OrionStar Inc.
5 # LICENSE: https://huggingface.co/OrionStarAI/Orion-14B-Base/blob/main/LICENSE
6 """Inference-only Orion-14B model compatible with HuggingFace weights."""
7 -from typing import Any, Dict, Iterable, List, Optional, Tuple
7 +from typing import Any, Dict, Iterable, List, Optional, Tuple, Union
8
9 import torch
10 from torch import nn
@@ -12,14 +12,13 @@ from transformers import PretrainedConfig
12
13 from vllm.attention import Attention, AttentionMetadata
14 from vllm.config import CacheConfig
15 -from vllm.distributed import get_tensor_model_parallel_world_size
15 +from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
16 from vllm.model_executor.layers.activation import SiluAndMul
17 from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
18 QKVParallelLinear,
19 RowParallelLinear)
20 from vllm.model_executor.layers.logits_processor import LogitsProcessor
21 -from vllm.model_executor.layers.quantization.base_config import (
22 - QuantizationConfig)
21 +from vllm.model_executor.layers.quantization import QuantizationConfig
22 from vllm.model_executor.layers.rotary_embedding import get_rope
23 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
24 from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -28,6 +27,10 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
27 from vllm.model_executor.sampling_metadata import SamplingMetadata
28 from vllm.sequence import IntermediateTensors
29
30 +from .interfaces import SupportsPP
31 +from .utils import (is_pp_missing_parameter,
32 + make_empty_intermediate_tensors_factory, make_layers)
33 +
34
35 class OrionMLP(nn.Module):
36
@@ -210,6 +213,7 @@ class OrionModel(nn.Module):
213 config: PretrainedConfig,
214 cache_config: Optional[CacheConfig] = None,
215 quant_config: Optional[QuantizationConfig] = None,
216 + prefix: str = "",
217 ) -> None:
218 super().__init__()
219 self.config = config
@@ -219,11 +223,18 @@ class OrionModel(nn.Module):
223 config.vocab_size,
224 config.hidden_size,
225 )
222 - self.layers = nn.ModuleList([
223 - OrionDecoderLayer(config, cache_config, quant_config)
224 - for _ in range(config.num_hidden_layers)
225 - ])
226 + self.start_layer, self.end_layer, self.layers = make_layers(
227 + config.num_hidden_layers,
228 + lambda prefix: OrionDecoderLayer(
229 + config,
230 + cache_config,
231 + quant_config,
232 + ),
233 + prefix=f"{prefix}.layers")
234 self.norm = nn.LayerNorm(config.hidden_size, eps=config.rms_norm_eps)
235 + self.make_empty_intermediate_tensors = (
236 + make_empty_intermediate_tensors_factory(
237 + ["hidden_states", "residual"], config.hidden_size))
238
239 def forward(
240 self,
@@ -231,23 +242,34 @@ class OrionModel(nn.Module):
242 positions: torch.Tensor,
243 kv_caches: List[torch.Tensor],
244 attn_metadata: AttentionMetadata,
234 - ) -> torch.Tensor:
235 - hidden_states = self.embed_tokens(input_ids)
236 - residual = None
237 - for i in range(len(self.layers)):
245 + intermediate_tensors: Optional[IntermediateTensors],
246 + ) -> Union[torch.Tensor, IntermediateTensors]:
247 + if get_pp_group().is_first_rank:
248 + hidden_states = self.embed_tokens(input_ids)
249 + residual = None
250 + else:
251 + assert intermediate_tensors
252 + hidden_states = intermediate_tensors["hidden_states"]
253 + residual = intermediate_tensors["residual"]
254 + for i in range(self.start_layer, self.end_layer):
255 layer = self.layers[i]
256 hidden_states, residual = layer(
257 positions,
258 hidden_states,
242 - kv_caches[i],
259 + kv_caches[i - self.start_layer],
260 attn_metadata,
261 residual,
262 )
263 + if not get_pp_group().is_last_rank:
264 + return IntermediateTensors({
265 + "hidden_states": hidden_states,
266 + "residual": residual
267 + })
268 hidden_states = self.norm(hidden_states)
269 return hidden_states
270
271
250 -class OrionForCausalLM(nn.Module):
272 +class OrionForCausalLM(nn.Module, SupportsPP):
273
274 def __init__(
275 self,
@@ -266,6 +288,8 @@ class OrionForCausalLM(nn.Module):
288 self.lm_head.weight = self.model.embed_tokens.weight
289 self.logits_processor = LogitsProcessor(config.vocab_size)
290 self.sampler = Sampler()
291 + self.make_empty_intermediate_tensors = (
292 + self.model.make_empty_intermediate_tensors)
293
294 def forward(
295 self,
@@ -274,9 +298,9 @@ class OrionForCausalLM(nn.Module):
298 kv_caches: List[torch.Tensor],
299 attn_metadata: AttentionMetadata,
300 intermediate_tensors: Optional[IntermediateTensors] = None,
277 - ) -> torch.Tensor:
301 + ) -> Union[torch.Tensor, IntermediateTensors]:
302 hidden_states = self.model(input_ids, positions, kv_caches,
279 - attn_metadata)
303 + attn_metadata, intermediate_tensors)
304 return hidden_states
305
306 def compute_logits(
@@ -321,6 +345,8 @@ class OrionForCausalLM(nn.Module):
345 # Skip loading extra bias for GPTQ models.
346 if name.endswith(".bias") and name not in params_dict:
347 continue
348 + if is_pp_missing_parameter(name, self):
349 + continue
350 param = params_dict[name]
351 weight_loader = param.weight_loader
352 weight_loader(param, loaded_weight, shard_id)
@@ -329,6 +355,8 @@ class OrionForCausalLM(nn.Module):
355 # Skip loading extra bias for GPTQ models.
356 if name.endswith(".bias") and name not in params_dict:
357 continue
358 + if is_pp_missing_parameter(name, self):
359 + continue
360 param = params_dict[name]
361 weight_loader = getattr(param, "weight_loader",
362 default_weight_loader)
vllm/model_executor/models/paligemma.py
+33 -26
@@ -9,9 +9,8 @@ from vllm.attention import AttentionMetadata
9 from vllm.config import CacheConfig, MultiModalConfig
10 from vllm.inputs import INPUT_REGISTRY, InputContext, LLMInputs
11 from vllm.logger import init_logger
12 -from vllm.model_executor.layers.logits_processor import LogitsProcessor
12 from vllm.model_executor.layers.quantization import QuantizationConfig
14 -from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
13 +from vllm.model_executor.layers.sampler import SamplerOutput
14 from vllm.model_executor.model_loader.weight_utils import default_weight_loader
15 from vllm.model_executor.models.gemma import GemmaForCausalLM
16 from vllm.model_executor.sampling_metadata import SamplingMetadata
@@ -19,7 +18,7 @@ from vllm.multimodal import MULTIMODAL_REGISTRY
18 from vllm.multimodal.utils import cached_get_tokenizer
19 from vllm.sequence import IntermediateTensors
20
22 -from .interfaces import SupportsMultiModal
21 +from .interfaces import SupportsMultiModal, SupportsPP
22 from .siglip import (SiglipVisionModel, dummy_image_for_siglip,
23 dummy_seq_data_for_siglip, get_max_siglip_image_tokens)
24 from .utils import group_weights_with_prefix, merge_multimodal_embeddings
@@ -129,7 +128,8 @@ class PaliGemmaMultiModalProjector(nn.Module):
128 @MULTIMODAL_REGISTRY.register_max_image_tokens(get_max_paligemma_image_tokens)
129 @INPUT_REGISTRY.register_dummy_data(dummy_data_for_paligemma)
130 @INPUT_REGISTRY.register_input_processor(input_processor_for_paligemma)
132 -class PaliGemmaForConditionalGeneration(nn.Module, SupportsMultiModal):
131 +class PaliGemmaForConditionalGeneration(nn.Module, SupportsMultiModal,
132 + SupportsPP):
133
134 def __init__(self,
135 config: PaliGemmaConfig,
@@ -149,12 +149,15 @@ class PaliGemmaForConditionalGeneration(nn.Module, SupportsMultiModal):
149 self.quant_config = quant_config
150 self.language_model = GemmaForCausalLM(config.text_config,
151 cache_config, quant_config)
152 - self.unpadded_vocab_size = config.text_config.vocab_size
152 logit_scale = getattr(config, "logit_scale", 1.0)
154 - self.logits_processor = LogitsProcessor(self.unpadded_vocab_size,
155 - config.text_config.vocab_size,
156 - logit_scale)
157 - self.sampler = Sampler()
153 + self.language_model.logits_processor.scale *= logit_scale
154 +
155 + self.make_empty_intermediate_tensors = (
156 + self.language_model.make_empty_intermediate_tensors)
157 +
158 + @property
159 + def sampler(self):
160 + return self.language_model.sampler
161
162 def _validate_pixel_values(self, data: torch.Tensor) -> torch.Tensor:
163 h = w = self.config.vision_config.image_size
@@ -239,32 +242,36 @@ class PaliGemmaForConditionalGeneration(nn.Module, SupportsMultiModal):
242 kv_caches: List[torch.Tensor],
243 attn_metadata: AttentionMetadata,
244 intermediate_tensors: Optional[IntermediateTensors] = None,
242 - **kwargs: object) -> SamplerOutput:
243 -
244 - parsed_image_input = self._parse_and_validate_image_input(**kwargs)
245 + **kwargs: object) -> Union[SamplerOutput, IntermediateTensors]:
246 + if intermediate_tensors is not None:
247 + input_ids = None
248 + inputs_embeds = None
249 + else:
250 + parsed_image_input = self._parse_and_validate_image_input(**kwargs)
251
246 - if parsed_image_input is not None:
247 - vision_embeddings = self._process_image_input(parsed_image_input)
248 - # https://github.com/huggingface/transformers/blob/main/src/transformers/models/paligemma/modeling_paligemma.py#L294 # noqa
249 - vision_embeddings = vision_embeddings * (self.config.hidden_size**
250 - -0.5)
252 + if parsed_image_input is not None:
253 + vision_embeddings = self._process_image_input(
254 + parsed_image_input)
255 + # https://github.com/huggingface/transformers/blob/main/src/transformers/models/paligemma/modeling_paligemma.py#L294 # noqa
256 + vision_embeddings = vision_embeddings * (
257 + self.config.hidden_size**-0.5)
258
252 - inputs_embeds = self.language_model.model.get_input_embeddings(
253 - input_ids)
259 + inputs_embeds = self.language_model.model.get_input_embeddings(
260 + input_ids)
261
255 - inputs_embeds = merge_multimodal_embeddings(
256 - input_ids, inputs_embeds, vision_embeddings,
257 - self.config.image_token_index)
262 + inputs_embeds = merge_multimodal_embeddings(
263 + input_ids, inputs_embeds, vision_embeddings,
264 + self.config.image_token_index)
265
259 - input_ids = None
260 - else:
261 - inputs_embeds = None
266 + input_ids = None
267 + else:
268 + inputs_embeds = None
269
270 hidden_states = self.language_model.model(input_ids,
271 positions,
272 kv_caches,
273 attn_metadata,
267 - None,
274 + intermediate_tensors,
275 inputs_embeds=inputs_embeds)
276
277 return hidden_states
vllm/model_executor/models/persimmon.py
+36 -18
@@ -20,7 +20,7 @@
20 # See the License for the specific language governing permissions and
21 # limitations under the License.
22 """Inference-only persimmon model compatible with HuggingFace weights."""
23 -from typing import Iterable, List, Optional, Tuple
23 +from typing import Iterable, List, Optional, Tuple, Union
24
25 import torch
26 from torch import nn
@@ -28,14 +28,13 @@ from transformers import PersimmonConfig
28
29 from vllm.attention import Attention, AttentionMetadata
30 from vllm.config import CacheConfig
31 -from vllm.distributed import get_tensor_model_parallel_world_size
31 +from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
32 from vllm.model_executor.layers.activation import get_act_fn
33 from vllm.model_executor.layers.linear import (ColumnParallelLinear,
34 QKVParallelLinear,
35 RowParallelLinear)
36 from vllm.model_executor.layers.logits_processor import LogitsProcessor
37 -from vllm.model_executor.layers.quantization.base_config import (
38 - QuantizationConfig)
37 +from vllm.model_executor.layers.quantization import QuantizationConfig
38 from vllm.model_executor.layers.rotary_embedding import get_rope
39 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
40 from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -44,6 +43,10 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
43 from vllm.model_executor.sampling_metadata import SamplingMetadata
44 from vllm.sequence import IntermediateTensors
45
46 +from .interfaces import SupportsPP
47 +from .utils import (is_pp_missing_parameter,
48 + make_empty_intermediate_tensors_factory, make_layers)
49 +
50
51 class PersimmonMLP(nn.Module):
52
@@ -211,20 +214,23 @@ class PersimmonModel(nn.Module):
214 def __init__(self,
215 config: PersimmonConfig,
216 cache_config: Optional[CacheConfig] = None,
214 - quant_config: Optional[QuantizationConfig] = None):
217 + quant_config: Optional[QuantizationConfig] = None,
218 + prefix: str = ""):
219 super().__init__()
220 self.vocab_size = config.vocab_size
221
222 self.embed_tokens = VocabParallelEmbedding(config.vocab_size,
223 config.hidden_size)
220 - self.layers = nn.ModuleList([
221 - PersimmonDecoderLayer(config,
222 - cache_config=cache_config,
223 - quant_config=quant_config)
224 - for _ in range(config.num_hidden_layers)
225 - ])
224 + self.start_layer, self.end_layer, self.layers = make_layers(
225 + config.num_hidden_layers,
226 + lambda prefix: PersimmonDecoderLayer(config, cache_config,
227 + quant_config),
228 + prefix=f"{prefix}.layers")
229 self.final_layernorm = nn.LayerNorm(config.hidden_size,
230 eps=config.layer_norm_eps)
231 + self.make_empty_intermediate_tensors = (
232 + make_empty_intermediate_tensors_factory(["hidden_states"],
233 + config.hidden_size))
234
235 def forward(
236 self,
@@ -232,24 +238,31 @@ class PersimmonModel(nn.Module):
238 positions: torch.Tensor,
239 kv_caches: List[torch.Tensor],
240 attn_metadata: AttentionMetadata,
241 + intermediate_tensors: Optional[IntermediateTensors],
242 inputs_embeds: Optional[torch.Tensor] = None,
236 - ) -> torch.Tensor:
237 - if inputs_embeds is not None:
238 - hidden_states = inputs_embeds
243 + ) -> Union[torch.Tensor, IntermediateTensors]:
244 + if get_pp_group().is_first_rank:
245 + if inputs_embeds is not None:
246 + hidden_states = inputs_embeds
247 + else:
248 + hidden_states = self.embed_tokens(input_ids)
249 else:
240 - hidden_states = self.embed_tokens(input_ids)
241 - for i in range(len(self.layers)):
250 + assert intermediate_tensors is not None
251 + hidden_states = intermediate_tensors["hidden_states"]
252 + for i in range(self.start_layer, self.end_layer):
253 hidden_states = self.layers[i](
254 positions,
255 hidden_states,
245 - kv_caches[i],
256 + kv_caches[i - self.start_layer],
257 attn_metadata,
258 )
259 + if not get_pp_group().is_last_rank:
260 + return IntermediateTensors({"hidden_states": hidden_states})
261 hidden_states = self.final_layernorm(hidden_states)
262 return hidden_states
263
264
252 -class PersimmonForCausalLM(nn.Module):
265 +class PersimmonForCausalLM(nn.Module, SupportsPP):
266
267 def __init__(self,
268 config: PersimmonConfig,
@@ -266,6 +279,8 @@ class PersimmonForCausalLM(nn.Module):
279 bias=False)
280 self.logits_processor = LogitsProcessor(config.vocab_size)
281 self.sampler = Sampler()
282 + self.make_empty_intermediate_tensors = (
283 + self.model.make_empty_intermediate_tensors)
284
285 def forward(
286 self,
@@ -281,6 +296,7 @@ class PersimmonForCausalLM(nn.Module):
296 positions=positions,
297 kv_caches=kv_caches,
298 attn_metadata=attn_metadata,
299 + intermediate_tensors=intermediate_tensors,
300 inputs_embeds=inputs_embeds,
301 )
302 return hidden_states
@@ -312,6 +328,8 @@ class PersimmonForCausalLM(nn.Module):
328 # Models trained using ColossalAI may include these tensors in
329 # the checkpoint. Skip them.
330 continue
331 + if is_pp_missing_parameter(name, self):
332 + continue
333 param = params_dict[name]
334
335 if "query_key_value" in name:
vllm/model_executor/models/phi.py
+36 -17
@@ -35,7 +35,7 @@
35 # OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
36 # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
37 """Inference-only Phi-1.5 model compatible with HuggingFace weights."""
38 -from typing import Iterable, List, Optional, Tuple
38 +from typing import Iterable, List, Optional, Tuple, Union
39
40 import torch
41 from torch import nn
@@ -43,14 +43,13 @@ from transformers import PhiConfig
43
44 from vllm.attention import Attention, AttentionMetadata
45 from vllm.config import CacheConfig, LoRAConfig
46 -from vllm.distributed import get_tensor_model_parallel_world_size
46 +from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
47 from vllm.model_executor.layers.activation import get_act_fn
48 from vllm.model_executor.layers.linear import (ColumnParallelLinear,
49 QKVParallelLinear,
50 RowParallelLinear)
51 from vllm.model_executor.layers.logits_processor import LogitsProcessor
52 -from vllm.model_executor.layers.quantization.base_config import (
53 - QuantizationConfig)
52 +from vllm.model_executor.layers.quantization import QuantizationConfig
53 from vllm.model_executor.layers.rotary_embedding import get_rope
54 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
55 from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -59,7 +58,9 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
58 from vllm.model_executor.sampling_metadata import SamplingMetadata
59 from vllm.sequence import IntermediateTensors
60
62 -from .interfaces import SupportsLoRA
61 +from .interfaces import SupportsLoRA, SupportsPP
62 +from .utils import (is_pp_missing_parameter,
63 + make_empty_intermediate_tensors_factory, make_layers)
64
65
66 class PhiAttention(nn.Module):
@@ -196,18 +197,22 @@ class PhiModel(nn.Module):
197 def __init__(self,
198 config: PhiConfig,
199 cache_config: Optional[CacheConfig] = None,
199 - quant_config: Optional[QuantizationConfig] = None):
200 + quant_config: Optional[QuantizationConfig] = None,
201 + prefix: str = ""):
202 super().__init__()
203 self.config = config
204 self.quant_config = quant_config
205 self.embed_tokens = VocabParallelEmbedding(config.vocab_size,
206 config.hidden_size)
205 - self.layers = nn.ModuleList([
206 - PhiLayer(config, cache_config, quant_config)
207 - for _ in range(config.num_hidden_layers)
208 - ])
207 + self.start_layer, self.end_layer, self.layers = make_layers(
208 + config.num_hidden_layers,
209 + lambda prefix: PhiLayer(config, cache_config, quant_config),
210 + prefix=f"{prefix}.layers")
211 self.final_layernorm = nn.LayerNorm(config.hidden_size,
212 eps=config.layer_norm_eps)
213 + self.make_empty_intermediate_tensors = (
214 + make_empty_intermediate_tensors_factory(["hidden_states"],
215 + config.hidden_size))
216
217 def forward(
218 self,
@@ -215,23 +220,31 @@ class PhiModel(nn.Module):
220 positions: torch.Tensor,
221 kv_caches: List[torch.Tensor],
222 attn_metadata: AttentionMetadata,
218 - ) -> torch.Tensor:
219 - hidden_states = self.embed_tokens(input_ids)
220 - for i in range(self.config.num_hidden_layers):
223 + intermediate_tensors: Optional[IntermediateTensors],
224 + ) -> Union[torch.Tensor, IntermediateTensors]:
225 + if get_pp_group().is_first_rank:
226 + hidden_states = self.embed_tokens(input_ids)
227 + else:
228 + assert intermediate_tensors is not None
229 + hidden_states = intermediate_tensors["hidden_states"]
230 + for i in range(self.start_layer, self.end_layer):
231 layer = self.layers[i]
232 hidden_states = layer(
233 positions,
234 hidden_states,
225 - kv_caches[i],
235 + kv_caches[i - self.start_layer],
236 attn_metadata,
237 )
238
239 + if not get_pp_group().is_last_rank:
240 + return IntermediateTensors({"hidden_states": hidden_states})
241 +
242 hidden_states = self.final_layernorm(hidden_states)
243
244 return hidden_states
245
246
234 -class PhiForCausalLM(nn.Module, SupportsLoRA):
247 +class PhiForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
248 packed_modules_mapping = {
249 "qkv_proj": [
250 "q_proj",
@@ -274,6 +287,8 @@ class PhiForCausalLM(nn.Module, SupportsLoRA):
287 quant_config=quant_config)
288 self.logits_processor = LogitsProcessor(config.vocab_size)
289 self.sampler = Sampler()
290 + self.make_empty_intermediate_tensors = (
291 + self.model.make_empty_intermediate_tensors)
292
293 def forward(
294 self,
@@ -282,9 +297,9 @@ class PhiForCausalLM(nn.Module, SupportsLoRA):
297 kv_caches: List[torch.Tensor],
298 attn_metadata: AttentionMetadata,
299 intermediate_tensors: Optional[IntermediateTensors] = None,
285 - ) -> torch.Tensor:
300 + ) -> Union[torch.Tensor, IntermediateTensors]:
301 hidden_states = self.model(input_ids, positions, kv_caches,
287 - attn_metadata)
302 + attn_metadata, intermediate_tensors)
303
304 return hidden_states
305
@@ -325,6 +340,8 @@ class PhiForCausalLM(nn.Module, SupportsLoRA):
340 # Skip loading extra bias for GPTQ models.
341 if name.endswith(".bias") and name not in params_dict:
342 continue
343 + if is_pp_missing_parameter(name, self):
344 + continue
345 param = params_dict[name]
346 weight_loader = param.weight_loader
347 weight_loader(param, loaded_weight, shard_id)
@@ -335,6 +352,8 @@ class PhiForCausalLM(nn.Module, SupportsLoRA):
352 continue
353 # pylint: disable=E1136
354
355 + if is_pp_missing_parameter(name, self):
356 + continue
357 param = params_dict[name]
358 weight_loader = getattr(param, "weight_loader",
359 default_weight_loader)
vllm/model_executor/models/phi3_small.py
+40 -20
@@ -1,5 +1,5 @@
1 import math
2 -from typing import Iterable, List, Optional, Tuple
2 +from typing import Iterable, List, Optional, Tuple, Union
3
4 import torch
5 from torch import nn
@@ -7,14 +7,13 @@ from transformers.configuration_utils import PretrainedConfig
7
8 from vllm.attention import Attention, AttentionMetadata
9 from vllm.config import CacheConfig, LoRAConfig
10 -from vllm.distributed import (get_tensor_model_parallel_rank,
10 +from vllm.distributed import (get_pp_group, get_tensor_model_parallel_rank,
11 get_tensor_model_parallel_world_size)
12 from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
13 QKVParallelLinear,
14 RowParallelLinear)
15 from vllm.model_executor.layers.logits_processor import LogitsProcessor
16 -from vllm.model_executor.layers.quantization.base_config import (
17 - QuantizationConfig)
16 +from vllm.model_executor.layers.quantization import QuantizationConfig
17 from vllm.model_executor.layers.rotary_embedding import get_rope
18 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
19 from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -23,6 +22,10 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
22 from vllm.model_executor.sampling_metadata import SamplingMetadata
23 from vllm.sequence import IntermediateTensors
24
25 +from .interfaces import SupportsPP
26 +from .utils import (is_pp_missing_parameter,
27 + make_empty_intermediate_tensors_factory, make_layers)
28 +
29
30 def load_column_parallel_weight(param: torch.nn.Parameter,
31 loaded_weight: torch.Tensor):
@@ -301,20 +304,25 @@ class Phi3SmallModel(nn.Module):
304 config: PretrainedConfig,
305 cache_config: Optional[CacheConfig] = None,
306 quant_config: Optional[QuantizationConfig] = None,
307 + prefix: str = "",
308 ):
309 super().__init__()
310 self.config = config
311 self.embed_tokens = VocabParallelEmbedding(config.vocab_size,
312 config.hidden_size)
313 self.mup_embedding_multiplier = config.mup_embedding_multiplier
310 - self.layers = nn.ModuleList([
311 - Phi3SmallDecoderLayer(config, layer_idx, cache_config,
312 - quant_config)
313 - for layer_idx in range(config.num_hidden_layers)
314 - ])
314 + self.start_layer, self.end_layer, self.layers = make_layers(
315 + config.num_hidden_layers,
316 + lambda prefix: Phi3SmallDecoderLayer(config,
317 + int(prefix.split('.')[-1]),
318 + cache_config, quant_config),
319 + prefix=f"{prefix}.layers")
320
321 self.final_layernorm = nn.LayerNorm(config.hidden_size,
322 eps=config.layer_norm_epsilon)
323 + self.make_empty_intermediate_tensors = (
324 + make_empty_intermediate_tensors_factory(["hidden_states"],
325 + config.hidden_size))
326
327 def get_input_embeddings(self):
328 return self.embed_tokens
@@ -327,30 +335,37 @@ class Phi3SmallModel(nn.Module):
335 input_ids: torch.LongTensor,
336 positions: Optional[torch.LongTensor],
337 kv_caches: List[torch.Tensor],
330 - attn_metadata: AttentionMetadata = None,
331 - ):
332 - hidden_states = self.embed_tokens(input_ids)
333 - if (self.mup_embedding_multiplier is not None
334 - and self.mup_embedding_multiplier > 0.0):
335 - hidden_states = hidden_states * self.mup_embedding_multiplier
336 - for i in range(len(self.layers)):
338 + attn_metadata: AttentionMetadata,
339 + intermediate_tensors: Optional[IntermediateTensors],
340 + ) -> Union[torch.Tensor, IntermediateTensors]:
341 + if get_pp_group().is_first_rank:
342 + hidden_states = self.embed_tokens(input_ids)
343 + if (self.mup_embedding_multiplier is not None
344 + and self.mup_embedding_multiplier > 0.0):
345 + hidden_states = hidden_states * self.mup_embedding_multiplier
346 + else:
347 + assert intermediate_tensors
348 + hidden_states = intermediate_tensors["hidden_states"]
349 + for i in range(self.start_layer, self.end_layer):
350 layer = self.layers[i]
351 hidden_states = layer(
352 positions,
353 hidden_states,
341 - kv_caches[i],
354 + kv_caches[i - self.start_layer],
355 attn_metadata,
356 )
357 + if not get_pp_group().is_last_rank:
358 + return IntermediateTensors({"hidden_states": hidden_states})
359 hidden_states = self.final_layernorm(hidden_states)
360 return hidden_states
361
362
348 -class Phi3SmallForCausalLM(nn.Module):
363 +class Phi3SmallForCausalLM(nn.Module, SupportsPP):
364 _tied_weights_keys = ["lm_head.weight"]
365
366 def __init__(
367 self,
353 - config,
368 + config: PretrainedConfig,
369 cache_config: Optional[CacheConfig] = None,
370 quant_config: Optional[QuantizationConfig] = None,
371 lora_config: Optional[LoRAConfig] = None,
@@ -372,6 +387,8 @@ class Phi3SmallForCausalLM(nn.Module):
387 self.lm_head.weight = self.model.embed_tokens.weight
388 self.logits_processor = LogitsProcessor(config.vocab_size)
389 self.sampler = Sampler()
390 + self.make_empty_intermediate_tensors = (
391 + self.model.make_empty_intermediate_tensors)
392
393 # tokens in tiktoken but not used
394 if hasattr(config, 'dummy_token_indices'):
@@ -419,12 +436,13 @@ class Phi3SmallForCausalLM(nn.Module):
436 kv_caches: List[torch.Tensor],
437 attn_metadata: AttentionMetadata,
438 intermediate_tensors: Optional[IntermediateTensors] = None,
422 - ) -> torch.Tensor:
439 + ) -> Union[torch.Tensor, IntermediateTensors]:
440 output_hidden_states = self.model(
441 input_ids=input_ids,
442 positions=positions,
443 kv_caches=kv_caches,
444 attn_metadata=attn_metadata,
445 + intermediate_tensors=intermediate_tensors,
446 )
447 output_hidden_states = output_hidden_states
448 return output_hidden_states
@@ -447,6 +465,8 @@ class Phi3SmallForCausalLM(nn.Module):
465 continue
466 if name.endswith(".bias") and name not in params_dict:
467 continue
468 + if is_pp_missing_parameter(name, self):
469 + continue
470 param = params_dict[name]
471 weight_loader = getattr(param, "weight_loader",
472 default_weight_loader)
vllm/model_executor/models/phi3v.py
+96 -85
@@ -15,7 +15,7 @@
15 # limitations under the License.
16 import itertools
17 import re
18 -from functools import lru_cache
18 +from functools import cached_property, lru_cache
19 from typing import (Any, Dict, Iterable, List, Literal, Mapping, Optional,
20 Tuple, TypedDict, Union)
21
@@ -29,13 +29,11 @@ from vllm.attention import AttentionMetadata
29 from vllm.config import CacheConfig, ModelConfig, MultiModalConfig
30 from vllm.inputs import INPUT_REGISTRY, InputContext, LLMInputs
31 from vllm.logger import init_logger
32 -from vllm.model_executor.layers.logits_processor import LogitsProcessor
32 from vllm.model_executor.layers.quantization import QuantizationConfig
33 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
35 -from vllm.model_executor.layers.vocab_parallel_embedding import ParallelLMHead
34 from vllm.model_executor.model_loader.weight_utils import default_weight_loader
35 from vllm.model_executor.models.clip import CLIPVisionModel
38 -from vllm.model_executor.models.llama import LlamaModel
36 +from vllm.model_executor.models.llama import LlamaForCausalLM
37 from vllm.model_executor.sampling_metadata import SamplingMetadata
38 from vllm.multimodal import MULTIMODAL_REGISTRY
39 from vllm.multimodal.utils import cached_get_tokenizer, repeat_and_pad_token
@@ -43,8 +41,9 @@ from vllm.sequence import IntermediateTensors
41 from vllm.utils import is_list_of
42
43 from .clip import dummy_image_for_clip, dummy_seq_data_for_clip
46 -from .interfaces import SupportsMultiModal
47 -from .utils import flatten_bn, merge_multimodal_embeddings
44 +from .interfaces import SupportsMultiModal, SupportsPP
45 +from .utils import (flatten_bn, group_weights_with_prefix,
46 + merge_multimodal_embeddings)
47
48 logger = init_logger(__name__)
49
@@ -295,6 +294,37 @@ class Phi3HDImageEmbedding(Phi3ImageEmbeddingBase):
294 dim=2).reshape(num_images, -1, hid_dim)
295 return image_features_hd_newline
296
297 + def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
298 + # prepare weight iterators for components
299 + weights_group = group_weights_with_prefix(weights)
300 +
301 + # load vision encoder
302 + self.img_processor.load_weights(weights_group["img_processor"])
303 +
304 + # load glb_GN
305 + for name, loaded_weight in weights_group["glb_GN"]:
306 + assert name == ""
307 + param = self.glb_GN
308 + weight_loader = getattr(param, "weight_loader",
309 + default_weight_loader)
310 + weight_loader(param, loaded_weight)
311 +
312 + # load sub_GN
313 + for name, loaded_weight in weights_group["sub_GN"]:
314 + assert name == ""
315 + param = self.sub_GN
316 + weight_loader = getattr(param, "weight_loader",
317 + default_weight_loader)
318 + weight_loader(param, loaded_weight)
319 +
320 + # load mlp projector
321 + mlp_params_dict = dict(self.img_projection.named_parameters())
322 + for name, loaded_weight in weights_group["img_projection"]:
323 + param = mlp_params_dict[name]
324 + weight_loader = getattr(param, "weight_loader",
325 + default_weight_loader)
326 + weight_loader(param, loaded_weight)
327 +
328
329 # Based on https://huggingface.co/microsoft/Phi-3-vision-128k-instruct/blob/main/image_processing_phi3_v.py#L57
330 def _calc_padded_size(*, width: int, height: int, padding_unit: int = 336):
@@ -508,7 +538,7 @@ def input_processor_for_phi3v(ctx: InputContext,
538 @MULTIMODAL_REGISTRY.register_max_image_tokens(get_max_phi3v_image_tokens)
539 @INPUT_REGISTRY.register_dummy_data(dummy_data_for_phi3v)
540 @INPUT_REGISTRY.register_input_processor(input_processor_for_phi3v)
511 -class Phi3VForCausalLM(nn.Module, SupportsMultiModal):
541 +class Phi3VForCausalLM(nn.Module, SupportsMultiModal, SupportsPP):
542
543 def __init__(self,
544 config: PretrainedConfig,
@@ -521,17 +551,21 @@ class Phi3VForCausalLM(nn.Module, SupportsMultiModal):
551 self.multimodal_config = multimodal_config
552 self.image_token_id = _IMAGE_TOKEN_ID
553
524 - self.model = LlamaModel(config, cache_config, quant_config)
525 -
554 # TODO: Optionally initializes this for supporting embeddings.
555 self.vision_embed_tokens = Phi3HDImageEmbedding(config)
528 - self.lm_head = ParallelLMHead(config.vocab_size,
529 - config.hidden_size,
530 - quant_config=quant_config)
531 - if self.config.tie_word_embeddings:
532 - self.lm_head.weight = self.model.embed_tokens.weight
533 - self.logits_processor = LogitsProcessor(config.vocab_size)
534 - self.sampler = Sampler()
556 +
557 + self.language_model = LlamaForCausalLM(config, cache_config,
558 + quant_config)
559 +
560 + self.make_empty_intermediate_tensors = (
561 + self.language_model.make_empty_intermediate_tensors)
562 +
563 + @cached_property
564 + def sampler(self):
565 + if hasattr(self.language_model, "sampler"):
566 + return self.language_model.sampler
567 +
568 + return Sampler()
569
570 def _validate_image_sizes(self, data: torch.Tensor) -> torch.Tensor:
571 expected_dims = (2, )
@@ -631,24 +665,29 @@ class Phi3VForCausalLM(nn.Module, SupportsMultiModal):
665 attn_metadata: AttentionMetadata,
666 intermediate_tensors: Optional[IntermediateTensors] = None,
667 **kwargs: object):
634 - image_input = self._parse_and_validate_image_input(**kwargs)
635 -
636 - if image_input is not None:
637 - vision_embeddings = self._process_image_input(image_input)
638 - inputs_embeds = self.model.get_input_embeddings(input_ids)
639 - inputs_embeds = merge_multimodal_embeddings(
640 - input_ids, inputs_embeds, vision_embeddings,
641 - self.image_token_id)
668 + if intermediate_tensors is not None:
669 input_ids = None
643 - else:
670 inputs_embeds = None
671 + else:
672 + image_input = self._parse_and_validate_image_input(**kwargs)
673 +
674 + if image_input is not None:
675 + vision_embeddings = self._process_image_input(image_input)
676 + inputs_embeds = self.language_model.model.get_input_embeddings(
677 + input_ids)
678 + inputs_embeds = merge_multimodal_embeddings(
679 + input_ids, inputs_embeds, vision_embeddings,
680 + self.image_token_id)
681 + input_ids = None
682 + else:
683 + inputs_embeds = None
684
646 - hidden_states = self.model(input_ids,
647 - positions,
648 - kv_caches,
649 - attn_metadata,
650 - intermediate_tensors,
651 - inputs_embeds=inputs_embeds)
685 + hidden_states = self.language_model.model(input_ids,
686 + positions,
687 + kv_caches,
688 + attn_metadata,
689 + intermediate_tensors,
690 + inputs_embeds=inputs_embeds)
691
692 return hidden_states
693
@@ -657,66 +696,38 @@ class Phi3VForCausalLM(nn.Module, SupportsMultiModal):
696 hidden_states: torch.Tensor,
697 sampling_metadata: SamplingMetadata,
698 ) -> Optional[torch.Tensor]:
660 - logits = self.logits_processor(self.lm_head, hidden_states,
661 - sampling_metadata)
662 - return logits
699 + return self.language_model.compute_logits(hidden_states,
700 + sampling_metadata)
701
702 def sample(
703 self,
704 logits: torch.Tensor,
705 sampling_metadata: SamplingMetadata,
706 ) -> Optional[SamplerOutput]:
669 - next_tokens = self.sampler(logits, sampling_metadata)
670 - return next_tokens
707 + return self.language_model.sample(logits, sampling_metadata)
708
709 def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
673 - stacked_params_mapping = [
674 - # (param_name, shard_name, shard_id)
675 - (".qkv_proj", ".q_proj", "q"),
676 - (".qkv_proj", ".k_proj", "k"),
677 - (".qkv_proj", ".v_proj", "v"),
678 - (".gate_up_proj", ".gate_proj", 0),
679 - (".gate_up_proj", ".up_proj", 1),
680 - ]
710 + hf_to_vllm_mapping = {
711 + "model.vision_embed_tokens.": "vision_embed_tokens.",
712 + "lm_head.": "language_model.lm_head.",
713 + "model.": "language_model.model.",
714 + }
715
682 - # TODO(ChristopherCho): This is a temporary fix to load
683 - # the vision weights with CLIPVisionModel.load_weights()
684 - vision_weights = []
685 - params_dict = dict(self.named_parameters())
686 - for name, loaded_weight in weights:
687 - if "rotary_emb.inv_freq" in name:
688 - continue
689 - # Skip loading the img_processor weights since they are
690 - # loaded separately.
691 - if "vision_embed_tokens.img_processor" in name:
692 - vision_weights.append((name, loaded_weight))
693 - continue
694 -
695 - for key_to_modify, new_key in _KEYS_TO_MODIFY_MAPPING.items():
696 - if key_to_modify in name:
697 - name = name.replace(key_to_modify, new_key)
698 - for (param_name, weight_name, shard_id) in stacked_params_mapping:
699 - if weight_name not in name:
700 - continue
701 -
702 - param = params_dict[name.replace(weight_name, param_name)]
703 - weight_loader = param.weight_loader
704 - weight_loader(param, loaded_weight, shard_id)
705 - break
706 - else:
707 - # Skip loading extra bias for GPTQ models.
708 - if name.endswith(".bias") and name not in params_dict:
709 - continue
710 - if name in params_dict:
711 - param = params_dict[name]
712 - weight_loader = getattr(param, "weight_loader",
713 - default_weight_loader)
714 - weight_loader(param, loaded_weight)
715 -
716 - # We use regex to extract the sub-module name
717 - # from "model.vision_embed_tokens.img_processor.*"
718 - vision_weights = [
719 - (re.search(r"vision_embed_tokens\.img_processor\.(.*)",
720 - n).group(1), w) for n, w in vision_weights
721 - ]
722 - self.vision_embed_tokens.img_processor.load_weights(vision_weights)
716 + def hf_to_vllm_name(key: str) -> str:
717 + for hf_name, vllm_name in hf_to_vllm_mapping.items():
718 + if key.startswith(hf_name):
719 + return key.replace(hf_name, vllm_name, 1)
720 +
721 + return key
722 +
723 + vllm_weights = {hf_to_vllm_name(k): v for k, v in weights}
724 +
725 + # prepare weight iterators for components
726 + weights_group = group_weights_with_prefix(vllm_weights.items())
727 +
728 + # load vision embeddings and encoder
729 + self.vision_embed_tokens.load_weights(
730 + weights_group["vision_embed_tokens"])
731 +
732 + # load llm backbone
733 + self.language_model.load_weights(weights_group["language_model"])
vllm/model_executor/models/phimoe.py
+55 -17
@@ -21,7 +21,7 @@
21 # See the License for the specific language governing permissions and
22 # limitations under the License.
23 """Inference-only PhiMoE model."""
24 -from typing import Iterable, List, Optional, Tuple
24 +from typing import Iterable, List, Optional, Tuple, Union
25
26 import torch
27 from torch import nn
@@ -29,7 +29,7 @@ from transformers.configuration_utils import PretrainedConfig
29
30 from vllm.attention import Attention, AttentionMetadata
31 from vllm.config import CacheConfig, LoRAConfig
32 -from vllm.distributed import get_tensor_model_parallel_world_size
32 +from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
33 from vllm.model_executor.layers.fused_moe import FusedMoE
34 from vllm.model_executor.layers.linear import (QKVParallelLinear,
35 ReplicatedLinear,
@@ -46,7 +46,9 @@ from vllm.model_executor.model_loader.weight_utils import (
46 from vllm.model_executor.sampling_metadata import SamplingMetadata
47 from vllm.sequence import IntermediateTensors
48
49 -from .interfaces import SupportsLoRA
49 +from .interfaces import SupportsLoRA, SupportsPP
50 +from .utils import (is_pp_missing_parameter,
51 + make_empty_intermediate_tensors_factory, make_layers)
52
53
54 class PhiMoEConfig(PretrainedConfig):
@@ -435,6 +437,7 @@ class PhiMoEModel(nn.Module):
437 cache_config: Optional[CacheConfig] = None,
438 quant_config: Optional[QuantizationConfig] = None,
439 lora_config: Optional[LoRAConfig] = None,
440 + prefix: str = "",
441 ) -> None:
442 super().__init__()
443 self.padding_idx = config.pad_token_id
@@ -448,33 +451,56 @@ class PhiMoEModel(nn.Module):
451 config.hidden_size,
452 org_num_embeddings=config.vocab_size,
453 )
451 - self.layers = nn.ModuleList([
452 - PhiMoEDecoderLayer(config, cache_config, quant_config=quant_config)
453 - for _ in range(config.num_hidden_layers)
454 - ])
454 + self.start_layer, self.end_layer, self.layers = make_layers(
455 + config.num_hidden_layers,
456 + lambda prefix: PhiMoEDecoderLayer(config, cache_config,
457 + quant_config),
458 + prefix=f"{prefix}.layers")
459 self.norm = nn.LayerNorm(config.hidden_size,
460 eps=config.rms_norm_eps,
461 elementwise_affine=True)
462
463 + self.make_empty_intermediate_tensors = (
464 + make_empty_intermediate_tensors_factory(
465 + ["hidden_states", "residual"], config.hidden_size))
466 +
467 def forward(
468 self,
469 input_ids: torch.Tensor,
470 positions: torch.Tensor,
471 kv_caches: List[torch.Tensor],
472 attn_metadata: AttentionMetadata,
465 - ) -> torch.Tensor:
466 - hidden_states = self.embed_tokens(input_ids)
467 - residual = None
468 - for i in range(len(self.layers)):
473 + intermediate_tensors: Optional[IntermediateTensors],
474 + ) -> Union[torch.Tensor, IntermediateTensors]:
475 + if get_pp_group().is_first_rank:
476 + hidden_states = self.embed_tokens(input_ids)
477 + residual = None
478 + else:
479 + assert intermediate_tensors is not None
480 + hidden_states = intermediate_tensors["hidden_states"]
481 + residual = intermediate_tensors["residual"]
482 +
483 + for i in range(self.start_layer, self.end_layer):
484 layer = self.layers[i]
470 - hidden_states, residual = layer(positions, hidden_states,
471 - kv_caches[i], attn_metadata,
472 - residual)
485 + hidden_states, residual = layer(
486 + positions,
487 + hidden_states,
488 + kv_caches[i - self.start_layer],
489 + attn_metadata,
490 + residual,
491 + )
492 +
493 + if not get_pp_group().is_last_rank:
494 + return IntermediateTensors({
495 + "hidden_states": hidden_states,
496 + "residual": residual
497 + })
498 +
499 hidden_states = self.norm(hidden_states)
500 return hidden_states
501
502
477 -class PhiMoEForCausalLM(nn.Module, SupportsLoRA):
503 +class PhiMoEForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
504 fall_back_to_pt_during_load = False
505
506 packed_modules_mapping = {
@@ -537,6 +563,9 @@ class PhiMoEForCausalLM(nn.Module, SupportsLoRA):
563 config.vocab_size)
564 self.sampler = Sampler()
565
566 + self.make_empty_intermediate_tensors = (
567 + self.model.make_empty_intermediate_tensors)
568 +
569 def forward(
570 self,
571 input_ids: torch.Tensor,
@@ -544,9 +573,9 @@ class PhiMoEForCausalLM(nn.Module, SupportsLoRA):
573 kv_caches: List[torch.Tensor],
574 attn_metadata: AttentionMetadata,
575 intermediate_tensors: Optional[IntermediateTensors] = None,
547 - ) -> torch.Tensor:
576 + ) -> Union[torch.Tensor, IntermediateTensors]:
577 hidden_states = self.model(input_ids, positions, kv_caches,
549 - attn_metadata)
578 + attn_metadata, intermediate_tensors)
579 return hidden_states
580
581 def compute_logits(self, hidden_states: torch.Tensor,
@@ -589,6 +618,9 @@ class PhiMoEForCausalLM(nn.Module, SupportsLoRA):
618 # Skip loading extra bias for GPTQ models.
619 if name.endswith(".bias") and name not in params_dict:
620 continue
621 + # Skip layers on other devices.
622 + if is_pp_missing_parameter(name, self):
623 + continue
624 param = params_dict[name]
625 weight_loader = param.weight_loader
626 weight_loader(param, loaded_weight, shard_id)
@@ -599,6 +631,9 @@ class PhiMoEForCausalLM(nn.Module, SupportsLoRA):
631 if weight_name not in name:
632 continue
633 name = name.replace(weight_name, param_name)
634 + # Skip layers on other devices.
635 + if is_pp_missing_parameter(name, self):
636 + continue
637 param = params_dict[name]
638 weight_loader = param.weight_loader
639 weight_loader(
@@ -613,6 +648,9 @@ class PhiMoEForCausalLM(nn.Module, SupportsLoRA):
648 # Skip loading extra bias for GPTQ models.
649 if name.endswith(".bias") and name not in params_dict:
650 continue
651 + # Skip layers on other devices.
652 + if is_pp_missing_parameter(name, self):
653 + continue
654 # Remapping the name of FP8 kv-scale.
655 name = maybe_remap_kv_scale_name(name, params_dict)
656 if name is None:
vllm/model_executor/models/pixtral.py
+32 -16
@@ -1,4 +1,5 @@
1 from dataclasses import dataclass, fields
2 +from functools import cached_property
3 from itertools import tee
4 from typing import Iterable, List, Mapping, Optional, Tuple, Union
5
@@ -16,7 +17,7 @@ from vllm.config import CacheConfig, MultiModalConfig
17 from vllm.inputs import INPUT_REGISTRY, InputContext, LLMInputs
18 from vllm.model_executor.layers.layernorm import RMSNorm
19 from vllm.model_executor.layers.quantization import QuantizationConfig
19 -from vllm.model_executor.layers.sampler import SamplerOutput
20 +from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
21 from vllm.model_executor.model_loader.weight_utils import default_weight_loader
22 from vllm.model_executor.models.utils import merge_multimodal_embeddings
23 from vllm.model_executor.sampling_metadata import SamplingMetadata
@@ -25,7 +26,7 @@ from vllm.multimodal.base import MultiModalInputs
26 from vllm.multimodal.utils import cached_get_tokenizer
27 from vllm.sequence import IntermediateTensors, SequenceData
28
28 -from .interfaces import SupportsMultiModal
29 +from .interfaces import SupportsMultiModal, SupportsPP
30 from .utils import init_vllm_registered_model
31
32
@@ -126,7 +127,8 @@ def input_processor_for_pixtral(ctx: InputContext, llm_inputs: LLMInputs):
127 @MULTIMODAL_REGISTRY.register_max_image_tokens(get_max_pixtral_image_tokens)
128 @INPUT_REGISTRY.register_dummy_data(dummy_data_for_pixtral)
129 @INPUT_REGISTRY.register_input_processor(input_processor_for_pixtral)
129 -class PixtralForConditionalGeneration(nn.Module, SupportsMultiModal):
130 +class PixtralForConditionalGeneration(nn.Module, SupportsMultiModal,
131 + SupportsPP):
132
133 def __init__(self,
134 config: PretrainedConfig,
@@ -155,6 +157,16 @@ class PixtralForConditionalGeneration(nn.Module, SupportsMultiModal):
157 self.vision_language_adapter = VisionLanguageAdapter(
158 self.vision_args, dim=config.text_config.hidden_size)
159
160 + self.make_empty_intermediate_tensors = (
161 + self.language_model.make_empty_intermediate_tensors)
162 +
163 + @cached_property
164 + def sampler(self):
165 + if hasattr(self.language_model, "sampler"):
166 + return self.language_model.sampler
167 +
168 + return Sampler()
169 +
170 def forward(
171 self,
172 input_ids: torch.Tensor,
@@ -163,32 +175,36 @@ class PixtralForConditionalGeneration(nn.Module, SupportsMultiModal):
175 attn_metadata: AttentionMetadata,
176 intermediate_tensors: Optional[IntermediateTensors] = None,
177 **kwargs: object,
166 - ) -> SamplerOutput:
178 + ) -> Union[torch.Tensor, IntermediateTensors]:
179 """Run forward pass for pixtral.
180
181 TODO
182
183 """
172 - image_input = self._parse_and_validate_image_input(**kwargs)
184 + if intermediate_tensors is not None:
185 + input_ids = None
186 + inputs_embeds = None
187 + else:
188 + image_input = self._parse_and_validate_image_input(**kwargs)
189
174 - if image_input is not None:
175 - vision_embeddings = self._process_image_input(image_input)
176 - inputs_embeds = self.language_model.model.get_input_embeddings(
177 - input_ids)
190 + if image_input is not None:
191 + vision_embeddings = self._process_image_input(image_input)
192 + inputs_embeds = self.language_model.model.get_input_embeddings(
193 + input_ids)
194
179 - inputs_embeds = merge_multimodal_embeddings(
180 - input_ids, inputs_embeds, vision_embeddings,
181 - self.vision_args.image_token_id)
195 + inputs_embeds = merge_multimodal_embeddings(
196 + input_ids, inputs_embeds, vision_embeddings,
197 + self.vision_args.image_token_id)
198
183 - input_ids = None
184 - else:
185 - inputs_embeds = None
199 + input_ids = None
200 + else:
201 + inputs_embeds = None
202
203 hidden_states = self.language_model.model(input_ids,
204 positions,
205 kv_caches,
206 attn_metadata,
191 - None,
207 + intermediate_tensors,
208 inputs_embeds=inputs_embeds)
209
210 return hidden_states
vllm/model_executor/models/qwen.py
+26 -28
@@ -31,15 +31,13 @@ from vllm.model_executor.layers.linear import (ColumnParallelLinear,
31 QKVParallelLinear,
32 RowParallelLinear)
33 from vllm.model_executor.layers.logits_processor import LogitsProcessor
34 -from vllm.model_executor.layers.quantization.base_config import (
35 - QuantizationConfig)
34 +from vllm.model_executor.layers.quantization import QuantizationConfig
35 from vllm.model_executor.layers.resampler import Resampler2, get_abs_pos
36 from vllm.model_executor.layers.rotary_embedding import get_rope
37 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
38 from vllm.model_executor.layers.vocab_parallel_embedding import (
39 ParallelLMHead, VocabParallelEmbedding)
40 from vllm.model_executor.model_loader.weight_utils import default_weight_loader
42 -from vllm.model_executor.models.interfaces import SupportsMultiModal
41 from vllm.model_executor.sampling_metadata import SamplingMetadata
42 from vllm.multimodal import MULTIMODAL_REGISTRY
43 from vllm.multimodal.base import MultiModalInputs
@@ -47,7 +45,9 @@ from vllm.multimodal.utils import cached_get_tokenizer
45 from vllm.sequence import IntermediateTensors, SequenceData
46 from vllm.utils import is_list_of
47
50 -from .utils import flatten_bn, is_pp_missing_parameter, make_layers
48 +from .interfaces import SupportsMultiModal, SupportsPP
49 +from .utils import (flatten_bn, is_pp_missing_parameter,
50 + make_empty_intermediate_tensors_factory, make_layers)
51
52 logger = init_logger(__name__)
53
@@ -568,6 +568,9 @@ class QWenModel(nn.Module):
568 lambda prefix: QWenBlock(config, cache_config, quant_config),
569 prefix=f"{prefix}.h")
570 self.ln_f = RMSNorm(config.hidden_size, eps=config.layer_norm_epsilon)
571 + self.make_empty_intermediate_tensors = (
572 + make_empty_intermediate_tensors_factory(
573 + ["hidden_states", "residual"], config.hidden_size))
574 self.visual = VisionTransformer(**config.visual,
575 quant_config=quant_config) if hasattr(
576 config, "visual") else None
@@ -580,7 +583,7 @@ class QWenModel(nn.Module):
583 attn_metadata: AttentionMetadata,
584 intermediate_tensors: Optional[IntermediateTensors],
585 pixel_values: Optional[QwenImageInputs],
583 - ) -> torch.Tensor:
586 + ) -> Union[torch.Tensor, IntermediateTensors]:
587 img_pos = None
588 # If pixel / visual embeddings are provided, this is a visual model
589 if pixel_values is not None and self.visual is not None:
@@ -860,7 +863,7 @@ def dummy_data_for_qwen(
863 @MULTIMODAL_REGISTRY.register_max_image_tokens(MAX_QWEN_IMG_TOKENS)
864 @INPUT_REGISTRY.register_dummy_data(dummy_data_for_qwen)
865 @INPUT_REGISTRY.register_input_processor(input_processor_for_qwen)
863 -class QWenLMHeadModel(nn.Module, SupportsMultiModal):
866 +class QWenLMHeadModel(nn.Module, SupportsMultiModal, SupportsPP):
867
868 def __init__(
869 self,
@@ -881,6 +884,8 @@ class QWenLMHeadModel(nn.Module, SupportsMultiModal):
884 self.lm_head.weight = self.transformer.wte.weight
885 self.logits_processor = LogitsProcessor(config.vocab_size)
886 self.sampler = Sampler()
887 + self.make_empty_intermediate_tensors = (
888 + self.transformer.make_empty_intermediate_tensors)
889
890 def _get_image_input_type(
891 self,
@@ -912,33 +917,26 @@ class QWenLMHeadModel(nn.Module, SupportsMultiModal):
917 )
918 return None
919
915 - def forward(self,
916 - input_ids: torch.Tensor,
917 - positions: torch.Tensor,
918 - kv_caches: List[torch.Tensor],
919 - attn_metadata: AttentionMetadata,
920 - intermediate_tensors: Optional[IntermediateTensors] = None,
921 - pixel_values: Optional[torch.Tensor] = None) -> torch.Tensor:
922 - pixel_values = self._get_image_input_type(pixel_values)
920 + def forward(
921 + self,
922 + input_ids: torch.Tensor,
923 + positions: torch.Tensor,
924 + kv_caches: List[torch.Tensor],
925 + attn_metadata: AttentionMetadata,
926 + intermediate_tensors: Optional[IntermediateTensors] = None,
927 + pixel_values: Optional[torch.Tensor] = None
928 + ) -> Union[torch.Tensor, IntermediateTensors]:
929 + if intermediate_tensors is not None:
930 + input_ids = None
931 + pixel_values = None
932 + else:
933 + pixel_values = self._get_image_input_type(pixel_values)
934 +
935 hidden_states = self.transformer(input_ids, positions, kv_caches,
936 attn_metadata, intermediate_tensors,
937 pixel_values)
938 return hidden_states
939
928 - def make_empty_intermediate_tensors(
929 - self, batch_size: int, dtype: torch.dtype,
930 - device: torch.device) -> IntermediateTensors:
931 - return IntermediateTensors({
932 - "hidden_states":
933 - torch.zeros((batch_size, self.config.hidden_size),
934 - dtype=dtype,
935 - device=device),
936 - "residual":
937 - torch.zeros((batch_size, self.config.hidden_size),
938 - dtype=dtype,
939 - device=device),
940 - })
941 -
940 def compute_logits(
941 self,
942 hidden_states: torch.Tensor,
vllm/model_executor/models/qwen2.py
+13 -22
@@ -22,7 +22,7 @@
22 # See the License for the specific language governing permissions and
23 # limitations under the License.
24 """Inference-only Qwen2 model compatible with HuggingFace weights."""
25 -from typing import Iterable, List, Optional, Tuple
25 +from typing import Iterable, List, Optional, Tuple, Union
26
27 import torch
28 from torch import nn
@@ -37,8 +37,7 @@ from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
37 QKVParallelLinear,
38 RowParallelLinear)
39 from vllm.model_executor.layers.logits_processor import LogitsProcessor
40 -from vllm.model_executor.layers.quantization.base_config import (
41 - QuantizationConfig)
40 +from vllm.model_executor.layers.quantization import QuantizationConfig
41 from vllm.model_executor.layers.rotary_embedding import get_rope
42 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
43 from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -48,8 +47,9 @@ from vllm.model_executor.model_loader.weight_utils import (
47 from vllm.model_executor.sampling_metadata import SamplingMetadata
48 from vllm.sequence import IntermediateTensors
49
51 -from .interfaces import SupportsLoRA
52 -from .utils import PPMissingLayer, is_pp_missing_parameter, make_layers
50 +from .interfaces import SupportsLoRA, SupportsPP
51 +from .utils import (PPMissingLayer, is_pp_missing_parameter,
52 + make_empty_intermediate_tensors_factory, make_layers)
53
54
55 class Qwen2MLP(nn.Module):
@@ -253,6 +253,9 @@ class Qwen2Model(nn.Module):
253 prefix=f"{prefix}.layers",
254 )
255
256 + self.make_empty_intermediate_tensors = (
257 + make_empty_intermediate_tensors_factory(
258 + ["hidden_states", "residual"], config.hidden_size))
259 if get_pp_group().is_last_rank:
260 self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
261 else:
@@ -269,7 +272,7 @@ class Qwen2Model(nn.Module):
272 attn_metadata: AttentionMetadata,
273 intermediate_tensors: Optional[IntermediateTensors] = None,
274 inputs_embeds: Optional[torch.Tensor] = None,
272 - ) -> torch.Tensor:
275 + ) -> Union[torch.Tensor, IntermediateTensors]:
276 if get_pp_group().is_first_rank:
277 if inputs_embeds is not None:
278 hidden_states = inputs_embeds
@@ -298,7 +301,7 @@ class Qwen2Model(nn.Module):
301 return hidden_states
302
303
301 -class Qwen2ForCausalLM(nn.Module, SupportsLoRA):
304 +class Qwen2ForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
305 packed_modules_mapping = {
306 "qkv_proj": [
307 "q_proj",
@@ -357,6 +360,8 @@ class Qwen2ForCausalLM(nn.Module, SupportsLoRA):
360
361 self.logits_processor = LogitsProcessor(config.vocab_size)
362 self.sampler = Sampler()
363 + self.make_empty_intermediate_tensors = (
364 + self.model.make_empty_intermediate_tensors)
365
366 def forward(
367 self,
@@ -365,7 +370,7 @@ class Qwen2ForCausalLM(nn.Module, SupportsLoRA):
370 kv_caches: List[torch.Tensor],
371 attn_metadata: AttentionMetadata,
372 intermediate_tensors: Optional[IntermediateTensors] = None,
368 - ) -> torch.Tensor:
373 + ) -> Union[torch.Tensor, IntermediateTensors]:
374 hidden_states = self.model(input_ids, positions, kv_caches,
375 attn_metadata, intermediate_tensors)
376 return hidden_states
@@ -379,20 +384,6 @@ class Qwen2ForCausalLM(nn.Module, SupportsLoRA):
384 sampling_metadata)
385 return logits
386
382 - def make_empty_intermediate_tensors(
383 - self, batch_size: int, dtype: torch.dtype,
384 - device: torch.device) -> IntermediateTensors:
385 - return IntermediateTensors({
386 - "hidden_states":
387 - torch.zeros((batch_size, self.config.hidden_size),
388 - dtype=dtype,
389 - device=device),
390 - "residual":
391 - torch.zeros((batch_size, self.config.hidden_size),
392 - dtype=dtype,
393 - device=device),
394 - })
395 -
387 def sample(
388 self,
389 logits: torch.Tensor,
vllm/model_executor/models/qwen2_moe.py
+13 -21
@@ -22,7 +22,7 @@
22 # See the License for the specific language governing permissions and
23 # limitations under the License.
24 """Inference-only Qwen2MoE model compatible with HuggingFace weights."""
25 -from typing import Any, Dict, Iterable, List, Optional, Tuple
25 +from typing import Any, Dict, Iterable, List, Optional, Tuple, Union
26
27 import torch
28 import torch.nn.functional as F
@@ -42,8 +42,7 @@ from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
42 ReplicatedLinear,
43 RowParallelLinear)
44 from vllm.model_executor.layers.logits_processor import LogitsProcessor
45 -from vllm.model_executor.layers.quantization.base_config import (
46 - QuantizationConfig)
45 +from vllm.model_executor.layers.quantization import QuantizationConfig
46 from vllm.model_executor.layers.rotary_embedding import get_rope
47 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
48 from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -53,7 +52,9 @@ from vllm.model_executor.sampling_metadata import SamplingMetadata
52 from vllm.sequence import IntermediateTensors
53 from vllm.utils import print_warning_once
54
56 -from .utils import is_pp_missing_parameter, make_layers
55 +from .interfaces import SupportsPP
56 +from .utils import (is_pp_missing_parameter,
57 + make_empty_intermediate_tensors_factory, make_layers)
58
59
60 class Qwen2MoeMLP(nn.Module):
@@ -338,6 +339,9 @@ class Qwen2MoeModel(nn.Module):
339 prefix=f"{prefix}.layers",
340 )
341 self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
342 + self.make_empty_intermediate_tensors = (
343 + make_empty_intermediate_tensors_factory(
344 + ["hidden_states", "residual"], config.hidden_size))
345
346 def forward(
347 self,
@@ -346,7 +350,7 @@ class Qwen2MoeModel(nn.Module):
350 kv_caches: List[torch.Tensor],
351 attn_metadata: AttentionMetadata,
352 intermediate_tensors: Optional[IntermediateTensors] = None,
349 - ) -> torch.Tensor:
353 + ) -> Union[torch.Tensor, IntermediateTensors]:
354 if get_pp_group().is_first_rank:
355 hidden_states = self.embed_tokens(input_ids)
356 residual = None
@@ -368,7 +372,7 @@ class Qwen2MoeModel(nn.Module):
372 return hidden_states
373
374
371 -class Qwen2MoeForCausalLM(nn.Module):
375 +class Qwen2MoeForCausalLM(nn.Module, SupportsPP):
376
377 fall_back_to_pt_during_load = False
378
@@ -389,6 +393,8 @@ class Qwen2MoeForCausalLM(nn.Module):
393 self.lm_head.weight = self.model.embed_tokens.weight
394 self.logits_processor = LogitsProcessor(config.vocab_size)
395 self.sampler = Sampler()
396 + self.make_empty_intermediate_tensors = (
397 + self.model.make_empty_intermediate_tensors)
398
399 def forward(
400 self,
@@ -397,7 +403,7 @@ class Qwen2MoeForCausalLM(nn.Module):
403 kv_caches: List[torch.Tensor],
404 attn_metadata: AttentionMetadata,
405 intermediate_tensors: Optional[IntermediateTensors] = None,
400 - ) -> torch.Tensor:
406 + ) -> Union[torch.Tensor, IntermediateTensors]:
407 hidden_states = self.model(input_ids, positions, kv_caches,
408 attn_metadata, intermediate_tensors)
409 return hidden_states
@@ -411,20 +417,6 @@ class Qwen2MoeForCausalLM(nn.Module):
417 sampling_metadata)
418 return logits
419
414 - def make_empty_intermediate_tensors(
415 - self, batch_size: int, dtype: torch.dtype,
416 - device: torch.device) -> IntermediateTensors:
417 - return IntermediateTensors({
418 - "hidden_states":
419 - torch.zeros((batch_size, self.config.hidden_size),
420 - dtype=dtype,
421 - device=device),
422 - "residual":
423 - torch.zeros((batch_size, self.config.hidden_size),
424 - dtype=dtype,
425 - device=device),
426 - })
427 -
420 def sample(
421 self,
422 logits: Optional[torch.Tensor],
vllm/model_executor/models/qwen2_vl.py
+38 -35
@@ -55,7 +55,6 @@ from vllm.model_executor.layers.quantization import QuantizationConfig
55 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
56 from vllm.model_executor.layers.vocab_parallel_embedding import ParallelLMHead
57 from vllm.model_executor.model_loader.weight_utils import default_weight_loader
58 -from vllm.model_executor.models.interfaces import SupportsMultiModal
58 from vllm.model_executor.models.qwen2 import Qwen2Model
59 from vllm.multimodal import (MULTIMODAL_REGISTRY, MultiModalDataDict,
60 MultiModalInputs)
@@ -68,6 +67,7 @@ from vllm.transformers_utils.configs.qwen2vl import (Qwen2VLConfig,
67 from vllm.transformers_utils.processor import get_processor
68 from vllm.utils import is_cpu
69
70 +from .interfaces import SupportsMultiModal, SupportsPP
71 from .utils import (PPMissingLayer, is_pp_missing_parameter,
72 make_empty_intermediate_tensors_factory)
73
@@ -883,7 +883,8 @@ def input_processor_for_qwen2_vl(ctx: InputContext,
883 "video", get_max_qwen2_vl_video_tokens)
884 @INPUT_REGISTRY.register_dummy_data(dummy_data_for_qwen2_vl)
885 @INPUT_REGISTRY.register_input_processor(input_processor_for_qwen2_vl)
886 -class Qwen2VLForConditionalGeneration(nn.Module, SupportsMultiModal):
886 +class Qwen2VLForConditionalGeneration(nn.Module, SupportsMultiModal,
887 + SupportsPP):
888
889 def __init__(self,
890 config: Qwen2VLConfig,
@@ -1027,7 +1028,7 @@ class Qwen2VLForConditionalGeneration(nn.Module, SupportsMultiModal):
1028 attn_metadata: AttentionMetadata,
1029 intermediate_tensors: Optional[IntermediateTensors] = None,
1030 **kwargs: object,
1030 - ) -> SamplerOutput:
1031 + ) -> Union[torch.Tensor, IntermediateTensors]:
1032 """Run forward pass for Qwen2-VL.
1033
1034 Args:
@@ -1047,41 +1048,43 @@ class Qwen2VLForConditionalGeneration(nn.Module, SupportsMultiModal):
1048 video_grid_thw: Tensor `(n_videos, 3)` of video 3D grid in LLM.
1049 `None` if no videos are passed.
1050 """
1050 -
1051 - image_input = self._parse_and_validate_image_input(**kwargs)
1052 - video_input = self._parse_and_validate_video_input(**kwargs)
1053 -
1054 - if (image_input is None
1055 - and video_input is None) or not get_pp_group().is_first_rank:
1051 + if intermediate_tensors is not None:
1052 + input_ids = None
1053 inputs_embeds = None
1054 else:
1058 - if getattr(self.config, "rope_scaling", {}).get("type",
1059 - None) == "mrope":
1060 - assert positions.ndim == 2 and positions.size(0) == 3, (
1061 - "multimodal section rotary embedding requires "
1062 - f"(3, seq_len) positions, but got {positions.size()}")
1063 -
1064 - inputs_embeds = self.model.embed_tokens(input_ids)
1065 -
1066 - if image_input is not None:
1067 - image_embeds = self._process_image_input(image_input)
1068 - inputs_embeds = self._merge_multimodal_embeddings(
1069 - input_ids,
1070 - inputs_embeds,
1071 - image_embeds,
1072 - placeholder_token_id=self.config.image_token_id,
1073 - )
1055 + image_input = self._parse_and_validate_image_input(**kwargs)
1056 + video_input = self._parse_and_validate_video_input(**kwargs)
1057
1075 - if video_input is not None:
1076 - video_embeds = self._process_video_input(video_input)
1077 - inputs_embeds = self._merge_multimodal_embeddings(
1078 - input_ids,
1079 - inputs_embeds,
1080 - video_embeds,
1081 - placeholder_token_id=self.config.video_token_id,
1082 - )
1083 -
1084 - input_ids = None
1058 + if image_input is None and video_input is None:
1059 + inputs_embeds = None
1060 + else:
1061 + rope_scaling = getattr(self.config, "rope_scaling", {})
1062 + if rope_scaling.get("type", None) == "mrope":
1063 + assert positions.ndim == 2 and positions.size(0) == 3, (
1064 + "multimodal section rotary embedding requires "
1065 + f"(3, seq_len) positions, but got {positions.size()}")
1066 +
1067 + inputs_embeds = self.model.embed_tokens(input_ids)
1068 +
1069 + if image_input is not None:
1070 + image_embeds = self._process_image_input(image_input)
1071 + inputs_embeds = self._merge_multimodal_embeddings(
1072 + input_ids,
1073 + inputs_embeds,
1074 + image_embeds,
1075 + placeholder_token_id=self.config.image_token_id,
1076 + )
1077 +
1078 + if video_input is not None:
1079 + video_embeds = self._process_video_input(video_input)
1080 + inputs_embeds = self._merge_multimodal_embeddings(
1081 + input_ids,
1082 + inputs_embeds,
1083 + video_embeds,
1084 + placeholder_token_id=self.config.video_token_id,
1085 + )
1086 +
1087 + input_ids = None
1088
1089 hidden_states = self.model(
1090 input_ids=input_ids,
vllm/model_executor/models/siglip.py
+2 -2
@@ -246,7 +246,7 @@ class SiglipParallelAttention(nn.Module):
246
247 def __init__(
248 self,
249 - config,
249 + config: SiglipVisionConfig,
250 quant_config: Optional[QuantizationConfig] = None,
251 ):
252 super().__init__()
@@ -312,7 +312,7 @@ class SiglipMLP(nn.Module):
312
313 def __init__(
314 self,
315 - config,
315 + config: SiglipVisionConfig,
316 quant_config: Optional[QuantizationConfig] = None,
317 ):
318 super().__init__()
vllm/model_executor/models/solar.py
+18 -29
@@ -26,6 +26,7 @@ from typing import Any, Dict, Iterable, List, Optional, Tuple, Union
26
27 import torch
28 from torch import nn
29 +from transformers import PretrainedConfig
30
31 from vllm.attention import Attention, AttentionMetadata
32 from vllm.config import CacheConfig, LoRAConfig
@@ -37,8 +38,7 @@ from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
38 QKVParallelLinear,
39 RowParallelLinear)
40 from vllm.model_executor.layers.logits_processor import LogitsProcessor
40 -from vllm.model_executor.layers.quantization.base_config import (
41 - QuantizationConfig)
41 +from vllm.model_executor.layers.quantization import QuantizationConfig
42 from vllm.model_executor.layers.quantization.compressed_tensors.utils import (
43 get_compressed_tensors_cache_scale)
44 from vllm.model_executor.layers.rotary_embedding import get_rope
@@ -47,14 +47,14 @@ from vllm.model_executor.layers.vocab_parallel_embedding import (
47 DEFAULT_VOCAB_PADDING_SIZE, ParallelLMHead, VocabParallelEmbedding)
48 from vllm.model_executor.model_loader.weight_utils import (
49 default_weight_loader, kv_cache_scales_loader, maybe_remap_kv_scale_name)
50 -from vllm.model_executor.models.interfaces import SupportsLoRA
51 -from vllm.model_executor.models.utils import (PPMissingLayer,
52 - is_pp_missing_parameter,
53 - make_layers)
50 from vllm.model_executor.sampling_metadata import SamplingMetadata
51 from vllm.sequence import IntermediateTensors
52 from vllm.utils import is_hip
53
54 +from .interfaces import SupportsLoRA, SupportsPP
55 +from .utils import (PPMissingLayer, is_pp_missing_parameter,
56 + make_empty_intermediate_tensors_factory, make_layers)
57 +
58
59 class SolarMLP(nn.Module):
60
@@ -98,7 +98,7 @@ class SolarAttention(nn.Module):
98
99 def __init__(
100 self,
101 - config,
101 + config: PretrainedConfig,
102 hidden_size: int,
103 num_heads: int,
104 num_kv_heads: int,
@@ -187,7 +187,7 @@ class SolarDecoderLayer(nn.Module):
187
188 def __init__(
189 self,
190 - config,
190 + config: PretrainedConfig,
191 cache_config: Optional[CacheConfig] = None,
192 quant_config: Optional[QuantizationConfig] = None,
193 prefix: str = "",
@@ -267,7 +267,7 @@ class SolarModel(nn.Module):
267
268 def __init__(
269 self,
270 - config,
270 + config: PretrainedConfig,
271 cache_config: Optional[CacheConfig] = None,
272 quant_config: Optional[QuantizationConfig] = None,
273 lora_config: Optional[LoRAConfig] = None,
@@ -304,6 +304,10 @@ class SolarModel(nn.Module):
304 else:
305 self.norm = PPMissingLayer()
306
307 + self.make_empty_intermediate_tensors = (
308 + make_empty_intermediate_tensors_factory(
309 + ["hidden_states", "residual"], config.hidden_size))
310 +
311 def get_input_embeddings(self, input_ids: torch.Tensor) -> torch.Tensor:
312 return self.embed_tokens(input_ids)
313
@@ -368,7 +372,7 @@ class SolarModel(nn.Module):
372 return hidden_states
373
374
371 -class SolarForCausalLM(nn.Module, SupportsLoRA):
375 +class SolarForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
376 packed_modules_mapping = {
377 "qkv_proj": [
378 "q_proj",
@@ -406,7 +410,7 @@ class SolarForCausalLM(nn.Module, SupportsLoRA):
410
411 def __init__(
412 self,
409 - config,
413 + config: PretrainedConfig,
414 cache_config: Optional[CacheConfig] = None,
415 quant_config: Optional[QuantizationConfig] = None,
416 lora_config: Optional[LoRAConfig] = None,
@@ -448,6 +452,9 @@ class SolarForCausalLM(nn.Module, SupportsLoRA):
452 else:
453 self.lm_head = PPMissingLayer()
454
455 + self.make_empty_intermediate_tensors = (
456 + self.model.make_empty_intermediate_tensors)
457 +
458 def forward(
459 self,
460 input_ids: torch.Tensor,
@@ -474,24 +481,6 @@ class SolarForCausalLM(nn.Module, SupportsLoRA):
481 next_tokens = self.sampler(logits, sampling_metadata)
482 return next_tokens
483
477 - def make_empty_intermediate_tensors(
478 - self, batch_size: int, dtype: torch.dtype,
479 - device: torch.device) -> IntermediateTensors:
480 - return IntermediateTensors({
481 - "hidden_states":
482 - torch.zeros(
483 - (batch_size, self.config.hidden_size),
484 - dtype=dtype,
485 - device=device,
486 - ),
487 - "residual":
488 - torch.zeros(
489 - (batch_size, self.config.hidden_size),
490 - dtype=dtype,
491 - device=device,
492 - ),
493 - })
494 -
484 def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
485 stacked_params_mapping = [
486 # (param_name, shard_name, shard_id)
vllm/model_executor/models/stablelm.py
+38 -16
@@ -19,7 +19,7 @@
19 # https://huggingface.co/stabilityai/stablelm-3b-4e1t/blob/main/config.json
20 """Inference-only StabeLM (https://github.com/Stability-AI/StableLM)
21 model compatible with HuggingFace weights."""
22 -from typing import Iterable, List, Optional, Tuple
22 +from typing import Iterable, List, Optional, Tuple, Union
23
24 import torch
25 from torch import nn
@@ -27,14 +27,13 @@ from transformers import PretrainedConfig
27
28 from vllm.attention import Attention, AttentionMetadata
29 from vllm.config import CacheConfig
30 -from vllm.distributed import get_tensor_model_parallel_world_size
30 +from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
31 from vllm.model_executor.layers.activation import SiluAndMul
32 from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
33 QKVParallelLinear,
34 RowParallelLinear)
35 from vllm.model_executor.layers.logits_processor import LogitsProcessor
36 -from vllm.model_executor.layers.quantization.base_config import (
37 - QuantizationConfig)
36 +from vllm.model_executor.layers.quantization import QuantizationConfig
37 from vllm.model_executor.layers.rotary_embedding import get_rope
38 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
39 from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -43,6 +42,10 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
42 from vllm.model_executor.sampling_metadata import SamplingMetadata
43 from vllm.sequence import IntermediateTensors
44
45 +from .interfaces import SupportsPP
46 +from .utils import (is_pp_missing_parameter,
47 + make_empty_intermediate_tensors_factory, make_layers)
48 +
49
50 class StablelmMLP(nn.Module):
51
@@ -194,19 +197,25 @@ class StableLMEpochModel(nn.Module):
197 def __init__(self,
198 config: PretrainedConfig,
199 cache_config: Optional[CacheConfig] = None,
197 - quant_config: Optional[QuantizationConfig] = None) -> None:
200 + quant_config: Optional[QuantizationConfig] = None,
201 + prefix: str = '') -> None:
202 super().__init__()
203 self.embed_tokens = VocabParallelEmbedding(
204 config.vocab_size,
205 config.hidden_size,
206 )
203 - self.layers = nn.ModuleList([
204 - StablelmDecoderLayer(config, cache_config, quant_config)
205 - for _ in range(config.num_hidden_layers)
206 - ])
207 + self.start_layer, self.end_layer, self.layers = make_layers(
208 + config.num_hidden_layers,
209 + lambda prefix: StablelmDecoderLayer(config, cache_config,
210 + quant_config),
211 + prefix=f"{prefix}.layers",
212 + )
213 norm_eps = getattr(config, "norm_eps",
214 getattr(config, "layer_norm_eps", 1e-05))
215 self.norm = nn.LayerNorm(config.hidden_size, eps=norm_eps)
216 + self.make_empty_intermediate_tensors = (
217 + make_empty_intermediate_tensors_factory(["hidden_states"],
218 + config.hidden_size))
219
220 def forward(
221 self,
@@ -214,21 +223,28 @@ class StableLMEpochModel(nn.Module):
223 positions: torch.Tensor,
224 kv_caches: List[torch.Tensor],
225 attn_metadata: AttentionMetadata,
217 - ) -> torch.Tensor:
218 - hidden_states = self.embed_tokens(input_ids)
219 - for i in range(len(self.layers)):
226 + intermediate_tensors: Optional[IntermediateTensors],
227 + ) -> Union[torch.Tensor, IntermediateTensors]:
228 + if get_pp_group().is_first_rank:
229 + hidden_states = self.embed_tokens(input_ids)
230 + else:
231 + assert intermediate_tensors is not None
232 + hidden_states = intermediate_tensors["hidden_states"]
233 + for i in range(self.start_layer, self.end_layer):
234 layer = self.layers[i]
235 hidden_states, residual = layer(
236 positions,
237 hidden_states,
224 - kv_caches[i],
238 + kv_caches[i - self.start_layer],
239 attn_metadata,
240 )
241 + if not get_pp_group().is_last_rank:
242 + return IntermediateTensors({"hidden_states": hidden_states})
243 hidden_states = self.norm(hidden_states)
244 return hidden_states
245
246
231 -class StablelmForCausalLM(nn.Module):
247 +class StablelmForCausalLM(nn.Module, SupportsPP):
248
249 def __init__(
250 self,
@@ -247,6 +263,8 @@ class StablelmForCausalLM(nn.Module):
263 self.lm_head.weight = self.model.embed_tokens.weight
264 self.logits_processor = LogitsProcessor(config.vocab_size)
265 self.sampler = Sampler()
266 + self.make_empty_intermediate_tensors = (
267 + self.model.make_empty_intermediate_tensors)
268
269 def forward(
270 self,
@@ -255,9 +273,9 @@ class StablelmForCausalLM(nn.Module):
273 kv_caches: List[torch.Tensor],
274 attn_metadata: AttentionMetadata,
275 intermediate_tensors: Optional[IntermediateTensors] = None,
258 - ) -> torch.Tensor:
276 + ) -> Union[torch.Tensor, IntermediateTensors]:
277 hidden_states = self.model(input_ids, positions, kv_caches,
260 - attn_metadata)
278 + attn_metadata, intermediate_tensors)
279 return hidden_states
280
281 def compute_logits(
@@ -302,6 +320,8 @@ class StablelmForCausalLM(nn.Module):
320 # Skip loading extra bias for GPTQ models.
321 if name.endswith(".bias") and name not in params_dict:
322 continue
323 + if is_pp_missing_parameter(name, self):
324 + continue
325 param = params_dict[name]
326 weight_loader = param.weight_loader
327 weight_loader(param, loaded_weight, shard_id)
@@ -310,6 +330,8 @@ class StablelmForCausalLM(nn.Module):
330 # Skip loading extra bias for GPTQ models.
331 if name.endswith(".bias") and name not in params_dict:
332 continue
333 + if is_pp_missing_parameter(name, self):
334 + continue
335 param = params_dict[name]
336 weight_loader = getattr(param, "weight_loader",
337 default_weight_loader)
vllm/model_executor/models/starcoder2.py
+39 -18
@@ -18,7 +18,7 @@
18 # See the License for the specific language governing permissions and
19 # limitations under the License.
20 """ PyTorch Starcoder2 model."""
21 -from typing import Iterable, List, Optional, Tuple
21 +from typing import Iterable, List, Optional, Tuple, Union
22
23 import torch
24 from torch import nn
@@ -26,14 +26,13 @@ from transformers import Starcoder2Config
26
27 from vllm.attention import Attention, AttentionMetadata
28 from vllm.config import CacheConfig
29 -from vllm.distributed import get_tensor_model_parallel_world_size
29 +from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
30 from vllm.model_executor.layers.activation import get_act_fn
31 from vllm.model_executor.layers.linear import (ColumnParallelLinear,
32 QKVParallelLinear,
33 RowParallelLinear)
34 from vllm.model_executor.layers.logits_processor import LogitsProcessor
35 -from vllm.model_executor.layers.quantization.base_config import (
36 - QuantizationConfig)
35 +from vllm.model_executor.layers.quantization import QuantizationConfig
36 from vllm.model_executor.layers.rotary_embedding import get_rope
37 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
38 from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -42,6 +41,10 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
41 from vllm.model_executor.sampling_metadata import SamplingMetadata
42 from vllm.sequence import IntermediateTensors
43
44 +from .interfaces import SupportsPP
45 +from .utils import (is_pp_missing_parameter,
46 + make_empty_intermediate_tensors_factory, make_layers)
47 +
48
49 class Starcoder2Attention(nn.Module):
50
@@ -195,7 +198,8 @@ class Starcoder2Model(nn.Module):
198 def __init__(self,
199 config: Starcoder2Config,
200 cache_config: Optional[CacheConfig] = None,
198 - quant_config: Optional[QuantizationConfig] = None):
201 + quant_config: Optional[QuantizationConfig] = None,
202 + prefix: str = ""):
203 super().__init__()
204 self.config = config
205 self.padding_idx = config.pad_token_id
@@ -204,13 +208,16 @@ class Starcoder2Model(nn.Module):
208 # TODO: consider padding_idx (currently removed)
209 self.embed_tokens = VocabParallelEmbedding(config.vocab_size,
210 config.hidden_size)
207 - self.layers = nn.ModuleList([
208 - Starcoder2DecoderLayer(config,
209 - cache_config,
210 - quant_config=quant_config)
211 - for _ in range(config.num_hidden_layers)
212 - ])
211 + self.start_layer, self.end_layer, self.layers = make_layers(
212 + config.num_hidden_layers,
213 + lambda prefix: Starcoder2DecoderLayer(
214 + config, cache_config, quant_config=quant_config),
215 + prefix=f"{prefix}.layers",
216 + )
217 self.norm = nn.LayerNorm(config.hidden_size, eps=config.norm_epsilon)
218 + self.make_empty_intermediate_tensors = (
219 + make_empty_intermediate_tensors_factory(["hidden_states"],
220 + config.hidden_size))
221
222 def forward(
223 self,
@@ -218,17 +225,25 @@ class Starcoder2Model(nn.Module):
225 positions: torch.Tensor,
226 kv_caches: List[torch.Tensor],
227 attn_metadata: AttentionMetadata,
221 - ) -> torch.Tensor:
222 - hidden_states = self.embed_tokens(input_ids)
223 - for i in range(len(self.layers)):
228 + intermediate_tensors: Optional[IntermediateTensors],
229 + ) -> Union[torch.Tensor, IntermediateTensors]:
230 + if get_pp_group().is_first_rank:
231 + hidden_states = self.embed_tokens(input_ids)
232 + else:
233 + assert intermediate_tensors is not None
234 + hidden_states = intermediate_tensors["hidden_states"]
235 + for i in range(self.start_layer, self.end_layer):
236 layer = self.layers[i]
225 - hidden_states = layer(positions, hidden_states, kv_caches[i],
237 + hidden_states = layer(positions, hidden_states,
238 + kv_caches[i - self.start_layer],
239 attn_metadata)
240 + if not get_pp_group().is_last_rank:
241 + return IntermediateTensors({"hidden_states": hidden_states})
242 hidden_states = self.norm(hidden_states)
243 return hidden_states
244
245
231 -class Starcoder2ForCausalLM(nn.Module):
246 +class Starcoder2ForCausalLM(nn.Module, SupportsPP):
247
248 def __init__(self,
249 config: Starcoder2Config,
@@ -255,6 +270,8 @@ class Starcoder2ForCausalLM(nn.Module):
270 self.logits_processor = LogitsProcessor(self.unpadded_vocab_size,
271 config.vocab_size)
272 self.sampler = Sampler()
273 + self.make_empty_intermediate_tensors = (
274 + self.model.make_empty_intermediate_tensors)
275
276 def forward(
277 self,
@@ -263,9 +280,9 @@ class Starcoder2ForCausalLM(nn.Module):
280 kv_caches: List[torch.Tensor],
281 attn_metadata: AttentionMetadata,
282 intermediate_tensors: Optional[IntermediateTensors] = None,
266 - ) -> torch.Tensor:
283 + ) -> Union[torch.Tensor, IntermediateTensors]:
284 hidden_states = self.model(input_ids, positions, kv_caches,
268 - attn_metadata)
285 + attn_metadata, intermediate_tensors)
286 return hidden_states
287
288 def compute_logits(
@@ -302,6 +319,8 @@ class Starcoder2ForCausalLM(nn.Module):
319 if weight_name not in name:
320 continue
321 name = name.replace(weight_name, param_name)
322 + if is_pp_missing_parameter(name, self):
323 + continue
324 param = params_dict[name]
325 weight_loader = param.weight_loader
326 weight_loader(param, loaded_weight, shard_id)
@@ -309,6 +328,8 @@ class Starcoder2ForCausalLM(nn.Module):
328 else:
329 if self.config.tie_word_embeddings and "lm_head.weight" in name:
330 continue
331 + if is_pp_missing_parameter(name, self):
332 + continue
333 param = params_dict[name]
334 weight_loader = getattr(param, "weight_loader",
335 default_weight_loader)
vllm/model_executor/models/ultravox.py
+33 -18
@@ -3,7 +3,7 @@
3
4 import math
5 from array import array
6 -from functools import lru_cache
6 +from functools import cached_property, lru_cache
7 from typing import (Iterable, List, Literal, Mapping, Optional, Tuple,
8 TypedDict, Union, cast)
9
@@ -22,12 +22,10 @@ from vllm.inputs.data import LLMInputs
22 from vllm.inputs.registry import InputContext
23 from vllm.model_executor.layers.activation import SiluAndMul, get_act_fn
24 from vllm.model_executor.layers.layernorm import RMSNorm
25 -from vllm.model_executor.layers.quantization.base_config import (
26 - QuantizationConfig)
27 -from vllm.model_executor.layers.sampler import SamplerOutput
25 +from vllm.model_executor.layers.quantization import QuantizationConfig
26 +from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
27 from vllm.model_executor.model_loader.loader import DefaultModelLoader
28 from vllm.model_executor.model_loader.weight_utils import default_weight_loader
30 -from vllm.model_executor.models.interfaces import SupportsMultiModal
29 from vllm.model_executor.models.utils import (flatten_bn,
30 group_weights_with_prefix,
31 init_vllm_registered_model,
@@ -37,9 +35,12 @@ from vllm.multimodal import MULTIMODAL_REGISTRY
35 from vllm.multimodal.base import MultiModalInputs, NestedTensors
36 from vllm.multimodal.utils import (cached_get_tokenizer,
37 repeat_and_pad_placeholder_tokens)
40 -from vllm.sequence import VLLM_TOKEN_ID_ARRAY_TYPE, SequenceData
38 +from vllm.sequence import (VLLM_TOKEN_ID_ARRAY_TYPE, IntermediateTensors,
39 + SequenceData)
40 from vllm.transformers_utils.configs.ultravox import UltravoxConfig
41
42 +from .interfaces import SupportsMultiModal, SupportsPP
43 +
44 _AUDIO_PLACEHOLDER_TOKEN = 128002
45 _AUDIO_TOKENS_PER_SECOND = 6.25
46
@@ -323,7 +324,7 @@ class ModifiedWhisperEncoder(WhisperEncoder):
324 "audio", get_ultravox_max_audio_tokens)
325 @INPUT_REGISTRY.register_dummy_data(dummy_data_for_ultravox)
326 @INPUT_REGISTRY.register_input_processor(input_processor_for_ultravox)
326 -class UltravoxModel(nn.Module, SupportsMultiModal):
327 +class UltravoxModel(nn.Module, SupportsMultiModal, SupportsPP):
328
329 def __init__(self,
330 config: UltravoxConfig,
@@ -353,6 +354,16 @@ class UltravoxModel(nn.Module, SupportsMultiModal):
354 revision=None,
355 prefix="language_model."))
356
357 + self.make_empty_intermediate_tensors = (
358 + self.language_model.make_empty_intermediate_tensors)
359 +
360 + @cached_property
361 + def sampler(self):
362 + if hasattr(self.language_model, "sampler"):
363 + return self.language_model.sampler
364 +
365 + return Sampler()
366 +
367 def _audio_features_to_embeddings(
368 self, input_features: torch.Tensor) -> torch.Tensor:
369 audio_input = input_features.to(self.audio_tower.dtype)
@@ -425,7 +436,7 @@ class UltravoxModel(nn.Module, SupportsMultiModal):
436 kv_caches: List[torch.Tensor],
437 attn_metadata: AttentionMetadata,
438 intermediate_tensors: Optional[torch.Tensor],
428 - **kwargs) -> SamplerOutput:
439 + **kwargs) -> Union[torch.Tensor, IntermediateTensors]:
440 """Run forward pass for Ultravox
441
442 One key thing to understand is the `input_ids` already accounts for the
@@ -438,18 +449,22 @@ class UltravoxModel(nn.Module, SupportsMultiModal):
449 Args:
450 audio_features: A batch of audio inputs [B, N, 80, M].
451 """
441 - audio_input = self._parse_and_validate_audio_input(**kwargs)
442 - if audio_input is not None:
443 - audio_embeddings = self._process_audio_input(audio_input)
444 - inputs_embeds = self.language_model.model.get_input_embeddings(
445 - input_ids)
446 -
447 - inputs_embeds = merge_multimodal_embeddings(
448 - input_ids, inputs_embeds, audio_embeddings,
449 - _AUDIO_PLACEHOLDER_TOKEN)
452 + if intermediate_tensors is not None:
453 input_ids = None
451 - else:
454 inputs_embeds = None
455 + else:
456 + audio_input = self._parse_and_validate_audio_input(**kwargs)
457 + if audio_input is not None:
458 + audio_embeddings = self._process_audio_input(audio_input)
459 + inputs_embeds = self.language_model.model.get_input_embeddings(
460 + input_ids)
461 +
462 + inputs_embeds = merge_multimodal_embeddings(
463 + input_ids, inputs_embeds, audio_embeddings,
464 + _AUDIO_PLACEHOLDER_TOKEN)
465 + input_ids = None
466 + else:
467 + inputs_embeds = None
468
469 hidden_states = self.language_model.model(
470 input_ids=input_ids,
vllm/model_executor/models/utils.py
+17 -13
@@ -24,7 +24,7 @@ class WeightsGroup(UserDict):
24 when attempting to access a weight component that does not exist.
25 """
26
27 - def __getitem__(self, key: str) -> int:
27 + def __getitem__(self, key: str) -> Iterable[Tuple[str, torch.Tensor]]:
28 try:
29 return super().__getitem__(key)
30 except KeyError as exc:
@@ -49,8 +49,7 @@ def filter_weights(weights: Iterable[Tuple[str, torch.Tensor]],
49
50
51 def group_weights_with_prefix(
52 - weights: Iterable[Tuple[str, torch.Tensor]]
53 -) -> Dict[str, Iterable[Tuple[str, torch.Tensor]]]:
52 + weights: Iterable[Tuple[str, torch.Tensor]], ) -> WeightsGroup:
53 """
54 Helper function to group weights with prefix
55 """
@@ -183,10 +182,7 @@ def merge_multimodal_embeddings(input_ids: torch.Tensor,
182
183 class LayerFn(Protocol):
184
186 - def __call__(
187 - self,
188 - prefix="",
189 - ) -> torch.nn.Module:
185 + def __call__(self, prefix: str) -> torch.nn.Module:
186 ...
187
188
@@ -319,8 +315,10 @@ def is_pp_missing_parameter(name: str, model: torch.nn.Module) -> bool:
315 def make_empty_intermediate_tensors_factory(keys: List[str], hidden_size: int):
316
317 def make_empty_intermediate_tensors(
322 - batch_size: int, dtype: torch.dtype,
323 - device: torch.device) -> IntermediateTensors:
318 + batch_size: int,
319 + dtype: torch.dtype,
320 + device: torch.device,
321 + ) -> IntermediateTensors:
322 return IntermediateTensors({
323 key: torch.zeros((batch_size, hidden_size),
324 dtype=dtype,
@@ -342,8 +340,14 @@ class LLMWrapper(nn.Module):
340 self.model_name = name
341 setattr(self, name, llm)
342
345 - def forward(self, *args, **kwargs) -> Any:
346 - return getattr(self, self.model_name)(*args, **kwargs)
343 + def __getattr__(self, key: str):
344 + llm = super().__getattr__(self.model_name)
345 + if key == self.model_name:
346 + return llm
347
348 - def embed_tokens(self, *args, **kwargs) -> Any:
349 - return getattr(self, self.model_name).embed_tokens(*args, **kwargs)
348 + return getattr(llm, key)
349 +
350 + # We need to explicitly override this
351 + def __call__(self, *args: Any, **kwargs: Any) -> Any:
352 + llm = super().__getattr__(self.model_name)
353 + return llm(*args, **kwargs)
vllm/model_executor/models/xverse.py
+39 -17
@@ -20,7 +20,7 @@
20 # See the License for the specific language governing permissions and
21 # limitations under the License.
22 """Inference-only Xverse model compatible with HuggingFace weights."""
23 -from typing import Any, Dict, Iterable, List, Optional, Tuple
23 +from typing import Any, Dict, Iterable, List, Optional, Tuple, Union
24
25 import torch
26 from torch import nn
@@ -28,15 +28,14 @@ from transformers import PretrainedConfig
28
29 from vllm.attention import Attention, AttentionMetadata
30 from vllm.config import CacheConfig, LoRAConfig
31 -from vllm.distributed import get_tensor_model_parallel_world_size
31 +from vllm.distributed import get_pp_group, get_tensor_model_parallel_world_size
32 from vllm.model_executor.layers.activation import SiluAndMul
33 from vllm.model_executor.layers.layernorm import RMSNorm
34 from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
35 QKVParallelLinear,
36 RowParallelLinear)
37 from vllm.model_executor.layers.logits_processor import LogitsProcessor
38 -from vllm.model_executor.layers.quantization.base_config import (
39 - QuantizationConfig)
38 +from vllm.model_executor.layers.quantization import QuantizationConfig
39 from vllm.model_executor.layers.rotary_embedding import get_rope
40 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
41 from vllm.model_executor.layers.vocab_parallel_embedding import (
@@ -45,7 +44,9 @@ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
44 from vllm.model_executor.sampling_metadata import SamplingMetadata
45 from vllm.sequence import IntermediateTensors
46
48 -from .interfaces import SupportsLoRA
47 +from .interfaces import SupportsLoRA, SupportsPP
48 +from .utils import (is_pp_missing_parameter,
49 + make_empty_intermediate_tensors_factory, make_layers)
50
51
52 class XverseMLP(nn.Module):
@@ -227,6 +228,7 @@ class XverseModel(nn.Module):
228 cache_config: Optional[CacheConfig] = None,
229 quant_config: Optional[QuantizationConfig] = None,
230 lora_config: Optional[LoRAConfig] = None,
231 + prefix: str = "",
232 ) -> None:
233 super().__init__()
234 self.config = config
@@ -240,11 +242,16 @@ class XverseModel(nn.Module):
242 config.hidden_size,
243 org_num_embeddings=config.vocab_size,
244 )
243 - self.layers = nn.ModuleList([
244 - XverseDecoderLayer(config, cache_config, quant_config)
245 - for _ in range(config.num_hidden_layers)
246 - ])
245 + self.start_layer, self.end_layer, self.layers = make_layers(
246 + config.num_hidden_layers,
247 + lambda prefix: XverseDecoderLayer(config, cache_config,
248 + quant_config),
249 + prefix=f"{prefix}.layers",
250 + )
251 self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
252 + self.make_empty_intermediate_tensors = (
253 + make_empty_intermediate_tensors_factory(
254 + ["hidden_states", "residual"], config.hidden_size))
255
256 def forward(
257 self,
@@ -252,23 +259,32 @@ class XverseModel(nn.Module):
259 positions: torch.Tensor,
260 kv_caches: List[torch.Tensor],
261 attn_metadata: AttentionMetadata,
255 - ) -> torch.Tensor:
256 - hidden_states = self.embed_tokens(input_ids)
257 - residual = None
258 - for i in range(len(self.layers)):
262 + intermediate_tensors: Optional[IntermediateTensors],
263 + ) -> Union[torch.Tensor, IntermediateTensors]:
264 + if get_pp_group().is_first_rank:
265 + hidden_states = self.embed_tokens(input_ids)
266 + residual = None
267 + else:
268 + hidden_states = intermediate_tensors["hidden_states"]
269 + for i in range(self.start_layer, self.end_layer):
270 layer = self.layers[i]
271 hidden_states, residual = layer(
272 positions,
273 hidden_states,
263 - kv_caches[i],
274 + kv_caches[i - self.start_layer],
275 attn_metadata,
276 residual,
277 )
278 + if not get_pp_group().is_last_rank:
279 + return IntermediateTensors({
280 + "hidden_states": hidden_states,
281 + "residual": residual
282 + })
283 hidden_states, _ = self.norm(hidden_states, residual)
284 return hidden_states
285
286
271 -class XverseForCausalLM(nn.Module, SupportsLoRA):
287 +class XverseForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
288 packed_modules_mapping = {
289 "qkv_proj": [
290 "q_proj",
@@ -317,6 +333,8 @@ class XverseForCausalLM(nn.Module, SupportsLoRA):
333 self.lm_head.weight = self.model.embed_tokens.weight
334 self.logits_processor = LogitsProcessor(config.vocab_size)
335 self.sampler = Sampler()
336 + self.make_empty_intermediate_tensors = (
337 + self.model.make_empty_intermediate_tensors)
338
339 def forward(
340 self,
@@ -325,9 +343,9 @@ class XverseForCausalLM(nn.Module, SupportsLoRA):
343 kv_caches: List[torch.Tensor],
344 attn_metadata: AttentionMetadata,
345 intermediate_tensors: Optional[IntermediateTensors] = None,
328 - ) -> torch.Tensor:
346 + ) -> Union[torch.Tensor, IntermediateTensors]:
347 hidden_states = self.model(input_ids, positions, kv_caches,
330 - attn_metadata)
348 + attn_metadata, intermediate_tensors)
349 return hidden_states
350
351 def compute_logits(
@@ -368,6 +386,8 @@ class XverseForCausalLM(nn.Module, SupportsLoRA):
386 # Skip loading extra bias for GPTQ models.
387 if name.endswith(".bias") and name not in params_dict:
388 continue
389 + if is_pp_missing_parameter(name, self):
390 + continue
391 param = params_dict[name]
392 weight_loader = param.weight_loader
393 weight_loader(param, loaded_weight, shard_id)
@@ -376,6 +396,8 @@ class XverseForCausalLM(nn.Module, SupportsLoRA):
396 # Skip loading extra bias for GPTQ models.
397 if name.endswith(".bias") and name not in params_dict:
398 continue
399 + if is_pp_missing_parameter(name, self):
400 + continue
401 param = params_dict[name]
402 weight_loader = getattr(param, "weight_loader",
403 default_weight_loader)