A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 10:45 UTC
[Model] VLM2Vec, the first multimodal embedding model in vLLM (#9303)
Cyrus Leung committed
Oct 16, 2024 at 14:31 UTC
7abba39ee64c1e2c84f48d7c38b2cd1c24bb0ebb
16 files changed
+465
-261
docs/source/models/supported_models.rst
+53
-26
@@ -3,7 +3,7 @@
3
Supported Models
4
================
5
6
-vLLM supports a variety of generative Transformer models in `HuggingFace Transformers <https://huggingface.co/models>`_.
6
+vLLM supports a variety of generative Transformer models in `HuggingFace (HF) Transformers <https://huggingface.co/models>`_.
7
The following is the list of model architectures that are currently supported by vLLM.
8
Alongside each architecture, we include some popular models that use it.
9
@@ -19,7 +19,7 @@ Text Generation
19
20
* - Architecture
21
- Models
22
- - Example HuggingFace Models
22
+ - Example HF Models
23
- :ref:`LoRA <lora>`
24
- :ref:`PP <distributed_serving>`
25
* - :code:`AquilaForCausalLM`
@@ -280,7 +280,7 @@ Text Embedding
280
281
* - Architecture
282
- Models
283
- - Example HuggingFace Models
283
+ - Example HF Models
284
- :ref:`LoRA <lora>`
285
- :ref:`PP <distributed_serving>`
286
* - :code:`Gemma2Model`
@@ -303,7 +303,7 @@ Reward Modeling
303
304
* - Architecture
305
- Models
306
- - Example HuggingFace Models
306
+ - Example HF Models
307
- :ref:`LoRA <lora>`
308
- :ref:`PP <distributed_serving>`
309
* - :code:`Qwen2ForRewardModel`
@@ -316,7 +316,14 @@ Reward Modeling
316
As an interim measure, these models are supported via Embeddings API. See `this RFC <https://github.com/vllm-project/vllm/issues/8967>`_ for upcoming changes.
317
318
Multimodal Language Models
319
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
319
+^^^^^^^^^^^^^^^^^^^^^^^^^^
320
+
321
+The following modalities are supported depending on the model:
322
+
323
+- **T**\ ext
324
+- **I**\ mage
325
+- **V**\ ideo
326
+- **A**\ udio
327
328
.. _supported_vlms:
329
@@ -324,78 +331,78 @@ Text Generation
331
---------------
332
333
.. list-table::
327
- :widths: 25 25 25 25 5 5
334
+ :widths: 25 25 15 25 5 5
335
:header-rows: 1
336
337
* - Architecture
338
- Models
332
- - Modalities
333
- - Example HuggingFace Models
339
+ - Inputs
340
+ - Example HF Models
341
- :ref:`LoRA <lora>`
342
- :ref:`PP <distributed_serving>`
343
* - :code:`Blip2ForConditionalGeneration`
344
- BLIP-2
338
- - Image\ :sup:`E`
345
+ - T + I\ :sup:`E`
346
- :code:`Salesforce/blip2-opt-2.7b`, :code:`Salesforce/blip2-opt-6.7b`, etc.
347
-
348
- ✅︎
349
* - :code:`ChameleonForConditionalGeneration`
350
- Chameleon
344
- - Image
351
+ - T + I
352
- :code:`facebook/chameleon-7b` etc.
353
-
354
- ✅︎
355
* - :code:`FuyuForCausalLM`
356
- Fuyu
350
- - Image
357
+ - T + I
358
- :code:`adept/fuyu-8b` etc.
359
-
360
- ✅︎
361
* - :code:`ChatGLMModel`
362
- GLM-4V
356
- - Image
363
+ - T + I
364
- :code:`THUDM/glm-4v-9b` etc.
365
-
366
- ✅︎
367
* - :code:`InternVLChatModel`
368
- InternVL2
362
- - Image\ :sup:`E+`
369
+ - T + I\ :sup:`E+`
370
- :code:`OpenGVLab/InternVL2-4B`, :code:`OpenGVLab/InternVL2-8B`, etc.
371
-
372
- ✅︎
373
* - :code:`LlavaForConditionalGeneration`
374
- LLaVA-1.5
368
- - Image\ :sup:`E+`
375
+ - T + I\ :sup:`E+`
376
- :code:`llava-hf/llava-1.5-7b-hf`, :code:`llava-hf/llava-1.5-13b-hf`, etc.
377
-
378
- ✅︎
379
* - :code:`LlavaNextForConditionalGeneration`
380
- LLaVA-NeXT
374
- - Image\ :sup:`E+`
381
+ - T + I\ :sup:`E+`
382
- :code:`llava-hf/llava-v1.6-mistral-7b-hf`, :code:`llava-hf/llava-v1.6-vicuna-7b-hf`, etc.
383
-
384
- ✅︎
385
* - :code:`LlavaNextVideoForConditionalGeneration`
386
- LLaVA-NeXT-Video
380
- - Video
387
+ - T + V
388
- :code:`llava-hf/LLaVA-NeXT-Video-7B-hf`, etc.
389
-
390
- ✅︎
391
* - :code:`LlavaOnevisionForConditionalGeneration`
392
- LLaVA-Onevision
386
- - Image\ :sup:`+` / Video
393
+ - T + I\ :sup:`+` + V
394
- :code:`llava-hf/llava-onevision-qwen2-7b-ov-hf`, :code:`llava-hf/llava-onevision-qwen2-0.5b-ov-hf`, etc.
395
-
396
- ✅︎
397
* - :code:`MiniCPMV`
398
- MiniCPM-V
392
- - Image\ :sup:`E+`
399
+ - T + I\ :sup:`E+`
400
- :code:`openbmb/MiniCPM-V-2` (see note), :code:`openbmb/MiniCPM-Llama3-V-2_5`, :code:`openbmb/MiniCPM-V-2_6`, etc.
401
- ✅︎
402
- ✅︎
403
* - :code:`MllamaForConditionalGeneration`
404
- Llama 3.2
398
- - Image
405
+ - T + I
406
- :code:`meta-llama/Llama-3.2-90B-Vision-Instruct`, :code:`meta-llama/Llama-3.2-11B-Vision`, etc.
407
-
408
-
@@ -407,43 +414,43 @@ Text Generation
414
- ✅︎
415
* - :code:`NVLM_D_Model`
416
- NVLM-D 1.0
410
- - Image\ :sup:`E+`
417
+ - T + I\ :sup:`E+`
418
- :code:`nvidia/NVLM-D-72B`, etc.
419
-
420
- ✅︎
421
* - :code:`PaliGemmaForConditionalGeneration`
422
- PaliGemma
416
- - Image\ :sup:`E`
423
+ - T + I\ :sup:`E`
424
- :code:`google/paligemma-3b-pt-224`, :code:`google/paligemma-3b-mix-224`, etc.
425
-
426
- ✅︎
427
* - :code:`Phi3VForCausalLM`
428
- Phi-3-Vision, Phi-3.5-Vision
422
- - Image\ :sup:`E+`
429
+ - T + I\ :sup:`E+`
430
- :code:`microsoft/Phi-3-vision-128k-instruct`, :code:`microsoft/Phi-3.5-vision-instruct` etc.
431
-
432
- ✅︎
433
* - :code:`PixtralForConditionalGeneration`
434
- Pixtral
428
- - Image\ :sup:`+`
435
+ - T + I\ :sup:`+`
436
- :code:`mistralai/Pixtral-12B-2409`
437
-
438
- ✅︎
439
* - :code:`QWenLMHeadModel`
440
- Qwen-VL
434
- - Image\ :sup:`E+`
441
+ - T + I\ :sup:`E+`
442
- :code:`Qwen/Qwen-VL`, :code:`Qwen/Qwen-VL-Chat`, etc.
443
-
444
- ✅︎
445
* - :code:`Qwen2VLForConditionalGeneration`
446
- Qwen2-VL
440
- - Image\ :sup:`E+` / Video\ :sup:`+`
447
+ - T + I\ :sup:`E+` + V\ :sup:`+`
448
- :code:`Qwen/Qwen2-VL-2B-Instruct`, :code:`Qwen/Qwen2-VL-7B-Instruct`, :code:`Qwen/Qwen2-VL-72B-Instruct`, etc.
449
-
450
- ✅︎
451
* - :code:`UltravoxModel`
452
- Ultravox
446
- - Audio\ :sup:`E+`
453
+ - T + A\ :sup:`E+`
454
- :code:`fixie-ai/ultravox-v0_3`
455
-
456
- ✅︎
@@ -455,6 +462,26 @@ Text Generation
462
For :code:`openbmb/MiniCPM-V-2`, the official repo doesn't work yet, so we need to use a fork (:code:`HwwwH/MiniCPM-V-2`) for now.
463
For more details, please see: https://github.com/vllm-project/vllm/pull/4087#issuecomment-2250397630
464
465
+Multimodal Embedding
466
+--------------------
467
+
468
+.. list-table::
469
+ :widths: 25 25 15 25 5 5
470
+ :header-rows: 1
471
+
472
+ * - Architecture
473
+ - Models
474
+ - Inputs
475
+ - Example HF Models
476
+ - :ref:`LoRA <lora>`
477
+ - :ref:`PP <distributed_serving>`
478
+ * - :code:`Phi3VForCausalLM`
479
+ - Phi-3-Vision-based
480
+ - T + I
481
+ - :code:`TIGER-Lab/VLM2Vec-Full`
482
+ - 🚧
483
+ - ✅︎
484
+
485
----
486
487
If your model uses one of the above model architectures, you can seamlessly run your model with vLLM.
examples/offline_inference_vision_language_embedding.py
new
+21
@@ -0,0 +1,21 @@
1
+from vllm import LLM
2
+from vllm.assets.image import ImageAsset
3
+
4
+image = ImageAsset("cherry_blossom").pil_image.convert("RGB")
5
+prompt = "<|image_1|> Represent the given image with the following question: What is in the image" # noqa: E501
6
+
7
+# Create an LLM.
8
+llm = LLM(
9
+ model="TIGER-Lab/VLM2Vec-Full",
10
+ trust_remote_code=True,
11
+ max_model_len=4096,
12
+ max_num_seqs=2,
13
+ mm_processor_kwargs={"num_crops": 16},
14
+)
15
+
16
+# Generate embedding. The output is a list of EmbeddingRequestOutputs.
17
+outputs = llm.encode({"prompt": prompt, "multi_modal_data": {"image": image}})
18
+
19
+# Print the outputs.
20
+for output in outputs:
21
+ print(output.outputs.embedding) # list of 3072 floats
tests/conftest.py
+98
-61
@@ -262,7 +262,7 @@ class HfRunner:
262
dtype: str = "half",
263
*,
264
model_kwargs: Optional[Dict[str, Any]] = None,
265
- is_embedding_model: bool = False,
265
+ is_sentence_transformer: bool = False,
266
auto_cls: Type[_BaseAutoModelClass] = AutoModelForCausalLM,
267
postprocess_inputs: Callable[[BatchEncoding],
268
BatchEncoding] = identity,
@@ -271,7 +271,7 @@ class HfRunner:
271
272
self.model_name = model_name
273
274
- if is_embedding_model:
274
+ if is_sentence_transformer:
275
# Lazy init required for AMD CI
276
from sentence_transformers import SentenceTransformer
277
self.model = self.wrap_device(
@@ -307,17 +307,23 @@ class HfRunner:
307
308
self.postprocess_inputs = postprocess_inputs
309
310
- def generate(
310
+ def get_inputs(
311
self,
312
prompts: List[str],
313
images: Optional[PromptImageInput] = None,
314
- videos: Optional[List[np.ndarray]] = None,
315
- **kwargs: Any,
316
- ) -> List[Tuple[List[List[int]], List[str]]]:
317
- if images:
314
+ videos: Optional[PromptVideoInput] = None,
315
+ audios: Optional[PromptAudioInput] = None,
316
+ ) -> List[BatchEncoding]:
317
+ if images is not None:
318
assert len(prompts) == len(images)
319
320
- outputs: List[Tuple[List[List[int]], List[str]]] = []
320
+ if videos is not None:
321
+ assert len(prompts) == len(videos)
322
+
323
+ if audios is not None:
324
+ assert len(prompts) == len(audios)
325
+
326
+ all_inputs: List[BatchEncoding] = []
327
for i, prompt in enumerate(prompts):
328
processor_kwargs: Dict[str, Any] = {
329
"text": prompt,
@@ -327,10 +333,33 @@ class HfRunner:
333
processor_kwargs["images"] = images[i]
334
if videos is not None and videos[i] is not None:
335
processor_kwargs["videos"] = videos[i]
336
+ if audios is not None and audios[i] is not None:
337
+ audio, sr = audios[i]
338
+ processor_kwargs["audio"] = audio
339
+ processor_kwargs["sampling_rate"] = sr
340
341
inputs = self.processor(**processor_kwargs)
342
inputs = self.postprocess_inputs(inputs)
343
344
+ all_inputs.append(inputs)
345
+
346
+ return all_inputs
347
+
348
+ def generate(
349
+ self,
350
+ prompts: List[str],
351
+ images: Optional[PromptImageInput] = None,
352
+ videos: Optional[List[np.ndarray]] = None,
353
+ audios: Optional[PromptAudioInput] = None,
354
+ **kwargs: Any,
355
+ ) -> List[Tuple[List[List[int]], List[str]]]:
356
+ all_inputs = self.get_inputs(prompts,
357
+ images=images,
358
+ videos=videos,
359
+ audios=audios)
360
+
361
+ outputs: List[Tuple[List[List[int]], List[str]]] = []
362
+ for inputs in all_inputs:
363
output_ids = self.model.generate(
364
**self.wrap_device(inputs, device=self.model.device.type),
365
use_cache=True,
@@ -350,12 +379,16 @@ class HfRunner:
379
prompts: List[str],
380
max_tokens: int,
381
images: Optional[PromptImageInput] = None,
382
+ videos: Optional[List[np.ndarray]] = None,
383
+ audios: Optional[PromptAudioInput] = None,
384
**kwargs: Any,
385
) -> List[Tuple[List[int], str]]:
386
outputs = self.generate(prompts,
387
do_sample=False,
388
max_new_tokens=max_tokens,
389
images=images,
390
+ videos=videos,
391
+ audios=audios,
392
**kwargs)
393
394
return [(output_ids[0], output_str[0])
@@ -388,22 +421,16 @@ class HfRunner:
421
max_tokens: int,
422
images: Optional[PromptImageInput] = None,
423
videos: Optional[List[np.ndarray]] = None,
424
+ audios: Optional[PromptAudioInput] = None,
425
**kwargs: Any,
426
) -> List[List[torch.Tensor]]:
393
- all_logprobs: List[List[torch.Tensor]] = []
394
- for i, prompt in enumerate(prompts):
395
- processor_kwargs: Dict[str, Any] = {
396
- "text": prompt,
397
- "return_tensors": "pt",
398
- }
399
- if images is not None and images[i] is not None:
400
- processor_kwargs["images"] = images[i]
401
- if videos is not None and videos[i] is not None:
402
- processor_kwargs["videos"] = videos[i]
403
-
404
- inputs = self.processor(**processor_kwargs)
405
- inputs = self.postprocess_inputs(inputs)
427
+ all_inputs = self.get_inputs(prompts,
428
+ images=images,
429
+ videos=videos,
430
+ audios=audios)
431
432
+ all_logprobs: List[List[torch.Tensor]] = []
433
+ for inputs in all_inputs:
434
output = self.model.generate(
435
**self.wrap_device(inputs, device=self.model.device.type),
436
use_cache=True,
@@ -475,28 +502,16 @@ class HfRunner:
502
videos: Optional[List[np.ndarray]] = None,
503
**kwargs: Any,
504
) -> List[TokensTextLogprobs]:
505
+ all_inputs = self.get_inputs(prompts,
506
+ images=images,
507
+ videos=videos,
508
+ audios=audios)
509
+
510
all_logprobs: List[List[Dict[int, float]]] = []
511
all_output_ids: List[List[int]] = []
512
all_output_strs: List[str] = []
513
482
- for i, prompt in enumerate(prompts):
483
- processor_kwargs: Dict[str, Any] = {
484
- "text": prompt,
485
- "return_tensors": "pt",
486
- }
487
- if images is not None and images[i] is not None:
488
- processor_kwargs["images"] = images[i]
489
-
490
- if audios is not None:
491
- audio, sr = audios[i]
492
- processor_kwargs["audio"] = audio
493
- processor_kwargs["sampling_rate"] = sr
494
-
495
- if videos is not None:
496
- processor_kwargs["videos"] = videos[i]
497
- inputs = self.processor(**processor_kwargs)
498
- inputs = self.postprocess_inputs(inputs)
499
-
514
+ for inputs in all_inputs:
515
output = self.model.generate(
516
**self.wrap_device(inputs, device=self.model.device.type),
517
use_cache=True,
@@ -632,20 +647,50 @@ class VllmRunner:
647
**kwargs,
648
)
649
635
- def generate(
650
+ def get_inputs(
651
self,
652
prompts: List[str],
638
- sampling_params: SamplingParams,
653
images: Optional[PromptImageInput] = None,
640
- ) -> List[Tuple[List[List[int]], List[str]]]:
654
+ videos: Optional[PromptVideoInput] = None,
655
+ audios: Optional[PromptAudioInput] = None,
656
+ ) -> List[TextPrompt]:
657
if images is not None:
658
assert len(prompts) == len(images)
659
660
+ if videos is not None:
661
+ assert len(prompts) == len(videos)
662
+
663
+ if audios is not None:
664
+ assert len(prompts) == len(audios)
665
+
666
inputs = [TextPrompt(prompt=prompt) for prompt in prompts]
667
if images is not None:
668
for i, image in enumerate(images):
669
inputs[i]["multi_modal_data"] = {"image": image}
670
671
+ if videos is not None:
672
+ for i, video in enumerate(videos):
673
+ inputs[i]["multi_modal_data"] = {"video": video}
674
+
675
+ if audios is not None:
676
+ for i, audio in enumerate(audios):
677
+ inputs[i]["multi_modal_data"] = {"audio": audio}
678
+
679
+ return inputs
680
+
681
+ def generate(
682
+ self,
683
+ prompts: List[str],
684
+ sampling_params: SamplingParams,
685
+ images: Optional[PromptImageInput] = None,
686
+ videos: Optional[PromptVideoInput] = None,
687
+ audios: Optional[PromptAudioInput] = None,
688
+ ) -> List[Tuple[List[List[int]], List[str]]]:
689
+ inputs = self.get_inputs(prompts,
690
+ images=images,
691
+ videos=videos,
692
+ audios=audios)
693
+
694
req_outputs = self.model.generate(inputs,
695
sampling_params=sampling_params)
696
@@ -687,24 +732,10 @@ class VllmRunner:
732
videos: Optional[PromptVideoInput] = None,
733
) -> Union[List[TokensTextLogprobs],
734
List[TokensTextLogprobsPromptLogprobs]]:
690
- if images is not None:
691
- assert len(prompts) == len(images)
692
-
693
- if videos is not None:
694
- assert len(prompts) == len(videos)
695
-
696
- inputs = [TextPrompt(prompt=prompt) for prompt in prompts]
697
- if images is not None:
698
- for i, image in enumerate(images):
699
- inputs[i]["multi_modal_data"] = {"image": image}
700
-
701
- if audios is not None:
702
- for i, audio in enumerate(audios):
703
- inputs[i]["multi_modal_data"] = {"audio": audio}
704
-
705
- if videos is not None:
706
- for i, video in enumerate(videos):
707
- inputs[i]["multi_modal_data"] = {"video": video}
735
+ inputs = self.get_inputs(prompts,
736
+ images=images,
737
+ videos=videos,
738
+ audios=audios)
739
740
req_outputs = self.model.generate(inputs,
741
sampling_params=sampling_params)
@@ -741,9 +772,15 @@ class VllmRunner:
772
prompts: List[str],
773
max_tokens: int,
774
images: Optional[PromptImageInput] = None,
775
+ videos: Optional[PromptVideoInput] = None,
776
+ audios: Optional[PromptAudioInput] = None,
777
) -> List[Tuple[List[int], str]]:
778
greedy_params = SamplingParams(temperature=0.0, max_tokens=max_tokens)
746
- outputs = self.generate(prompts, greedy_params, images=images)
779
+ outputs = self.generate(prompts,
780
+ greedy_params,
781
+ images=images,
782
+ videos=videos,
783
+ audios=audios)
784
return [(output_ids[0], output_str[0])
785
for output_ids, output_str in outputs]
786
tests/models/embedding/language/test_embedding.py
+12
-18
@@ -1,10 +1,10 @@
1
-"""Compare the outputs of HF and vLLM for Mistral models using greedy sampling.
1
+"""Compare the embedding outputs of HF and vLLM models.
2
3
Run `pytest tests/models/embedding/language/test_embedding.py`.
4
"""
5
import pytest
6
-import torch
7
-import torch.nn.functional as F
6
+
7
+from ..utils import check_embeddings_close
8
9
MODELS = [
10
"intfloat/e5-mistral-7b-instruct",
@@ -12,14 +12,6 @@ MODELS = [
12
]
13
14
15
-def compare_embeddings(embeddings1, embeddings2):
16
- similarities = [
17
- F.cosine_similarity(torch.tensor(e1), torch.tensor(e2), dim=0)
18
- for e1, e2 in zip(embeddings1, embeddings2)
19
- ]
20
- return similarities
21
-
22
-
15
@pytest.mark.parametrize("model", MODELS)
16
@pytest.mark.parametrize("dtype", ["half"])
17
def test_models(
@@ -37,15 +29,17 @@ def test_models(
29
# So we need to strip the input texts to avoid test failing.
30
example_prompts = [str(s).strip() for s in example_prompts]
31
40
- with hf_runner(model, dtype=dtype, is_embedding_model=True) as hf_model:
32
+ with hf_runner(model, dtype=dtype,
33
+ is_sentence_transformer=True) as hf_model:
34
hf_outputs = hf_model.encode(example_prompts)
35
36
with vllm_runner(model, dtype=dtype) as vllm_model:
37
vllm_outputs = vllm_model.encode(example_prompts)
38
46
- similarities = compare_embeddings(hf_outputs, vllm_outputs)
47
- all_similarities = torch.stack(similarities)
48
- tolerance = 1e-2
49
- assert torch.all((all_similarities <= 1.0 + tolerance)
50
- & (all_similarities >= 1.0 - tolerance)
51
- ), f"Not all values are within {tolerance} of 1.0"
39
+ check_embeddings_close(
40
+ embeddings_0_lst=hf_outputs,
41
+ embeddings_1_lst=vllm_outputs,
42
+ name_0="hf",
43
+ name_1="vllm",
44
+ tol=1e-2,
45
+ )
tests/models/embedding/utils.py
new
+29
@@ -0,0 +1,29 @@
1
+from typing import List, Sequence
2
+
3
+import torch
4
+import torch.nn.functional as F
5
+
6
+
7
+def check_embeddings_close(
8
+ *,
9
+ embeddings_0_lst: Sequence[List[float]],
10
+ embeddings_1_lst: Sequence[List[float]],
11
+ name_0: str,
12
+ name_1: str,
13
+ tol: float = 1e-3,
14
+) -> None:
15
+ assert len(embeddings_0_lst) == len(embeddings_1_lst)
16
+
17
+ for prompt_idx, (embeddings_0, embeddings_1) in enumerate(
18
+ zip(embeddings_0_lst, embeddings_1_lst)):
19
+ assert len(embeddings_0) == len(embeddings_1)
20
+
21
+ sim = F.cosine_similarity(torch.tensor(embeddings_0),
22
+ torch.tensor(embeddings_1),
23
+ dim=0)
24
+
25
+ fail_msg = (f"Test{prompt_idx}:"
26
+ f"\n{name_0}:\t{embeddings_0!r}"
27
+ f"\n{name_1}:\t{embeddings_1!r}")
28
+
29
+ assert sim >= 1 - tol, fail_msg
tests/models/embedding/vision_language/__init__.py
tests/models/embedding/vision_language/test_phi3v.py
new
+62
@@ -0,0 +1,62 @@
1
+import pytest
2
+import torch.nn.functional as F
3
+
4
+from ....conftest import IMAGE_ASSETS
5
+from ..utils import check_embeddings_close
6
+
7
+HF_IMAGE_PROMPTS = IMAGE_ASSETS.prompts({
8
+ "stop_sign":
9
+ "<|image_1|> Select the portion of the image that isolates the object of the given label: The label of the object is stop sign", # noqa: E501
10
+ "cherry_blossom":
11
+ "<|image_1|> Represent the given image with the following question: What is in the image", # noqa: E501
12
+})
13
+
14
+MODELS = ["TIGER-Lab/VLM2Vec-Full"]
15
+
16
+
17
+@pytest.mark.parametrize("model", MODELS)
18
+@pytest.mark.parametrize("dtype", ["half"])
19
+def test_models(
20
+ hf_runner,
21
+ vllm_runner,
22
+ example_prompts,
23
+ model: str,
24
+ dtype: str,
25
+) -> None:
26
+ # NOTE: take care of the order. run vLLM first, and then run HF.
27
+ # vLLM needs a fresh new process without cuda initialization.
28
+ # if we run HF first, the cuda initialization will be done and it
29
+ # will hurt multiprocessing backend with fork method (the default method).
30
+ with vllm_runner(model,
31
+ max_model_len=4096,
32
+ max_num_seqs=2,
33
+ dtype=dtype,
34
+ enforce_eager=True) as vllm_model:
35
+ vllm_outputs = vllm_model.encode(example_prompts)
36
+
37
+ with hf_runner(model, dtype=dtype) as hf_model:
38
+ all_inputs = hf_model.get_inputs(example_prompts)
39
+
40
+ all_outputs = []
41
+ for inputs in all_inputs:
42
+ # Based on: https://github.com/TIGER-AI-Lab/VLM2Vec/blob/db3b951bccabba220c1f53ab46a734e50dd2fc08/src/model.py
43
+ outputs = hf_model.model(
44
+ **hf_model.wrap_device(inputs,
45
+ device=hf_model.model.device.type),
46
+ return_dict=True,
47
+ output_hidden_states=True,
48
+ )
49
+ last_hidden_state = outputs.hidden_states[-1][0]
50
+ reps = last_hidden_state[inputs.attention_mask[0].sum() - 1]
51
+ pooled_output = F.normalize(reps, p=2, dim=-1)
52
+
53
+ all_outputs.append(pooled_output.tolist())
54
+
55
+ hf_outputs = all_outputs
56
+
57
+ check_embeddings_close(
58
+ embeddings_0_lst=hf_outputs,
59
+ embeddings_1_lst=vllm_outputs,
60
+ name_0="hf",
61
+ name_1="vllm",
62
+ )
tests/plugins/vllm_add_dummy_model/vllm_add_dummy_model/my_gemma_embedding.py
+1
-1
@@ -3,7 +3,7 @@ from typing import List, Optional, Union
3
import torch
4
5
from vllm.attention import AttentionMetadata
6
-from vllm.model_executor.models.gemma2_embedding import Gemma2EmbeddingModel
6
+from vllm.model_executor.models.gemma2 import Gemma2EmbeddingModel
7
from vllm.sequence import IntermediateTensors
8
9
vllm/config.py
+10
-1
@@ -237,7 +237,16 @@ class ModelConfig:
237
238
def _verify_embedding_mode(self) -> None:
239
architectures = getattr(self.hf_config, "architectures", [])
240
- self.embedding_mode = ModelRegistry.is_embedding_model(architectures)
240
+
241
+ # TODO: Allow the same model architecture to be specified as either
242
+ # generation or embedding model
243
+ if "Phi3VForCausalLM" in architectures:
244
+ # Match both remote and local names
245
+ embedding_mode = "/VLM2Vec" in self.model
246
+ else:
247
+ embedding_mode = ModelRegistry.is_embedding_model(architectures)
248
+
249
+ self.embedding_mode = embedding_mode
250
251
def _parse_quant_hf_config(self):
252
quant_cfg = getattr(self.hf_config, "quantization_config", None)
vllm/model_executor/models/gemma2.py
+50
-1
@@ -31,14 +31,16 @@ from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
31
QKVParallelLinear,
32
RowParallelLinear)
33
from vllm.model_executor.layers.logits_processor import LogitsProcessor
34
+from vllm.model_executor.layers.pooler import Pooler, PoolingType
35
from vllm.model_executor.layers.quantization import QuantizationConfig
36
from vllm.model_executor.layers.rotary_embedding import get_rope
37
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
38
from vllm.model_executor.layers.vocab_parallel_embedding import (
39
VocabParallelEmbedding)
40
from vllm.model_executor.model_loader.weight_utils import default_weight_loader
41
+from vllm.model_executor.pooling_metadata import PoolingMetadata
42
from vllm.model_executor.sampling_metadata import SamplingMetadata
41
-from vllm.sequence import IntermediateTensors
43
+from vllm.sequence import IntermediateTensors, PoolerOutput
44
45
from .interfaces import SupportsLoRA, SupportsPP
46
from .utils import (AutoWeightsLoader, is_pp_missing_parameter,
@@ -461,3 +463,50 @@ class Gemma2ForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
463
if self.config.tie_word_embeddings else None),
464
)
465
loader.load_weights(weights)
466
+
467
+
468
+class Gemma2EmbeddingModel(nn.Module, SupportsPP):
469
+ """
470
+ A model that uses Gemma2 with additional embedding functionalities.
471
+
472
+ This class encapsulates the Gemma2Model and provides an interface for
473
+ embedding operations and customized pooling functions.
474
+
475
+ Attributes:
476
+ model: An instance of Gemma2Model used for forward operations.
477
+ _pooler: An instance of Pooler used for pooling operations.
478
+ """
479
+
480
+ def __init__(
481
+ self,
482
+ **kwargs,
483
+ ) -> None:
484
+ super().__init__()
485
+
486
+ self.model = Gemma2Model(**kwargs)
487
+ self._pooler = Pooler(pooling_type=PoolingType.LAST, normalize=True)
488
+
489
+ self.make_empty_intermediate_tensors = (
490
+ self.model.make_empty_intermediate_tensors)
491
+
492
+ def forward(
493
+ self,
494
+ input_ids: Optional[torch.Tensor],
495
+ positions: torch.Tensor,
496
+ kv_caches: List[torch.Tensor],
497
+ attn_metadata: AttentionMetadata,
498
+ intermediate_tensors: Optional[IntermediateTensors] = None,
499
+ inputs_embeds: Optional[torch.Tensor] = None,
500
+ ) -> Union[torch.Tensor, IntermediateTensors]:
501
+ return self.model(input_ids, positions, kv_caches, attn_metadata,
502
+ intermediate_tensors, inputs_embeds)
503
+
504
+ def pooler(
505
+ self,
506
+ hidden_states: torch.Tensor,
507
+ pooling_metadata: PoolingMetadata,
508
+ ) -> Optional[PoolerOutput]:
509
+ return self._pooler(hidden_states, pooling_metadata)
510
+
511
+ def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
512
+ self.model.load_weights(weights)
vllm/model_executor/models/gemma2_embedding.py
deleted
-57
@@ -1,57 +0,0 @@
1
-from typing import Iterable, List, Optional, Tuple, Union
2
-
3
-import torch
4
-from torch import nn
5
-
6
-from vllm.attention import AttentionMetadata
7
-from vllm.model_executor.layers.pooler import Pooler, PoolingType
8
-from vllm.model_executor.pooling_metadata import PoolingMetadata
9
-from vllm.sequence import IntermediateTensors, PoolerOutput
10
-
11
-from .gemma2 import Gemma2Model
12
-from .interfaces import SupportsPP
13
-
14
-
15
-class Gemma2EmbeddingModel(nn.Module, SupportsPP):
16
- """A model that uses Gemma2 with additional embedding functionalities.
17
-
18
- This class encapsulates the Gemma2Model and provides an interface for
19
- embedding operations and customized pooling functions.
20
-
21
- Attributes:
22
- model: An instance of Gemma2Model used for forward operations.
23
- _pooler: An instance of Pooler used for pooling operations.
24
- """
25
-
26
- def __init__(
27
- self,
28
- **kwargs,
29
- ) -> None:
30
- super().__init__()
31
- self.model = Gemma2Model(**kwargs)
32
- self._pooler = Pooler(pooling_type=PoolingType.LAST, normalize=True)
33
-
34
- self.make_empty_intermediate_tensors = (
35
- self.model.make_empty_intermediate_tensors)
36
-
37
- def forward(
38
- self,
39
- input_ids: Optional[torch.Tensor],
40
- positions: torch.Tensor,
41
- kv_caches: List[torch.Tensor],
42
- attn_metadata: AttentionMetadata,
43
- intermediate_tensors: Optional[IntermediateTensors] = None,
44
- inputs_embeds: Optional[torch.Tensor] = None,
45
- ) -> Union[torch.Tensor, IntermediateTensors]:
46
- return self.model(input_ids, positions, kv_caches, attn_metadata,
47
- intermediate_tensors, inputs_embeds)
48
-
49
- def pooler(
50
- self,
51
- hidden_states: torch.Tensor,
52
- pooling_metadata: PoolingMetadata,
53
- ) -> Optional[PoolerOutput]:
54
- return self._pooler(hidden_states, pooling_metadata)
55
-
56
- def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
57
- self.model.load_weights(weights)
vllm/model_executor/models/llama.py
+52
-1
@@ -38,6 +38,7 @@ from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
38
QKVParallelLinear,
39
RowParallelLinear)
40
from vllm.model_executor.layers.logits_processor import LogitsProcessor
41
+from vllm.model_executor.layers.pooler import Pooler, PoolingType
42
from vllm.model_executor.layers.quantization import QuantizationConfig
43
from vllm.model_executor.layers.quantization.compressed_tensors.utils import (
44
get_compressed_tensors_cache_scale)
@@ -47,8 +48,9 @@ from vllm.model_executor.layers.vocab_parallel_embedding import (
48
DEFAULT_VOCAB_PADDING_SIZE, ParallelLMHead, VocabParallelEmbedding)
49
from vllm.model_executor.model_loader.weight_utils import (
50
default_weight_loader, kv_cache_scales_loader, maybe_remap_kv_scale_name)
51
+from vllm.model_executor.pooling_metadata import PoolingMetadata
52
from vllm.model_executor.sampling_metadata import SamplingMetadata
51
-from vllm.sequence import IntermediateTensors
53
+from vllm.sequence import IntermediateTensors, PoolerOutput
54
from vllm.utils import is_hip
55
56
from .interfaces import SupportsLoRA, SupportsPP
@@ -615,3 +617,52 @@ class LlamaForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
617
name = name.replace(item, mapping[item])
618
619
return name, loaded_weight
620
+
621
+
622
+class LlamaEmbeddingModel(nn.Module, SupportsPP):
623
+ """
624
+ A model that uses Llama with additional embedding functionalities.
625
+
626
+ This class encapsulates the LlamaModel and provides an interface for
627
+ embedding operations and customized pooling functions.
628
+
629
+ Attributes:
630
+ model: An instance of LlamaModel used for forward operations.
631
+ _pooler: An instance of Pooler used for pooling operations.
632
+ """
633
+
634
+ def __init__(
635
+ self,
636
+ **kwargs,
637
+ ) -> None:
638
+ super().__init__()
639
+
640
+ self.model = LlamaModel(**kwargs)
641
+ self._pooler = Pooler(pooling_type=PoolingType.LAST, normalize=True)
642
+ self.make_empty_intermediate_tensors = (
643
+ self.model.make_empty_intermediate_tensors)
644
+
645
+ def forward(
646
+ self,
647
+ input_ids: Optional[torch.Tensor],
648
+ positions: torch.Tensor,
649
+ kv_caches: List[torch.Tensor],
650
+ attn_metadata: AttentionMetadata,
651
+ intermediate_tensors: Optional[IntermediateTensors] = None,
652
+ inputs_embeds: Optional[torch.Tensor] = None,
653
+ ) -> Union[torch.Tensor, IntermediateTensors]:
654
+ return self.model(input_ids, positions, kv_caches, attn_metadata,
655
+ intermediate_tensors, inputs_embeds)
656
+
657
+ def pooler(
658
+ self,
659
+ hidden_states: torch.Tensor,
660
+ pooling_metadata: PoolingMetadata,
661
+ ) -> Optional[PoolerOutput]:
662
+ return self._pooler(hidden_states, pooling_metadata)
663
+
664
+ def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
665
+ self.model.load_weights(weights)
666
+
667
+ def load_kv_cache_scales(self, quantization_param_path: str) -> None:
668
+ self.model.load_kv_cache_scales(quantization_param_path)
vllm/model_executor/models/llama_embedding.py
deleted
-59
@@ -1,59 +0,0 @@
1
-from typing import Iterable, List, Optional, Tuple, Union
2
-
3
-import torch
4
-from torch import nn
5
-
6
-from vllm.attention import AttentionMetadata
7
-from vllm.model_executor.layers.pooler import Pooler, PoolingType
8
-from vllm.model_executor.pooling_metadata import PoolingMetadata
9
-from vllm.sequence import IntermediateTensors, PoolerOutput
10
-
11
-from .interfaces import SupportsPP
12
-from .llama import LlamaModel
13
-
14
-
15
-class LlamaEmbeddingModel(nn.Module, SupportsPP):
16
- """A model that uses Llama with additional embedding functionalities.
17
-
18
- This class encapsulates the LlamaModel and provides an interface for
19
- embedding operations and customized pooling functions.
20
-
21
- Attributes:
22
- model: An instance of LlamaModel used for forward operations.
23
- _pooler: An instance of Pooler used for pooling operations.
24
- """
25
-
26
- def __init__(
27
- self,
28
- **kwargs,
29
- ) -> None:
30
- super().__init__()
31
- self.model = LlamaModel(**kwargs)
32
- self._pooler = Pooler(pooling_type=PoolingType.LAST, normalize=True)
33
- self.make_empty_intermediate_tensors = (
34
- self.model.make_empty_intermediate_tensors)
35
-
36
- def forward(
37
- self,
38
- input_ids: Optional[torch.Tensor],
39
- positions: torch.Tensor,
40
- kv_caches: List[torch.Tensor],
41
- attn_metadata: AttentionMetadata,
42
- intermediate_tensors: Optional[IntermediateTensors] = None,
43
- inputs_embeds: Optional[torch.Tensor] = None,
44
- ) -> Union[torch.Tensor, IntermediateTensors]:
45
- return self.model(input_ids, positions, kv_caches, attn_metadata,
46
- intermediate_tensors, inputs_embeds)
47
-
48
- def pooler(
49
- self,
50
- hidden_states: torch.Tensor,
51
- pooling_metadata: PoolingMetadata,
52
- ) -> Optional[PoolerOutput]:
53
- return self._pooler(hidden_states, pooling_metadata)
54
-
55
- def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
56
- self.model.load_weights(weights)
57
-
58
- def load_kv_cache_scales(self, quantization_param_path: str) -> None:
59
- self.model.load_kv_cache_scales(quantization_param_path)
vllm/model_executor/models/phi3v.py
+58
-25
@@ -29,14 +29,18 @@ from vllm.attention import AttentionMetadata
29
from vllm.config import CacheConfig, ModelConfig, MultiModalConfig
30
from vllm.inputs import INPUT_REGISTRY, InputContext, LLMInputs
31
from vllm.logger import init_logger
32
+from vllm.model_executor.layers.pooler import Pooler, PoolingType
33
from vllm.model_executor.layers.quantization import QuantizationConfig
34
from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
35
+from vllm.model_executor.layers.vocab_parallel_embedding import (
36
+ VocabParallelEmbedding)
37
from vllm.model_executor.models.clip import CLIPVisionModel
38
from vllm.model_executor.models.llama import LlamaForCausalLM
39
+from vllm.model_executor.pooling_metadata import PoolingMetadata
40
from vllm.model_executor.sampling_metadata import SamplingMetadata
41
from vllm.multimodal import MULTIMODAL_REGISTRY
42
from vllm.multimodal.utils import cached_get_tokenizer, repeat_and_pad_token
39
-from vllm.sequence import IntermediateTensors
43
+from vllm.sequence import IntermediateTensors, PoolerOutput
44
from vllm.utils import is_list_of
45
46
from .clip import dummy_image_for_clip, dummy_seq_data_for_clip
@@ -289,10 +293,6 @@ class Phi3HDImageEmbedding(Phi3ImageEmbeddingBase):
293
dim=2).reshape(num_images, -1, hid_dim)
294
return image_features_hd_newline
295
292
- def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
293
- loader = AutoWeightsLoader(self)
294
- loader.load_weights(weights)
295
-
296
297
# Based on https://huggingface.co/microsoft/Phi-3-vision-128k-instruct/blob/main/image_processing_phi3_v.py#L57
298
def _calc_padded_size(*, width: int, height: int, padding_unit: int = 336):
@@ -385,23 +385,28 @@ def dummy_data_for_phi3v(ctx: InputContext,
385
return seq_data, mm_data
386
387
388
-# Reserve this function to also handle placeholders for additional images
389
-# [ref: PR #5820]
388
@lru_cache
391
-def _get_image_placeholder_token_ids(model_config: ModelConfig,
392
- idx: int) -> List[int]:
389
+def _get_image_placeholder_token_id_candidates(
390
+ model_config: ModelConfig,
391
+ idx: int,
392
+) -> List[List[int]]:
393
assert idx > 0
394
395
tokenizer = cached_get_tokenizer(model_config.tokenizer)
396
397
+ # This is used when the image token is at the start of the string
398
+ start_candidate = tokenizer.encode(f"<|image_{idx}|>",
399
+ add_special_tokens=False)
400
+
401
+ # This is used when the image token is in the middle of the string
402
# We need to get the token for "<", not "▁<"
403
# https://huggingface.co/microsoft/Phi-3-vision-128k-instruct/raw/main/tokenizer.json
404
a_token_id, = tokenizer.encode("a", add_special_tokens=False)
400
- a_token_id_, *image_placeholder_token_ids = tokenizer.encode(
401
- f"a<|image_{idx}|>", add_special_tokens=False)
405
+ a_token_id_, *middle_candidate = tokenizer.encode(f"a<|image_{idx}|>",
406
+ add_special_tokens=False)
407
assert a_token_id == a_token_id_
408
404
- return image_placeholder_token_ids
409
+ return [start_candidate, middle_candidate]
410
411
412
def input_processor_for_phi3v(ctx: InputContext,
@@ -461,16 +466,20 @@ def input_processor_for_phi3v(ctx: InputContext,
466
467
prompt_token_ids = llm_inputs["prompt_token_ids"].copy()
468
464
- # masked place_holder with image token id
469
+ print("prompt_token_ids (old)", prompt_token_ids)
470
+
471
+ # masked placeholder with image token id
472
for idx in image_idx:
466
- image_token_ids = _get_image_placeholder_token_ids(model_config,
467
- idx=idx)
468
- for i in range(len(prompt_token_ids) - len(image_token_ids) + 1):
469
- if prompt_token_ids[i:i + len(image_token_ids)] == image_token_ids:
470
- prompt_token_ids[i:i + len(image_token_ids)] = [
471
- _IMAGE_TOKEN_ID
472
- ] * len(image_token_ids)
473
- break
473
+ candidates = _get_image_placeholder_token_id_candidates(model_config,
474
+ idx=idx)
475
+
476
+ for candidate in candidates:
477
+ for i in range(len(prompt_token_ids) - len(candidate) + 1):
478
+ if prompt_token_ids[i:i + len(candidate)] == candidate:
479
+ prompt_token_ids[i:i +
480
+ len(candidate)] = ([_IMAGE_TOKEN_ID] *
481
+ len(candidate))
482
+ break
483
484
# merge consecutive tag ids
485
merged_token_ids: List[int] = []
@@ -520,12 +529,23 @@ class Phi3VForCausalLM(nn.Module, SupportsMultiModal, SupportsPP):
529
self.multimodal_config = multimodal_config
530
self.image_token_id = _IMAGE_TOKEN_ID
531
523
- # TODO: Optionally initializes this for supporting embeddings.
532
+ self.embed_tokens = VocabParallelEmbedding(
533
+ config.vocab_size,
534
+ config.hidden_size,
535
+ org_num_embeddings=config.vocab_size,
536
+ quant_config=quant_config,
537
+ )
538
+
539
+ # TODO: Optionally initializes this for supporting input embeddings.
540
self.vision_embed_tokens = Phi3HDImageEmbedding(config)
541
542
self.language_model = LlamaForCausalLM(config, cache_config,
543
quant_config)
544
545
+ # The same model class supports both language generation and embedding
546
+ # because the architecture name is the same
547
+ self._pooler = Pooler(pooling_type=PoolingType.LAST, normalize=True)
548
+
549
self.make_empty_intermediate_tensors = (
550
self.language_model.make_empty_intermediate_tensors)
551
@@ -649,8 +669,7 @@ class Phi3VForCausalLM(nn.Module, SupportsMultiModal, SupportsPP):
669
670
if image_input is not None:
671
vision_embeddings = self._process_image_input(image_input)
652
- inputs_embeds = self.language_model.model.get_input_embeddings(
653
- input_ids)
672
+ inputs_embeds = self.embed_tokens(input_ids)
673
inputs_embeds = merge_multimodal_embeddings(
674
input_ids, inputs_embeds, vision_embeddings,
675
self.image_token_id)
@@ -682,13 +701,27 @@ class Phi3VForCausalLM(nn.Module, SupportsMultiModal, SupportsPP):
701
) -> Optional[SamplerOutput]:
702
return self.language_model.sample(logits, sampling_metadata)
703
704
+ def pooler(
705
+ self,
706
+ hidden_states: torch.Tensor,
707
+ pooling_metadata: PoolingMetadata,
708
+ ) -> Optional[PoolerOutput]:
709
+ return self._pooler(hidden_states, pooling_metadata)
710
+
711
def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
712
hf_to_vllm_mapper = WeightsMapper(
713
orig_to_new_prefix={
714
+ "model.vision_embed_tokens.wte": "embed_tokens",
715
"model.vision_embed_tokens.": "vision_embed_tokens.",
716
"lm_head.": "language_model.lm_head.",
717
"model.": "language_model.model.",
718
})
719
720
loader = AutoWeightsLoader(self)
694
- loader.load_weights(weights, mapper=hf_to_vllm_mapper)
721
+ autoloaded_weights = loader.load_weights(weights,
722
+ mapper=hf_to_vllm_mapper)
723
+
724
+ # The HF config doesn't specify whether these are tied,
725
+ # so we detect it this way
726
+ if "embed_tokens" not in autoloaded_weights:
727
+ self.embed_tokens = self.language_model.model.embed_tokens
vllm/model_executor/models/registry.py
+5
-2
@@ -86,9 +86,12 @@ _TEXT_GENERATION_MODELS = {
86
}
87
88
_EMBEDDING_MODELS = {
89
- "MistralModel": ("llama_embedding", "LlamaEmbeddingModel"),
89
+ # [Text-only]
90
+ "Gemma2Model": ("gemma2", "Gemma2EmbeddingModel"),
91
+ "MistralModel": ("llama", "LlamaEmbeddingModel"),
92
"Qwen2ForRewardModel": ("qwen2_rm", "Qwen2ForRewardModel"),
91
- "Gemma2Model": ("gemma2_embedding", "Gemma2EmbeddingModel"),
93
+ # [Multimodal]
94
+ "Phi3VForCausalLM": ("phi3v", "Phi3VForCausalLM"),
95
}
96
97
_MULTIMODAL_MODELS = {
vllm/model_executor/models/utils.py
+14
-9
@@ -124,7 +124,7 @@ class AutoWeightsLoader:
124
base_prefix: str,
125
param: nn.Parameter,
126
weights: Iterable[Tuple[str, torch.Tensor]],
127
- ) -> None:
127
+ ) -> Iterable[str]:
128
for weight_name, weight_data in weights:
129
weight_qualname = self._get_qualname(base_prefix, weight_name)
130
@@ -143,12 +143,14 @@ class AutoWeightsLoader:
143
default_weight_loader)
144
weight_loader(param, weight_data)
145
146
+ yield weight_qualname
147
+
148
def _load_module(
149
self,
150
base_prefix: str,
151
module: nn.Module,
152
weights: Iterable[Tuple[str, torch.Tensor]],
151
- ) -> None:
153
+ ) -> Iterable[str]:
154
if isinstance(module, PPMissingLayer):
155
return
156
@@ -170,14 +172,16 @@ class AutoWeightsLoader:
172
continue
173
174
if child_prefix in child_modules:
173
- self._load_module(prefix, child_modules[child_prefix],
174
- child_weights)
175
+ yield from self._load_module(prefix,
176
+ child_modules[child_prefix],
177
+ child_weights)
178
elif child_prefix in child_params:
176
- self._load_param(prefix, child_params[child_prefix],
177
- child_weights)
179
+ yield from self._load_param(prefix, child_params[child_prefix],
180
+ child_weights)
181
else:
182
if not self._can_ignore_unexpected(prefix):
180
- msg = f"There is no module or parameter named '{prefix}'"
183
+ msg = (f"There is no module or parameter named '{prefix}' "
184
+ f"in {type(self.module).__name__}")
185
raise ValueError(msg)
186
187
def load_weights(
@@ -185,11 +189,12 @@ class AutoWeightsLoader:
189
weights: Iterable[Tuple[str, torch.Tensor]],
190
*,
191
mapper: Optional[WeightsMapper] = None,
188
- ) -> None:
192
+ ) -> List[str]:
193
if mapper is not None:
194
weights = mapper.apply(weights)
195
192
- self._load_module("", self.module, weights)
196
+ autoloaded_weights = list(self._load_module("", self.module, weights))
197
+ return autoloaded_weights
198
199
200
def init_vllm_registered_model(