@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 7abba39ee6
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 10:45 UTC

[Model] VLM2Vec, the first multimodal embedding model in vLLM (#9303)

Cyrus Leung committed Oct 16, 2024 at 14:31 UTC 7abba39ee64c1e2c84f48d7c38b2cd1c24bb0ebb
16 files changed +465 -261
docs/source/models/supported_models.rst
+53 -26
@@ -3,7 +3,7 @@
3 Supported Models
4 ================
5
6 -vLLM supports a variety of generative Transformer models in `HuggingFace Transformers <https://huggingface.co/models>`_.
6 +vLLM supports a variety of generative Transformer models in `HuggingFace (HF) Transformers <https://huggingface.co/models>`_.
7 The following is the list of model architectures that are currently supported by vLLM.
8 Alongside each architecture, we include some popular models that use it.
9
@@ -19,7 +19,7 @@ Text Generation
19
20 * - Architecture
21 - Models
22 - - Example HuggingFace Models
22 + - Example HF Models
23 - :ref:`LoRA <lora>`
24 - :ref:`PP <distributed_serving>`
25 * - :code:`AquilaForCausalLM`
@@ -280,7 +280,7 @@ Text Embedding
280
281 * - Architecture
282 - Models
283 - - Example HuggingFace Models
283 + - Example HF Models
284 - :ref:`LoRA <lora>`
285 - :ref:`PP <distributed_serving>`
286 * - :code:`Gemma2Model`
@@ -303,7 +303,7 @@ Reward Modeling
303
304 * - Architecture
305 - Models
306 - - Example HuggingFace Models
306 + - Example HF Models
307 - :ref:`LoRA <lora>`
308 - :ref:`PP <distributed_serving>`
309 * - :code:`Qwen2ForRewardModel`
@@ -316,7 +316,14 @@ Reward Modeling
316 As an interim measure, these models are supported via Embeddings API. See `this RFC <https://github.com/vllm-project/vllm/issues/8967>`_ for upcoming changes.
317
318 Multimodal Language Models
319 -^^^^^^^^^^^^^^^^^^^^^^^^^^^^
319 +^^^^^^^^^^^^^^^^^^^^^^^^^^
320 +
321 +The following modalities are supported depending on the model:
322 +
323 +- **T**\ ext
324 +- **I**\ mage
325 +- **V**\ ideo
326 +- **A**\ udio
327
328 .. _supported_vlms:
329
@@ -324,78 +331,78 @@ Text Generation
331 ---------------
332
333 .. list-table::
327 - :widths: 25 25 25 25 5 5
334 + :widths: 25 25 15 25 5 5
335 :header-rows: 1
336
337 * - Architecture
338 - Models
332 - - Modalities
333 - - Example HuggingFace Models
339 + - Inputs
340 + - Example HF Models
341 - :ref:`LoRA <lora>`
342 - :ref:`PP <distributed_serving>`
343 * - :code:`Blip2ForConditionalGeneration`
344 - BLIP-2
338 - - Image\ :sup:`E`
345 + - T + I\ :sup:`E`
346 - :code:`Salesforce/blip2-opt-2.7b`, :code:`Salesforce/blip2-opt-6.7b`, etc.
347 -
348 - ✅︎
349 * - :code:`ChameleonForConditionalGeneration`
350 - Chameleon
344 - - Image
351 + - T + I
352 - :code:`facebook/chameleon-7b` etc.
353 -
354 - ✅︎
355 * - :code:`FuyuForCausalLM`
356 - Fuyu
350 - - Image
357 + - T + I
358 - :code:`adept/fuyu-8b` etc.
359 -
360 - ✅︎
361 * - :code:`ChatGLMModel`
362 - GLM-4V
356 - - Image
363 + - T + I
364 - :code:`THUDM/glm-4v-9b` etc.
365 -
366 - ✅︎
367 * - :code:`InternVLChatModel`
368 - InternVL2
362 - - Image\ :sup:`E+`
369 + - T + I\ :sup:`E+`
370 - :code:`OpenGVLab/InternVL2-4B`, :code:`OpenGVLab/InternVL2-8B`, etc.
371 -
372 - ✅︎
373 * - :code:`LlavaForConditionalGeneration`
374 - LLaVA-1.5
368 - - Image\ :sup:`E+`
375 + - T + I\ :sup:`E+`
376 - :code:`llava-hf/llava-1.5-7b-hf`, :code:`llava-hf/llava-1.5-13b-hf`, etc.
377 -
378 - ✅︎
379 * - :code:`LlavaNextForConditionalGeneration`
380 - LLaVA-NeXT
374 - - Image\ :sup:`E+`
381 + - T + I\ :sup:`E+`
382 - :code:`llava-hf/llava-v1.6-mistral-7b-hf`, :code:`llava-hf/llava-v1.6-vicuna-7b-hf`, etc.
383 -
384 - ✅︎
385 * - :code:`LlavaNextVideoForConditionalGeneration`
386 - LLaVA-NeXT-Video
380 - - Video
387 + - T + V
388 - :code:`llava-hf/LLaVA-NeXT-Video-7B-hf`, etc.
389 -
390 - ✅︎
391 * - :code:`LlavaOnevisionForConditionalGeneration`
392 - LLaVA-Onevision
386 - - Image\ :sup:`+` / Video
393 + - T + I\ :sup:`+` + V
394 - :code:`llava-hf/llava-onevision-qwen2-7b-ov-hf`, :code:`llava-hf/llava-onevision-qwen2-0.5b-ov-hf`, etc.
395 -
396 - ✅︎
397 * - :code:`MiniCPMV`
398 - MiniCPM-V
392 - - Image\ :sup:`E+`
399 + - T + I\ :sup:`E+`
400 - :code:`openbmb/MiniCPM-V-2` (see note), :code:`openbmb/MiniCPM-Llama3-V-2_5`, :code:`openbmb/MiniCPM-V-2_6`, etc.
401 - ✅︎
402 - ✅︎
403 * - :code:`MllamaForConditionalGeneration`
404 - Llama 3.2
398 - - Image
405 + - T + I
406 - :code:`meta-llama/Llama-3.2-90B-Vision-Instruct`, :code:`meta-llama/Llama-3.2-11B-Vision`, etc.
407 -
408 -
@@ -407,43 +414,43 @@ Text Generation
414 - ✅︎
415 * - :code:`NVLM_D_Model`
416 - NVLM-D 1.0
410 - - Image\ :sup:`E+`
417 + - T + I\ :sup:`E+`
418 - :code:`nvidia/NVLM-D-72B`, etc.
419 -
420 - ✅︎
421 * - :code:`PaliGemmaForConditionalGeneration`
422 - PaliGemma
416 - - Image\ :sup:`E`
423 + - T + I\ :sup:`E`
424 - :code:`google/paligemma-3b-pt-224`, :code:`google/paligemma-3b-mix-224`, etc.
425 -
426 - ✅︎
427 * - :code:`Phi3VForCausalLM`
428 - Phi-3-Vision, Phi-3.5-Vision
422 - - Image\ :sup:`E+`
429 + - T + I\ :sup:`E+`
430 - :code:`microsoft/Phi-3-vision-128k-instruct`, :code:`microsoft/Phi-3.5-vision-instruct` etc.
431 -
432 - ✅︎
433 * - :code:`PixtralForConditionalGeneration`
434 - Pixtral
428 - - Image\ :sup:`+`
435 + - T + I\ :sup:`+`
436 - :code:`mistralai/Pixtral-12B-2409`
437 -
438 - ✅︎
439 * - :code:`QWenLMHeadModel`
440 - Qwen-VL
434 - - Image\ :sup:`E+`
441 + - T + I\ :sup:`E+`
442 - :code:`Qwen/Qwen-VL`, :code:`Qwen/Qwen-VL-Chat`, etc.
443 -
444 - ✅︎
445 * - :code:`Qwen2VLForConditionalGeneration`
446 - Qwen2-VL
440 - - Image\ :sup:`E+` / Video\ :sup:`+`
447 + - T + I\ :sup:`E+` + V\ :sup:`+`
448 - :code:`Qwen/Qwen2-VL-2B-Instruct`, :code:`Qwen/Qwen2-VL-7B-Instruct`, :code:`Qwen/Qwen2-VL-72B-Instruct`, etc.
449 -
450 - ✅︎
451 * - :code:`UltravoxModel`
452 - Ultravox
446 - - Audio\ :sup:`E+`
453 + - T + A\ :sup:`E+`
454 - :code:`fixie-ai/ultravox-v0_3`
455 -
456 - ✅︎
@@ -455,6 +462,26 @@ Text Generation
462 For :code:`openbmb/MiniCPM-V-2`, the official repo doesn't work yet, so we need to use a fork (:code:`HwwwH/MiniCPM-V-2`) for now.
463 For more details, please see: https://github.com/vllm-project/vllm/pull/4087#issuecomment-2250397630
464
465 +Multimodal Embedding
466 +--------------------
467 +
468 +.. list-table::
469 + :widths: 25 25 15 25 5 5
470 + :header-rows: 1
471 +
472 + * - Architecture
473 + - Models
474 + - Inputs
475 + - Example HF Models
476 + - :ref:`LoRA <lora>`
477 + - :ref:`PP <distributed_serving>`
478 + * - :code:`Phi3VForCausalLM`
479 + - Phi-3-Vision-based
480 + - T + I
481 + - :code:`TIGER-Lab/VLM2Vec-Full`
482 + - 🚧
483 + - ✅︎
484 +
485 ----
486
487 If your model uses one of the above model architectures, you can seamlessly run your model with vLLM.
examples/offline_inference_vision_language_embedding.py new
+21
@@ -0,0 +1,21 @@
1 +from vllm import LLM
2 +from vllm.assets.image import ImageAsset
3 +
4 +image = ImageAsset("cherry_blossom").pil_image.convert("RGB")
5 +prompt = "<|image_1|> Represent the given image with the following question: What is in the image" # noqa: E501
6 +
7 +# Create an LLM.
8 +llm = LLM(
9 + model="TIGER-Lab/VLM2Vec-Full",
10 + trust_remote_code=True,
11 + max_model_len=4096,
12 + max_num_seqs=2,
13 + mm_processor_kwargs={"num_crops": 16},
14 +)
15 +
16 +# Generate embedding. The output is a list of EmbeddingRequestOutputs.
17 +outputs = llm.encode({"prompt": prompt, "multi_modal_data": {"image": image}})
18 +
19 +# Print the outputs.
20 +for output in outputs:
21 + print(output.outputs.embedding) # list of 3072 floats
tests/conftest.py
+98 -61
@@ -262,7 +262,7 @@ class HfRunner:
262 dtype: str = "half",
263 *,
264 model_kwargs: Optional[Dict[str, Any]] = None,
265 - is_embedding_model: bool = False,
265 + is_sentence_transformer: bool = False,
266 auto_cls: Type[_BaseAutoModelClass] = AutoModelForCausalLM,
267 postprocess_inputs: Callable[[BatchEncoding],
268 BatchEncoding] = identity,
@@ -271,7 +271,7 @@ class HfRunner:
271
272 self.model_name = model_name
273
274 - if is_embedding_model:
274 + if is_sentence_transformer:
275 # Lazy init required for AMD CI
276 from sentence_transformers import SentenceTransformer
277 self.model = self.wrap_device(
@@ -307,17 +307,23 @@ class HfRunner:
307
308 self.postprocess_inputs = postprocess_inputs
309
310 - def generate(
310 + def get_inputs(
311 self,
312 prompts: List[str],
313 images: Optional[PromptImageInput] = None,
314 - videos: Optional[List[np.ndarray]] = None,
315 - **kwargs: Any,
316 - ) -> List[Tuple[List[List[int]], List[str]]]:
317 - if images:
314 + videos: Optional[PromptVideoInput] = None,
315 + audios: Optional[PromptAudioInput] = None,
316 + ) -> List[BatchEncoding]:
317 + if images is not None:
318 assert len(prompts) == len(images)
319
320 - outputs: List[Tuple[List[List[int]], List[str]]] = []
320 + if videos is not None:
321 + assert len(prompts) == len(videos)
322 +
323 + if audios is not None:
324 + assert len(prompts) == len(audios)
325 +
326 + all_inputs: List[BatchEncoding] = []
327 for i, prompt in enumerate(prompts):
328 processor_kwargs: Dict[str, Any] = {
329 "text": prompt,
@@ -327,10 +333,33 @@ class HfRunner:
333 processor_kwargs["images"] = images[i]
334 if videos is not None and videos[i] is not None:
335 processor_kwargs["videos"] = videos[i]
336 + if audios is not None and audios[i] is not None:
337 + audio, sr = audios[i]
338 + processor_kwargs["audio"] = audio
339 + processor_kwargs["sampling_rate"] = sr
340
341 inputs = self.processor(**processor_kwargs)
342 inputs = self.postprocess_inputs(inputs)
343
344 + all_inputs.append(inputs)
345 +
346 + return all_inputs
347 +
348 + def generate(
349 + self,
350 + prompts: List[str],
351 + images: Optional[PromptImageInput] = None,
352 + videos: Optional[List[np.ndarray]] = None,
353 + audios: Optional[PromptAudioInput] = None,
354 + **kwargs: Any,
355 + ) -> List[Tuple[List[List[int]], List[str]]]:
356 + all_inputs = self.get_inputs(prompts,
357 + images=images,
358 + videos=videos,
359 + audios=audios)
360 +
361 + outputs: List[Tuple[List[List[int]], List[str]]] = []
362 + for inputs in all_inputs:
363 output_ids = self.model.generate(
364 **self.wrap_device(inputs, device=self.model.device.type),
365 use_cache=True,
@@ -350,12 +379,16 @@ class HfRunner:
379 prompts: List[str],
380 max_tokens: int,
381 images: Optional[PromptImageInput] = None,
382 + videos: Optional[List[np.ndarray]] = None,
383 + audios: Optional[PromptAudioInput] = None,
384 **kwargs: Any,
385 ) -> List[Tuple[List[int], str]]:
386 outputs = self.generate(prompts,
387 do_sample=False,
388 max_new_tokens=max_tokens,
389 images=images,
390 + videos=videos,
391 + audios=audios,
392 **kwargs)
393
394 return [(output_ids[0], output_str[0])
@@ -388,22 +421,16 @@ class HfRunner:
421 max_tokens: int,
422 images: Optional[PromptImageInput] = None,
423 videos: Optional[List[np.ndarray]] = None,
424 + audios: Optional[PromptAudioInput] = None,
425 **kwargs: Any,
426 ) -> List[List[torch.Tensor]]:
393 - all_logprobs: List[List[torch.Tensor]] = []
394 - for i, prompt in enumerate(prompts):
395 - processor_kwargs: Dict[str, Any] = {
396 - "text": prompt,
397 - "return_tensors": "pt",
398 - }
399 - if images is not None and images[i] is not None:
400 - processor_kwargs["images"] = images[i]
401 - if videos is not None and videos[i] is not None:
402 - processor_kwargs["videos"] = videos[i]
403 -
404 - inputs = self.processor(**processor_kwargs)
405 - inputs = self.postprocess_inputs(inputs)
427 + all_inputs = self.get_inputs(prompts,
428 + images=images,
429 + videos=videos,
430 + audios=audios)
431
432 + all_logprobs: List[List[torch.Tensor]] = []
433 + for inputs in all_inputs:
434 output = self.model.generate(
435 **self.wrap_device(inputs, device=self.model.device.type),
436 use_cache=True,
@@ -475,28 +502,16 @@ class HfRunner:
502 videos: Optional[List[np.ndarray]] = None,
503 **kwargs: Any,
504 ) -> List[TokensTextLogprobs]:
505 + all_inputs = self.get_inputs(prompts,
506 + images=images,
507 + videos=videos,
508 + audios=audios)
509 +
510 all_logprobs: List[List[Dict[int, float]]] = []
511 all_output_ids: List[List[int]] = []
512 all_output_strs: List[str] = []
513
482 - for i, prompt in enumerate(prompts):
483 - processor_kwargs: Dict[str, Any] = {
484 - "text": prompt,
485 - "return_tensors": "pt",
486 - }
487 - if images is not None and images[i] is not None:
488 - processor_kwargs["images"] = images[i]
489 -
490 - if audios is not None:
491 - audio, sr = audios[i]
492 - processor_kwargs["audio"] = audio
493 - processor_kwargs["sampling_rate"] = sr
494 -
495 - if videos is not None:
496 - processor_kwargs["videos"] = videos[i]
497 - inputs = self.processor(**processor_kwargs)
498 - inputs = self.postprocess_inputs(inputs)
499 -
514 + for inputs in all_inputs:
515 output = self.model.generate(
516 **self.wrap_device(inputs, device=self.model.device.type),
517 use_cache=True,
@@ -632,20 +647,50 @@ class VllmRunner:
647 **kwargs,
648 )
649
635 - def generate(
650 + def get_inputs(
651 self,
652 prompts: List[str],
638 - sampling_params: SamplingParams,
653 images: Optional[PromptImageInput] = None,
640 - ) -> List[Tuple[List[List[int]], List[str]]]:
654 + videos: Optional[PromptVideoInput] = None,
655 + audios: Optional[PromptAudioInput] = None,
656 + ) -> List[TextPrompt]:
657 if images is not None:
658 assert len(prompts) == len(images)
659
660 + if videos is not None:
661 + assert len(prompts) == len(videos)
662 +
663 + if audios is not None:
664 + assert len(prompts) == len(audios)
665 +
666 inputs = [TextPrompt(prompt=prompt) for prompt in prompts]
667 if images is not None:
668 for i, image in enumerate(images):
669 inputs[i]["multi_modal_data"] = {"image": image}
670
671 + if videos is not None:
672 + for i, video in enumerate(videos):
673 + inputs[i]["multi_modal_data"] = {"video": video}
674 +
675 + if audios is not None:
676 + for i, audio in enumerate(audios):
677 + inputs[i]["multi_modal_data"] = {"audio": audio}
678 +
679 + return inputs
680 +
681 + def generate(
682 + self,
683 + prompts: List[str],
684 + sampling_params: SamplingParams,
685 + images: Optional[PromptImageInput] = None,
686 + videos: Optional[PromptVideoInput] = None,
687 + audios: Optional[PromptAudioInput] = None,
688 + ) -> List[Tuple[List[List[int]], List[str]]]:
689 + inputs = self.get_inputs(prompts,
690 + images=images,
691 + videos=videos,
692 + audios=audios)
693 +
694 req_outputs = self.model.generate(inputs,
695 sampling_params=sampling_params)
696
@@ -687,24 +732,10 @@ class VllmRunner:
732 videos: Optional[PromptVideoInput] = None,
733 ) -> Union[List[TokensTextLogprobs],
734 List[TokensTextLogprobsPromptLogprobs]]:
690 - if images is not None:
691 - assert len(prompts) == len(images)
692 -
693 - if videos is not None:
694 - assert len(prompts) == len(videos)
695 -
696 - inputs = [TextPrompt(prompt=prompt) for prompt in prompts]
697 - if images is not None:
698 - for i, image in enumerate(images):
699 - inputs[i]["multi_modal_data"] = {"image": image}
700 -
701 - if audios is not None:
702 - for i, audio in enumerate(audios):
703 - inputs[i]["multi_modal_data"] = {"audio": audio}
704 -
705 - if videos is not None:
706 - for i, video in enumerate(videos):
707 - inputs[i]["multi_modal_data"] = {"video": video}
735 + inputs = self.get_inputs(prompts,
736 + images=images,
737 + videos=videos,
738 + audios=audios)
739
740 req_outputs = self.model.generate(inputs,
741 sampling_params=sampling_params)
@@ -741,9 +772,15 @@ class VllmRunner:
772 prompts: List[str],
773 max_tokens: int,
774 images: Optional[PromptImageInput] = None,
775 + videos: Optional[PromptVideoInput] = None,
776 + audios: Optional[PromptAudioInput] = None,
777 ) -> List[Tuple[List[int], str]]:
778 greedy_params = SamplingParams(temperature=0.0, max_tokens=max_tokens)
746 - outputs = self.generate(prompts, greedy_params, images=images)
779 + outputs = self.generate(prompts,
780 + greedy_params,
781 + images=images,
782 + videos=videos,
783 + audios=audios)
784 return [(output_ids[0], output_str[0])
785 for output_ids, output_str in outputs]
786
tests/models/embedding/language/test_embedding.py
+12 -18
@@ -1,10 +1,10 @@
1 -"""Compare the outputs of HF and vLLM for Mistral models using greedy sampling.
1 +"""Compare the embedding outputs of HF and vLLM models.
2
3 Run `pytest tests/models/embedding/language/test_embedding.py`.
4 """
5 import pytest
6 -import torch
7 -import torch.nn.functional as F
6 +
7 +from ..utils import check_embeddings_close
8
9 MODELS = [
10 "intfloat/e5-mistral-7b-instruct",
@@ -12,14 +12,6 @@ MODELS = [
12 ]
13
14
15 -def compare_embeddings(embeddings1, embeddings2):
16 - similarities = [
17 - F.cosine_similarity(torch.tensor(e1), torch.tensor(e2), dim=0)
18 - for e1, e2 in zip(embeddings1, embeddings2)
19 - ]
20 - return similarities
21 -
22 -
15 @pytest.mark.parametrize("model", MODELS)
16 @pytest.mark.parametrize("dtype", ["half"])
17 def test_models(
@@ -37,15 +29,17 @@ def test_models(
29 # So we need to strip the input texts to avoid test failing.
30 example_prompts = [str(s).strip() for s in example_prompts]
31
40 - with hf_runner(model, dtype=dtype, is_embedding_model=True) as hf_model:
32 + with hf_runner(model, dtype=dtype,
33 + is_sentence_transformer=True) as hf_model:
34 hf_outputs = hf_model.encode(example_prompts)
35
36 with vllm_runner(model, dtype=dtype) as vllm_model:
37 vllm_outputs = vllm_model.encode(example_prompts)
38
46 - similarities = compare_embeddings(hf_outputs, vllm_outputs)
47 - all_similarities = torch.stack(similarities)
48 - tolerance = 1e-2
49 - assert torch.all((all_similarities <= 1.0 + tolerance)
50 - & (all_similarities >= 1.0 - tolerance)
51 - ), f"Not all values are within {tolerance} of 1.0"
39 + check_embeddings_close(
40 + embeddings_0_lst=hf_outputs,
41 + embeddings_1_lst=vllm_outputs,
42 + name_0="hf",
43 + name_1="vllm",
44 + tol=1e-2,
45 + )
tests/models/embedding/utils.py new
+29
@@ -0,0 +1,29 @@
1 +from typing import List, Sequence
2 +
3 +import torch
4 +import torch.nn.functional as F
5 +
6 +
7 +def check_embeddings_close(
8 + *,
9 + embeddings_0_lst: Sequence[List[float]],
10 + embeddings_1_lst: Sequence[List[float]],
11 + name_0: str,
12 + name_1: str,
13 + tol: float = 1e-3,
14 +) -> None:
15 + assert len(embeddings_0_lst) == len(embeddings_1_lst)
16 +
17 + for prompt_idx, (embeddings_0, embeddings_1) in enumerate(
18 + zip(embeddings_0_lst, embeddings_1_lst)):
19 + assert len(embeddings_0) == len(embeddings_1)
20 +
21 + sim = F.cosine_similarity(torch.tensor(embeddings_0),
22 + torch.tensor(embeddings_1),
23 + dim=0)
24 +
25 + fail_msg = (f"Test{prompt_idx}:"
26 + f"\n{name_0}:\t{embeddings_0!r}"
27 + f"\n{name_1}:\t{embeddings_1!r}")
28 +
29 + assert sim >= 1 - tol, fail_msg
tests/models/embedding/vision_language/__init__.py
tests/models/embedding/vision_language/test_phi3v.py new
+62
@@ -0,0 +1,62 @@
1 +import pytest
2 +import torch.nn.functional as F
3 +
4 +from ....conftest import IMAGE_ASSETS
5 +from ..utils import check_embeddings_close
6 +
7 +HF_IMAGE_PROMPTS = IMAGE_ASSETS.prompts({
8 + "stop_sign":
9 + "<|image_1|> Select the portion of the image that isolates the object of the given label: The label of the object is stop sign", # noqa: E501
10 + "cherry_blossom":
11 + "<|image_1|> Represent the given image with the following question: What is in the image", # noqa: E501
12 +})
13 +
14 +MODELS = ["TIGER-Lab/VLM2Vec-Full"]
15 +
16 +
17 +@pytest.mark.parametrize("model", MODELS)
18 +@pytest.mark.parametrize("dtype", ["half"])
19 +def test_models(
20 + hf_runner,
21 + vllm_runner,
22 + example_prompts,
23 + model: str,
24 + dtype: str,
25 +) -> None:
26 + # NOTE: take care of the order. run vLLM first, and then run HF.
27 + # vLLM needs a fresh new process without cuda initialization.
28 + # if we run HF first, the cuda initialization will be done and it
29 + # will hurt multiprocessing backend with fork method (the default method).
30 + with vllm_runner(model,
31 + max_model_len=4096,
32 + max_num_seqs=2,
33 + dtype=dtype,
34 + enforce_eager=True) as vllm_model:
35 + vllm_outputs = vllm_model.encode(example_prompts)
36 +
37 + with hf_runner(model, dtype=dtype) as hf_model:
38 + all_inputs = hf_model.get_inputs(example_prompts)
39 +
40 + all_outputs = []
41 + for inputs in all_inputs:
42 + # Based on: https://github.com/TIGER-AI-Lab/VLM2Vec/blob/db3b951bccabba220c1f53ab46a734e50dd2fc08/src/model.py
43 + outputs = hf_model.model(
44 + **hf_model.wrap_device(inputs,
45 + device=hf_model.model.device.type),
46 + return_dict=True,
47 + output_hidden_states=True,
48 + )
49 + last_hidden_state = outputs.hidden_states[-1][0]
50 + reps = last_hidden_state[inputs.attention_mask[0].sum() - 1]
51 + pooled_output = F.normalize(reps, p=2, dim=-1)
52 +
53 + all_outputs.append(pooled_output.tolist())
54 +
55 + hf_outputs = all_outputs
56 +
57 + check_embeddings_close(
58 + embeddings_0_lst=hf_outputs,
59 + embeddings_1_lst=vllm_outputs,
60 + name_0="hf",
61 + name_1="vllm",
62 + )
tests/plugins/vllm_add_dummy_model/vllm_add_dummy_model/my_gemma_embedding.py
+1 -1
@@ -3,7 +3,7 @@ from typing import List, Optional, Union
3 import torch
4
5 from vllm.attention import AttentionMetadata
6 -from vllm.model_executor.models.gemma2_embedding import Gemma2EmbeddingModel
6 +from vllm.model_executor.models.gemma2 import Gemma2EmbeddingModel
7 from vllm.sequence import IntermediateTensors
8
9
vllm/config.py
+10 -1
@@ -237,7 +237,16 @@ class ModelConfig:
237
238 def _verify_embedding_mode(self) -> None:
239 architectures = getattr(self.hf_config, "architectures", [])
240 - self.embedding_mode = ModelRegistry.is_embedding_model(architectures)
240 +
241 + # TODO: Allow the same model architecture to be specified as either
242 + # generation or embedding model
243 + if "Phi3VForCausalLM" in architectures:
244 + # Match both remote and local names
245 + embedding_mode = "/VLM2Vec" in self.model
246 + else:
247 + embedding_mode = ModelRegistry.is_embedding_model(architectures)
248 +
249 + self.embedding_mode = embedding_mode
250
251 def _parse_quant_hf_config(self):
252 quant_cfg = getattr(self.hf_config, "quantization_config", None)
vllm/model_executor/models/gemma2.py
+50 -1
@@ -31,14 +31,16 @@ from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
31 QKVParallelLinear,
32 RowParallelLinear)
33 from vllm.model_executor.layers.logits_processor import LogitsProcessor
34 +from vllm.model_executor.layers.pooler import Pooler, PoolingType
35 from vllm.model_executor.layers.quantization import QuantizationConfig
36 from vllm.model_executor.layers.rotary_embedding import get_rope
37 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
38 from vllm.model_executor.layers.vocab_parallel_embedding import (
39 VocabParallelEmbedding)
40 from vllm.model_executor.model_loader.weight_utils import default_weight_loader
41 +from vllm.model_executor.pooling_metadata import PoolingMetadata
42 from vllm.model_executor.sampling_metadata import SamplingMetadata
41 -from vllm.sequence import IntermediateTensors
43 +from vllm.sequence import IntermediateTensors, PoolerOutput
44
45 from .interfaces import SupportsLoRA, SupportsPP
46 from .utils import (AutoWeightsLoader, is_pp_missing_parameter,
@@ -461,3 +463,50 @@ class Gemma2ForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
463 if self.config.tie_word_embeddings else None),
464 )
465 loader.load_weights(weights)
466 +
467 +
468 +class Gemma2EmbeddingModel(nn.Module, SupportsPP):
469 + """
470 + A model that uses Gemma2 with additional embedding functionalities.
471 +
472 + This class encapsulates the Gemma2Model and provides an interface for
473 + embedding operations and customized pooling functions.
474 +
475 + Attributes:
476 + model: An instance of Gemma2Model used for forward operations.
477 + _pooler: An instance of Pooler used for pooling operations.
478 + """
479 +
480 + def __init__(
481 + self,
482 + **kwargs,
483 + ) -> None:
484 + super().__init__()
485 +
486 + self.model = Gemma2Model(**kwargs)
487 + self._pooler = Pooler(pooling_type=PoolingType.LAST, normalize=True)
488 +
489 + self.make_empty_intermediate_tensors = (
490 + self.model.make_empty_intermediate_tensors)
491 +
492 + def forward(
493 + self,
494 + input_ids: Optional[torch.Tensor],
495 + positions: torch.Tensor,
496 + kv_caches: List[torch.Tensor],
497 + attn_metadata: AttentionMetadata,
498 + intermediate_tensors: Optional[IntermediateTensors] = None,
499 + inputs_embeds: Optional[torch.Tensor] = None,
500 + ) -> Union[torch.Tensor, IntermediateTensors]:
501 + return self.model(input_ids, positions, kv_caches, attn_metadata,
502 + intermediate_tensors, inputs_embeds)
503 +
504 + def pooler(
505 + self,
506 + hidden_states: torch.Tensor,
507 + pooling_metadata: PoolingMetadata,
508 + ) -> Optional[PoolerOutput]:
509 + return self._pooler(hidden_states, pooling_metadata)
510 +
511 + def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
512 + self.model.load_weights(weights)
vllm/model_executor/models/gemma2_embedding.py deleted
-57
@@ -1,57 +0,0 @@
1 -from typing import Iterable, List, Optional, Tuple, Union
2 -
3 -import torch
4 -from torch import nn
5 -
6 -from vllm.attention import AttentionMetadata
7 -from vllm.model_executor.layers.pooler import Pooler, PoolingType
8 -from vllm.model_executor.pooling_metadata import PoolingMetadata
9 -from vllm.sequence import IntermediateTensors, PoolerOutput
10 -
11 -from .gemma2 import Gemma2Model
12 -from .interfaces import SupportsPP
13 -
14 -
15 -class Gemma2EmbeddingModel(nn.Module, SupportsPP):
16 - """A model that uses Gemma2 with additional embedding functionalities.
17 -
18 - This class encapsulates the Gemma2Model and provides an interface for
19 - embedding operations and customized pooling functions.
20 -
21 - Attributes:
22 - model: An instance of Gemma2Model used for forward operations.
23 - _pooler: An instance of Pooler used for pooling operations.
24 - """
25 -
26 - def __init__(
27 - self,
28 - **kwargs,
29 - ) -> None:
30 - super().__init__()
31 - self.model = Gemma2Model(**kwargs)
32 - self._pooler = Pooler(pooling_type=PoolingType.LAST, normalize=True)
33 -
34 - self.make_empty_intermediate_tensors = (
35 - self.model.make_empty_intermediate_tensors)
36 -
37 - def forward(
38 - self,
39 - input_ids: Optional[torch.Tensor],
40 - positions: torch.Tensor,
41 - kv_caches: List[torch.Tensor],
42 - attn_metadata: AttentionMetadata,
43 - intermediate_tensors: Optional[IntermediateTensors] = None,
44 - inputs_embeds: Optional[torch.Tensor] = None,
45 - ) -> Union[torch.Tensor, IntermediateTensors]:
46 - return self.model(input_ids, positions, kv_caches, attn_metadata,
47 - intermediate_tensors, inputs_embeds)
48 -
49 - def pooler(
50 - self,
51 - hidden_states: torch.Tensor,
52 - pooling_metadata: PoolingMetadata,
53 - ) -> Optional[PoolerOutput]:
54 - return self._pooler(hidden_states, pooling_metadata)
55 -
56 - def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
57 - self.model.load_weights(weights)
vllm/model_executor/models/llama.py
+52 -1
@@ -38,6 +38,7 @@ from vllm.model_executor.layers.linear import (MergedColumnParallelLinear,
38 QKVParallelLinear,
39 RowParallelLinear)
40 from vllm.model_executor.layers.logits_processor import LogitsProcessor
41 +from vllm.model_executor.layers.pooler import Pooler, PoolingType
42 from vllm.model_executor.layers.quantization import QuantizationConfig
43 from vllm.model_executor.layers.quantization.compressed_tensors.utils import (
44 get_compressed_tensors_cache_scale)
@@ -47,8 +48,9 @@ from vllm.model_executor.layers.vocab_parallel_embedding import (
48 DEFAULT_VOCAB_PADDING_SIZE, ParallelLMHead, VocabParallelEmbedding)
49 from vllm.model_executor.model_loader.weight_utils import (
50 default_weight_loader, kv_cache_scales_loader, maybe_remap_kv_scale_name)
51 +from vllm.model_executor.pooling_metadata import PoolingMetadata
52 from vllm.model_executor.sampling_metadata import SamplingMetadata
51 -from vllm.sequence import IntermediateTensors
53 +from vllm.sequence import IntermediateTensors, PoolerOutput
54 from vllm.utils import is_hip
55
56 from .interfaces import SupportsLoRA, SupportsPP
@@ -615,3 +617,52 @@ class LlamaForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
617 name = name.replace(item, mapping[item])
618
619 return name, loaded_weight
620 +
621 +
622 +class LlamaEmbeddingModel(nn.Module, SupportsPP):
623 + """
624 + A model that uses Llama with additional embedding functionalities.
625 +
626 + This class encapsulates the LlamaModel and provides an interface for
627 + embedding operations and customized pooling functions.
628 +
629 + Attributes:
630 + model: An instance of LlamaModel used for forward operations.
631 + _pooler: An instance of Pooler used for pooling operations.
632 + """
633 +
634 + def __init__(
635 + self,
636 + **kwargs,
637 + ) -> None:
638 + super().__init__()
639 +
640 + self.model = LlamaModel(**kwargs)
641 + self._pooler = Pooler(pooling_type=PoolingType.LAST, normalize=True)
642 + self.make_empty_intermediate_tensors = (
643 + self.model.make_empty_intermediate_tensors)
644 +
645 + def forward(
646 + self,
647 + input_ids: Optional[torch.Tensor],
648 + positions: torch.Tensor,
649 + kv_caches: List[torch.Tensor],
650 + attn_metadata: AttentionMetadata,
651 + intermediate_tensors: Optional[IntermediateTensors] = None,
652 + inputs_embeds: Optional[torch.Tensor] = None,
653 + ) -> Union[torch.Tensor, IntermediateTensors]:
654 + return self.model(input_ids, positions, kv_caches, attn_metadata,
655 + intermediate_tensors, inputs_embeds)
656 +
657 + def pooler(
658 + self,
659 + hidden_states: torch.Tensor,
660 + pooling_metadata: PoolingMetadata,
661 + ) -> Optional[PoolerOutput]:
662 + return self._pooler(hidden_states, pooling_metadata)
663 +
664 + def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
665 + self.model.load_weights(weights)
666 +
667 + def load_kv_cache_scales(self, quantization_param_path: str) -> None:
668 + self.model.load_kv_cache_scales(quantization_param_path)
vllm/model_executor/models/llama_embedding.py deleted
-59
@@ -1,59 +0,0 @@
1 -from typing import Iterable, List, Optional, Tuple, Union
2 -
3 -import torch
4 -from torch import nn
5 -
6 -from vllm.attention import AttentionMetadata
7 -from vllm.model_executor.layers.pooler import Pooler, PoolingType
8 -from vllm.model_executor.pooling_metadata import PoolingMetadata
9 -from vllm.sequence import IntermediateTensors, PoolerOutput
10 -
11 -from .interfaces import SupportsPP
12 -from .llama import LlamaModel
13 -
14 -
15 -class LlamaEmbeddingModel(nn.Module, SupportsPP):
16 - """A model that uses Llama with additional embedding functionalities.
17 -
18 - This class encapsulates the LlamaModel and provides an interface for
19 - embedding operations and customized pooling functions.
20 -
21 - Attributes:
22 - model: An instance of LlamaModel used for forward operations.
23 - _pooler: An instance of Pooler used for pooling operations.
24 - """
25 -
26 - def __init__(
27 - self,
28 - **kwargs,
29 - ) -> None:
30 - super().__init__()
31 - self.model = LlamaModel(**kwargs)
32 - self._pooler = Pooler(pooling_type=PoolingType.LAST, normalize=True)
33 - self.make_empty_intermediate_tensors = (
34 - self.model.make_empty_intermediate_tensors)
35 -
36 - def forward(
37 - self,
38 - input_ids: Optional[torch.Tensor],
39 - positions: torch.Tensor,
40 - kv_caches: List[torch.Tensor],
41 - attn_metadata: AttentionMetadata,
42 - intermediate_tensors: Optional[IntermediateTensors] = None,
43 - inputs_embeds: Optional[torch.Tensor] = None,
44 - ) -> Union[torch.Tensor, IntermediateTensors]:
45 - return self.model(input_ids, positions, kv_caches, attn_metadata,
46 - intermediate_tensors, inputs_embeds)
47 -
48 - def pooler(
49 - self,
50 - hidden_states: torch.Tensor,
51 - pooling_metadata: PoolingMetadata,
52 - ) -> Optional[PoolerOutput]:
53 - return self._pooler(hidden_states, pooling_metadata)
54 -
55 - def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
56 - self.model.load_weights(weights)
57 -
58 - def load_kv_cache_scales(self, quantization_param_path: str) -> None:
59 - self.model.load_kv_cache_scales(quantization_param_path)
vllm/model_executor/models/phi3v.py
+58 -25
@@ -29,14 +29,18 @@ from vllm.attention import AttentionMetadata
29 from vllm.config import CacheConfig, ModelConfig, MultiModalConfig
30 from vllm.inputs import INPUT_REGISTRY, InputContext, LLMInputs
31 from vllm.logger import init_logger
32 +from vllm.model_executor.layers.pooler import Pooler, PoolingType
33 from vllm.model_executor.layers.quantization import QuantizationConfig
34 from vllm.model_executor.layers.sampler import Sampler, SamplerOutput
35 +from vllm.model_executor.layers.vocab_parallel_embedding import (
36 + VocabParallelEmbedding)
37 from vllm.model_executor.models.clip import CLIPVisionModel
38 from vllm.model_executor.models.llama import LlamaForCausalLM
39 +from vllm.model_executor.pooling_metadata import PoolingMetadata
40 from vllm.model_executor.sampling_metadata import SamplingMetadata
41 from vllm.multimodal import MULTIMODAL_REGISTRY
42 from vllm.multimodal.utils import cached_get_tokenizer, repeat_and_pad_token
39 -from vllm.sequence import IntermediateTensors
43 +from vllm.sequence import IntermediateTensors, PoolerOutput
44 from vllm.utils import is_list_of
45
46 from .clip import dummy_image_for_clip, dummy_seq_data_for_clip
@@ -289,10 +293,6 @@ class Phi3HDImageEmbedding(Phi3ImageEmbeddingBase):
293 dim=2).reshape(num_images, -1, hid_dim)
294 return image_features_hd_newline
295
292 - def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
293 - loader = AutoWeightsLoader(self)
294 - loader.load_weights(weights)
295 -
296
297 # Based on https://huggingface.co/microsoft/Phi-3-vision-128k-instruct/blob/main/image_processing_phi3_v.py#L57
298 def _calc_padded_size(*, width: int, height: int, padding_unit: int = 336):
@@ -385,23 +385,28 @@ def dummy_data_for_phi3v(ctx: InputContext,
385 return seq_data, mm_data
386
387
388 -# Reserve this function to also handle placeholders for additional images
389 -# [ref: PR #5820]
388 @lru_cache
391 -def _get_image_placeholder_token_ids(model_config: ModelConfig,
392 - idx: int) -> List[int]:
389 +def _get_image_placeholder_token_id_candidates(
390 + model_config: ModelConfig,
391 + idx: int,
392 +) -> List[List[int]]:
393 assert idx > 0
394
395 tokenizer = cached_get_tokenizer(model_config.tokenizer)
396
397 + # This is used when the image token is at the start of the string
398 + start_candidate = tokenizer.encode(f"<|image_{idx}|>",
399 + add_special_tokens=False)
400 +
401 + # This is used when the image token is in the middle of the string
402 # We need to get the token for "<", not "▁<"
403 # https://huggingface.co/microsoft/Phi-3-vision-128k-instruct/raw/main/tokenizer.json
404 a_token_id, = tokenizer.encode("a", add_special_tokens=False)
400 - a_token_id_, *image_placeholder_token_ids = tokenizer.encode(
401 - f"a<|image_{idx}|>", add_special_tokens=False)
405 + a_token_id_, *middle_candidate = tokenizer.encode(f"a<|image_{idx}|>",
406 + add_special_tokens=False)
407 assert a_token_id == a_token_id_
408
404 - return image_placeholder_token_ids
409 + return [start_candidate, middle_candidate]
410
411
412 def input_processor_for_phi3v(ctx: InputContext,
@@ -461,16 +466,20 @@ def input_processor_for_phi3v(ctx: InputContext,
466
467 prompt_token_ids = llm_inputs["prompt_token_ids"].copy()
468
464 - # masked place_holder with image token id
469 + print("prompt_token_ids (old)", prompt_token_ids)
470 +
471 + # masked placeholder with image token id
472 for idx in image_idx:
466 - image_token_ids = _get_image_placeholder_token_ids(model_config,
467 - idx=idx)
468 - for i in range(len(prompt_token_ids) - len(image_token_ids) + 1):
469 - if prompt_token_ids[i:i + len(image_token_ids)] == image_token_ids:
470 - prompt_token_ids[i:i + len(image_token_ids)] = [
471 - _IMAGE_TOKEN_ID
472 - ] * len(image_token_ids)
473 - break
473 + candidates = _get_image_placeholder_token_id_candidates(model_config,
474 + idx=idx)
475 +
476 + for candidate in candidates:
477 + for i in range(len(prompt_token_ids) - len(candidate) + 1):
478 + if prompt_token_ids[i:i + len(candidate)] == candidate:
479 + prompt_token_ids[i:i +
480 + len(candidate)] = ([_IMAGE_TOKEN_ID] *
481 + len(candidate))
482 + break
483
484 # merge consecutive tag ids
485 merged_token_ids: List[int] = []
@@ -520,12 +529,23 @@ class Phi3VForCausalLM(nn.Module, SupportsMultiModal, SupportsPP):
529 self.multimodal_config = multimodal_config
530 self.image_token_id = _IMAGE_TOKEN_ID
531
523 - # TODO: Optionally initializes this for supporting embeddings.
532 + self.embed_tokens = VocabParallelEmbedding(
533 + config.vocab_size,
534 + config.hidden_size,
535 + org_num_embeddings=config.vocab_size,
536 + quant_config=quant_config,
537 + )
538 +
539 + # TODO: Optionally initializes this for supporting input embeddings.
540 self.vision_embed_tokens = Phi3HDImageEmbedding(config)
541
542 self.language_model = LlamaForCausalLM(config, cache_config,
543 quant_config)
544
545 + # The same model class supports both language generation and embedding
546 + # because the architecture name is the same
547 + self._pooler = Pooler(pooling_type=PoolingType.LAST, normalize=True)
548 +
549 self.make_empty_intermediate_tensors = (
550 self.language_model.make_empty_intermediate_tensors)
551
@@ -649,8 +669,7 @@ class Phi3VForCausalLM(nn.Module, SupportsMultiModal, SupportsPP):
669
670 if image_input is not None:
671 vision_embeddings = self._process_image_input(image_input)
652 - inputs_embeds = self.language_model.model.get_input_embeddings(
653 - input_ids)
672 + inputs_embeds = self.embed_tokens(input_ids)
673 inputs_embeds = merge_multimodal_embeddings(
674 input_ids, inputs_embeds, vision_embeddings,
675 self.image_token_id)
@@ -682,13 +701,27 @@ class Phi3VForCausalLM(nn.Module, SupportsMultiModal, SupportsPP):
701 ) -> Optional[SamplerOutput]:
702 return self.language_model.sample(logits, sampling_metadata)
703
704 + def pooler(
705 + self,
706 + hidden_states: torch.Tensor,
707 + pooling_metadata: PoolingMetadata,
708 + ) -> Optional[PoolerOutput]:
709 + return self._pooler(hidden_states, pooling_metadata)
710 +
711 def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
712 hf_to_vllm_mapper = WeightsMapper(
713 orig_to_new_prefix={
714 + "model.vision_embed_tokens.wte": "embed_tokens",
715 "model.vision_embed_tokens.": "vision_embed_tokens.",
716 "lm_head.": "language_model.lm_head.",
717 "model.": "language_model.model.",
718 })
719
720 loader = AutoWeightsLoader(self)
694 - loader.load_weights(weights, mapper=hf_to_vllm_mapper)
721 + autoloaded_weights = loader.load_weights(weights,
722 + mapper=hf_to_vllm_mapper)
723 +
724 + # The HF config doesn't specify whether these are tied,
725 + # so we detect it this way
726 + if "embed_tokens" not in autoloaded_weights:
727 + self.embed_tokens = self.language_model.model.embed_tokens
vllm/model_executor/models/registry.py
+5 -2
@@ -86,9 +86,12 @@ _TEXT_GENERATION_MODELS = {
86 }
87
88 _EMBEDDING_MODELS = {
89 - "MistralModel": ("llama_embedding", "LlamaEmbeddingModel"),
89 + # [Text-only]
90 + "Gemma2Model": ("gemma2", "Gemma2EmbeddingModel"),
91 + "MistralModel": ("llama", "LlamaEmbeddingModel"),
92 "Qwen2ForRewardModel": ("qwen2_rm", "Qwen2ForRewardModel"),
91 - "Gemma2Model": ("gemma2_embedding", "Gemma2EmbeddingModel"),
93 + # [Multimodal]
94 + "Phi3VForCausalLM": ("phi3v", "Phi3VForCausalLM"),
95 }
96
97 _MULTIMODAL_MODELS = {
vllm/model_executor/models/utils.py
+14 -9
@@ -124,7 +124,7 @@ class AutoWeightsLoader:
124 base_prefix: str,
125 param: nn.Parameter,
126 weights: Iterable[Tuple[str, torch.Tensor]],
127 - ) -> None:
127 + ) -> Iterable[str]:
128 for weight_name, weight_data in weights:
129 weight_qualname = self._get_qualname(base_prefix, weight_name)
130
@@ -143,12 +143,14 @@ class AutoWeightsLoader:
143 default_weight_loader)
144 weight_loader(param, weight_data)
145
146 + yield weight_qualname
147 +
148 def _load_module(
149 self,
150 base_prefix: str,
151 module: nn.Module,
152 weights: Iterable[Tuple[str, torch.Tensor]],
151 - ) -> None:
153 + ) -> Iterable[str]:
154 if isinstance(module, PPMissingLayer):
155 return
156
@@ -170,14 +172,16 @@ class AutoWeightsLoader:
172 continue
173
174 if child_prefix in child_modules:
173 - self._load_module(prefix, child_modules[child_prefix],
174 - child_weights)
175 + yield from self._load_module(prefix,
176 + child_modules[child_prefix],
177 + child_weights)
178 elif child_prefix in child_params:
176 - self._load_param(prefix, child_params[child_prefix],
177 - child_weights)
179 + yield from self._load_param(prefix, child_params[child_prefix],
180 + child_weights)
181 else:
182 if not self._can_ignore_unexpected(prefix):
180 - msg = f"There is no module or parameter named '{prefix}'"
183 + msg = (f"There is no module or parameter named '{prefix}' "
184 + f"in {type(self.module).__name__}")
185 raise ValueError(msg)
186
187 def load_weights(
@@ -185,11 +189,12 @@ class AutoWeightsLoader:
189 weights: Iterable[Tuple[str, torch.Tensor]],
190 *,
191 mapper: Optional[WeightsMapper] = None,
188 - ) -> None:
192 + ) -> List[str]:
193 if mapper is not None:
194 weights = mapper.apply(weights)
195
192 - self._load_module("", self.module, weights)
196 + autoloaded_weights = list(self._load_module("", self.module, weights))
197 + return autoloaded_weights
198
199
200 def init_vllm_registered_model(