A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 05:30 UTC
[Doc] Improve MM Pooling model documentation (#25966)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Cyrus Leung committed
Oct 1, 2025 at 02:58 UTC
2f652e6cdf09da407e78a60327832f913b32e26e
9 files changed
+290
-98
docs/features/multimodal_inputs.md
+1
-1
@@ -428,7 +428,7 @@ Our OpenAI-compatible server accepts multi-modal data via the [Chat Completions
428
If no fallback is available, an error is raised and you have to provide the chat template manually via the `--chat-template` argument.
429
430
For certain models, we provide alternative chat templates inside <gh-dir:examples>.
431
- For example, VLM2Vec uses <gh-file:examples/template_vlm2vec.jinja> which is different from the default one for Phi-3-Vision.
431
+ For example, VLM2Vec uses <gh-file:examples/template_vlm2vec_phi3v.jinja> which is different from the default one for Phi-3-Vision.
432
433
### Image Inputs
434
docs/models/supported_models.md
+25
-3
@@ -626,7 +626,29 @@ See [this page](../features/multimodal_inputs.md) on how to pass multi-modal inp
626
For hybrid-only models such as Llama-4, Step3 and Mistral-3, a text-only mode can be enabled by setting all supported multimodal modalities to 0 (e.g, `--limit-mm-per-prompt '{"image":0}`) so that their multimodal modules will not be loaded to free up more GPU memory for KV cache.
627
628
!!! note
629
- vLLM currently only supports adding LoRA to the language backbone of multimodal models.
629
+ vLLM currently only supports dynamic LoRA adapters on the language backbone of multimodal models.
630
+ If you wish to use a model with LoRA in the multi-modal encoder,
631
+ please merge the weights into the base model first before running it in vLLM like a regular model.
632
+
633
+ ```python
634
+ from peft import PeftConfig, PeftModel
635
+ from transformers import AutoModelForImageTextToText, AutoProcessor
636
+
637
+ def merge_and_save(model_id: str, output_dir: str):
638
+ base_model = AutoModelForImageTextToText.from_pretrained(model_id)
639
+ lora_model = PeftModel.from_pretrained(
640
+ base_model,
641
+ model_id,
642
+ config=PeftConfig.from_pretrained(model_id),
643
+ )
644
+ model = lora_model.merge_and_unload().to(dtype=base_model.dtype)
645
+ model._hf_peft_config_loaded = False # Needed to save the merged model
646
+
647
+ processor = AutoProcessor.from_pretrained(model_id)
648
+
649
+ model.save_pretrained(output_dir)
650
+ processor.save_pretrained(output_dir)
651
+ ```
652
653
### Generative Models
654
@@ -805,8 +827,8 @@ The following table lists those that are tested in vLLM.
827
828
| Architecture | Models | Inputs | Example HF Models | [LoRA](../features/lora.md) | [PP](../serving/parallelism_scaling.md) | [V1](gh-issue:8779) |
829
|--------------|--------|--------|-------------------|----------------------|---------------------------|---------------------|
808
-| `LlavaNextForConditionalGeneration`<sup>C</sup> | LLaVA-NeXT-based | T / I | `royokong/e5-v` | | | |
809
-| `Phi3VForCausalLM`<sup>C</sup> | Phi-3-Vision-based | T + I | `TIGER-Lab/VLM2Vec-Full` | 🚧 | ✅︎ | |
830
+| `LlavaNextForConditionalGeneration`<sup>C</sup> | LLaVA-NeXT-based | T / I | `royokong/e5-v` | | ✅︎ | ✅︎ |
831
+| `Phi3VForCausalLM`<sup>C</sup> | Phi-3-Vision-based | T + I | `TIGER-Lab/VLM2Vec-Full` | | ✅︎ | ✅︎ |
832
| `*ForConditionalGeneration`<sup>C</sup>, `*ForCausalLM`<sup>C</sup>, etc. | Generative models | \* | N/A | \* | \* | \* |
833
834
<sup>C</sup> Automatically converted into an embedding model via `--convert embed`. ([details](./pooling_models.md#model-conversion))
docs/serving/openai_compatible_server.md
+41
-17
@@ -236,10 +236,32 @@ The following extra parameters are supported:
236
Our Embeddings API is compatible with [OpenAI's Embeddings API](https://platform.openai.com/docs/api-reference/embeddings);
237
you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.
238
239
+Code example: <gh-file:examples/online_serving/pooling/openai_embedding_client.py>
240
+
241
If the model has a [chat template][chat-template], you can replace `inputs` with a list of `messages` (same schema as [Chat API][chat-api])
240
-which will be treated as a single prompt to the model.
242
+which will be treated as a single prompt to the model. Here is a convenience function for calling the API while retaining OpenAI's type annotations:
243
242
-Code example: <gh-file:examples/online_serving/pooling/openai_embedding_client.py>
244
+??? code
245
+
246
+ ```python
247
+ from openai import OpenAI
248
+ from openai._types import NOT_GIVEN, NotGiven
249
+ from openai.types.chat import ChatCompletionMessageParam
250
+ from openai.types.create_embedding_response import CreateEmbeddingResponse
251
+
252
+ def create_chat_embeddings(
253
+ client: OpenAI,
254
+ *,
255
+ messages: list[ChatCompletionMessageParam],
256
+ model: str,
257
+ encoding_format: Union[Literal["base64", "float"], NotGiven] = NOT_GIVEN,
258
+ ) -> CreateEmbeddingResponse:
259
+ return client.post(
260
+ "/embeddings",
261
+ cast_to=CreateEmbeddingResponse,
262
+ body={"messages": messages, "model": model, "encoding_format": encoding_format},
263
+ )
264
+ ```
265
266
#### Multi-modal inputs
267
@@ -254,7 +276,7 @@ and passing a list of `messages` in the request. Refer to the examples below for
276
vllm serve TIGER-Lab/VLM2Vec-Full --runner pooling \
277
--trust-remote-code \
278
--max-model-len 4096 \
257
- --chat-template examples/template_vlm2vec.jinja
279
+ --chat-template examples/template_vlm2vec_phi3v.jinja
280
```
281
282
!!! important
@@ -262,34 +284,36 @@ and passing a list of `messages` in the request. Refer to the examples below for
284
to run this model in embedding mode instead of text generation mode.
285
286
The custom chat template is completely different from the original one for this model,
265
- and can be found here: <gh-file:examples/template_vlm2vec.jinja>
287
+ and can be found here: <gh-file:examples/template_vlm2vec_phi3v.jinja>
288
289
Since the request schema is not defined by OpenAI client, we post a request to the server using the lower-level `requests` library:
290
291
??? code
292
293
```python
272
- import requests
273
-
294
+ from openai import OpenAI
295
+ client = OpenAI(
296
+ base_url="http://localhost:8000/v1",
297
+ api_key="EMPTY",
298
+ )
299
image_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
300
276
- response = requests.post(
277
- "http://localhost:8000/v1/embeddings",
278
- json={
279
- "model": "TIGER-Lab/VLM2Vec-Full",
280
- "messages": [{
301
+ response = create_chat_embeddings(
302
+ client,
303
+ model="TIGER-Lab/VLM2Vec-Full",
304
+ messages=[
305
+ {
306
"role": "user",
307
"content": [
308
{"type": "image_url", "image_url": {"url": image_url}},
309
{"type": "text", "text": "Represent the given image."},
310
],
286
- }],
287
- "encoding_format": "float",
288
- },
311
+ }
312
+ ],
313
+ encoding_format="float",
314
)
290
- response.raise_for_status()
291
- response_json = response.json()
292
- print("Embedding output:", response_json["data"][0]["embedding"])
315
+
316
+ print("Image embedding output:", response.data[0].embedding)
317
```
318
319
=== "DSE-Qwen2-MRL"
examples/offline_inference/vision_language_pooling.py
+77
-8
@@ -10,6 +10,7 @@ on HuggingFace model repository.
10
11
from argparse import Namespace
12
from dataclasses import asdict
13
+from pathlib import Path
14
from typing import Literal, NamedTuple, Optional, TypedDict, Union, get_args
15
16
from PIL.Image import Image
@@ -19,6 +20,9 @@ from vllm.entrypoints.score_utils import ScoreMultiModalParam
20
from vllm.multimodal.utils import fetch_image
21
from vllm.utils import FlexibleArgumentParser
22
23
+ROOT_DIR = Path(__file__).parent.parent.parent
24
+EXAMPLES_DIR = ROOT_DIR / "examples"
25
+
26
27
class TextQuery(TypedDict):
28
modality: Literal["text"]
@@ -82,23 +86,27 @@ def run_e5_v(query: Query) -> ModelRequestData:
86
)
87
88
85
-def run_vlm2vec(query: Query) -> ModelRequestData:
89
+def _get_vlm2vec_prompt_image(query: Query, image_token: str):
90
if query["modality"] == "text":
91
text = query["text"]
92
prompt = f"Find me an everyday image that matches the given caption: {text}" # noqa: E501
93
image = None
94
elif query["modality"] == "image":
91
- prompt = "<|image_1|> Find a day-to-day image that looks similar to the provided image." # noqa: E501
95
+ prompt = f"{image_token} Find a day-to-day image that looks similar to the provided image." # noqa: E501
96
image = query["image"]
97
elif query["modality"] == "text+image":
98
text = query["text"]
95
- prompt = (
96
- f"<|image_1|> Represent the given image with the following question: {text}" # noqa: E501
97
- )
99
+ prompt = f"{image_token} Represent the given image with the following question: {text}" # noqa: E501
100
image = query["image"]
101
else:
102
modality = query["modality"]
101
- raise ValueError(f"Unsupported query modality: '{modality}'")
103
+ raise ValueError(f"Unsupported query modality: {modality!r}")
104
+
105
+ return prompt, image
106
+
107
+
108
+def run_vlm2vec_phi3v(query: Query) -> ModelRequestData:
109
+ prompt, image = _get_vlm2vec_prompt_image(query, "<|image_1|>")
110
111
engine_args = EngineArgs(
112
model="TIGER-Lab/VLM2Vec-Full",
@@ -116,6 +124,66 @@ def run_vlm2vec(query: Query) -> ModelRequestData:
124
)
125
126
127
+def run_vlm2vec_qwen2vl(query: Query) -> ModelRequestData:
128
+ # vLLM does not support LoRA adapters on multi-modal encoder,
129
+ # so we merge the weights first
130
+ from huggingface_hub.constants import HF_HUB_CACHE
131
+ from peft import PeftConfig, PeftModel
132
+ from transformers import AutoModelForImageTextToText, AutoProcessor
133
+
134
+ from vllm.entrypoints.chat_utils import load_chat_template
135
+
136
+ model_id = "TIGER-Lab/VLM2Vec-Qwen2VL-2B"
137
+
138
+ base_model = AutoModelForImageTextToText.from_pretrained(model_id)
139
+ lora_model = PeftModel.from_pretrained(
140
+ base_model,
141
+ model_id,
142
+ config=PeftConfig.from_pretrained(model_id),
143
+ )
144
+ model = lora_model.merge_and_unload().to(dtype=base_model.dtype)
145
+ model._hf_peft_config_loaded = False # Needed to save the merged model
146
+
147
+ processor = AutoProcessor.from_pretrained(
148
+ model_id,
149
+ # `min_pixels` and `max_pixels` are deprecated
150
+ size={"shortest_edge": 3136, "longest_edge": 12845056},
151
+ )
152
+ processor.chat_template = load_chat_template(
153
+ # The original chat template is not correct
154
+ EXAMPLES_DIR / "template_vlm2vec_qwen2vl.jinja",
155
+ )
156
+
157
+ merged_path = str(
158
+ Path(HF_HUB_CACHE) / ("models--" + model_id.replace("/", "--") + "-vllm")
159
+ )
160
+ print(f"Saving merged model to {merged_path}...")
161
+ print(
162
+ "NOTE: This directory is not tracked by `huggingface_hub` "
163
+ "so you have to delete this manually if you don't want it anymore."
164
+ )
165
+ model.save_pretrained(merged_path)
166
+ processor.save_pretrained(merged_path)
167
+ print("Done!")
168
+
169
+ prompt, image = _get_vlm2vec_prompt_image(query, "<|image_pad|>")
170
+
171
+ engine_args = EngineArgs(
172
+ model=merged_path,
173
+ runner="pooling",
174
+ max_model_len=4096,
175
+ trust_remote_code=True,
176
+ mm_processor_kwargs={"num_crops": 4},
177
+ limit_mm_per_prompt={"image": 1},
178
+ )
179
+
180
+ return ModelRequestData(
181
+ engine_args=engine_args,
182
+ prompt=prompt,
183
+ image=image,
184
+ )
185
+
186
+
187
def run_jinavl_reranker(query: Query) -> ModelRequestData:
188
if query["modality"] != "text+images":
189
raise ValueError(f"Unsupported query modality: '{query['modality']}'")
@@ -232,7 +300,8 @@ def run_score(model: str, modality: QueryModality, seed: Optional[int]):
300
301
model_example_map = {
302
"e5_v": run_e5_v,
235
- "vlm2vec": run_vlm2vec,
303
+ "vlm2vec_phi3v": run_vlm2vec_phi3v,
304
+ "vlm2vec_qwen2vl": run_vlm2vec_qwen2vl,
305
"jinavl_reranker": run_jinavl_reranker,
306
}
307
@@ -246,7 +315,7 @@ def parse_args():
315
"--model-name",
316
"-m",
317
type=str,
249
- default="vlm2vec",
318
+ default="vlm2vec_phi3v",
319
choices=model_example_map.keys(),
320
help="The name of the embedding model.",
321
)
examples/online_serving/pooling/openai_chat_embedding_client_for_multimodal.py
+128
-67
@@ -4,69 +4,137 @@
4
"""Example Python client for multimodal embedding API using vLLM API server
5
NOTE:
6
start a supported multimodal embeddings model server with `vllm serve`, e.g.
7
- vllm serve TIGER-Lab/VLM2Vec-Full --runner pooling --trust_remote_code --max_model_len=1024
7
+ vllm serve TIGER-Lab/VLM2Vec-Full \
8
+ --runner pooling \
9
+ --trust-remote-code \
10
+ --max-model-len 4096 \
11
+ --chat-template examples/template_vlm2vec_phi3v.jinja
12
"""
13
14
import argparse
15
import base64
16
import io
17
+from typing import Literal, Union
18
14
-import requests
19
+from openai import OpenAI
20
+from openai._types import NOT_GIVEN, NotGiven
21
+from openai.types.chat import ChatCompletionMessageParam
22
+from openai.types.create_embedding_response import CreateEmbeddingResponse
23
from PIL import Image
24
25
+# Modify OpenAI's API key and API base to use vLLM's API server.
26
+openai_api_key = "EMPTY"
27
+openai_api_base = "http://localhost:8000/v1"
28
+
29
image_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
30
31
20
-def vlm2vec():
21
- response = requests.post(
22
- "http://localhost:8000/v1/embeddings",
23
- json={
24
- "model": "TIGER-Lab/VLM2Vec-Full",
25
- "messages": [
26
- {
27
- "role": "user",
28
- "content": [
29
- {"type": "image_url", "image_url": {"url": image_url}},
30
- {"type": "text", "text": "Represent the given image."},
31
- ],
32
- }
33
- ],
34
- "encoding_format": "float",
35
- },
32
+def create_chat_embeddings(
33
+ client: OpenAI,
34
+ *,
35
+ messages: list[ChatCompletionMessageParam],
36
+ model: str,
37
+ encoding_format: Union[Literal["base64", "float"], NotGiven] = NOT_GIVEN,
38
+) -> CreateEmbeddingResponse:
39
+ """
40
+ Convenience function for accessing vLLM's Chat Embeddings API,
41
+ which is an extension of OpenAI's existing Embeddings API.
42
+ """
43
+ return client.post(
44
+ "/embeddings",
45
+ cast_to=CreateEmbeddingResponse,
46
+ body={"messages": messages, "model": model, "encoding_format": encoding_format},
47
+ )
48
+
49
+
50
+def run_vlm2vec(client: OpenAI, model: str):
51
+ response = create_chat_embeddings(
52
+ client,
53
+ messages=[
54
+ {
55
+ "role": "user",
56
+ "content": [
57
+ {"type": "image_url", "image_url": {"url": image_url}},
58
+ {"type": "text", "text": "Represent the given image."},
59
+ ],
60
+ }
61
+ ],
62
+ model=model,
63
+ encoding_format="float",
64
+ )
65
+
66
+ print("Image embedding output:", response.data[0].embedding)
67
+
68
+ response = create_chat_embeddings(
69
+ client,
70
+ messages=[
71
+ {
72
+ "role": "user",
73
+ "content": [
74
+ {"type": "image_url", "image_url": {"url": image_url}},
75
+ {
76
+ "type": "text",
77
+ "text": "Represent the given image with the following question: What is in the image.",
78
+ },
79
+ ],
80
+ }
81
+ ],
82
+ model=model,
83
+ encoding_format="float",
84
+ )
85
+
86
+ print("Image+Text embedding output:", response.data[0].embedding)
87
+
88
+ response = create_chat_embeddings(
89
+ client,
90
+ messages=[
91
+ {
92
+ "role": "user",
93
+ "content": [
94
+ {"type": "text", "text": "A cat and a dog"},
95
+ ],
96
+ }
97
+ ],
98
+ model=model,
99
+ encoding_format="float",
100
)
37
- response.raise_for_status()
38
- response_json = response.json()
101
40
- print("Embedding output:", response_json["data"][0]["embedding"])
102
+ print("Text embedding output:", response.data[0].embedding)
103
104
43
-def dse_qwen2_vl(inp: dict):
44
- # Embedding an Image
45
- if inp["type"] == "image":
46
- messages = [
105
+def run_dse_qwen2_vl(client: OpenAI, model: str):
106
+ response = create_chat_embeddings(
107
+ client,
108
+ messages=[
109
{
110
"role": "user",
111
"content": [
112
{
113
"type": "image_url",
114
"image_url": {
53
- "url": inp["image_url"],
115
+ "url": image_url,
116
},
117
},
118
{"type": "text", "text": "What is shown in this image?"},
119
],
120
}
59
- ]
60
- # Embedding a Text Query
61
- else:
62
- # MrLight/dse-qwen2-2b-mrl-v1 requires a placeholder image
63
- # of the minimum input size
64
- buffer = io.BytesIO()
65
- image_placeholder = Image.new("RGB", (56, 56))
66
- image_placeholder.save(buffer, "png")
67
- buffer.seek(0)
68
- image_placeholder = base64.b64encode(buffer.read()).decode("utf-8")
69
- messages = [
121
+ ],
122
+ model=model,
123
+ encoding_format="float",
124
+ )
125
+
126
+ print("Image embedding output:", response.data[0].embedding)
127
+
128
+ # MrLight/dse-qwen2-2b-mrl-v1 requires a placeholder image
129
+ # of the minimum input size
130
+ buffer = io.BytesIO()
131
+ image_placeholder = Image.new("RGB", (56, 56))
132
+ image_placeholder.save(buffer, "png")
133
+ buffer.seek(0)
134
+ image_placeholder = base64.b64encode(buffer.read()).decode("utf-8")
135
+ response = create_chat_embeddings(
136
+ client,
137
+ messages=[
138
{
139
"role": "user",
140
"content": [
@@ -76,23 +144,21 @@ def dse_qwen2_vl(inp: dict):
144
"url": f"data:image/jpeg;base64,{image_placeholder}",
145
},
146
},
79
- {"type": "text", "text": f"Query: {inp['content']}"},
147
+ {"type": "text", "text": "Query: What is the weather like today?"},
148
],
149
}
82
- ]
83
-
84
- response = requests.post(
85
- "http://localhost:8000/v1/embeddings",
86
- json={
87
- "model": "MrLight/dse-qwen2-2b-mrl-v1",
88
- "messages": messages,
89
- "encoding_format": "float",
90
- },
150
+ ],
151
+ model=model,
152
+ encoding_format="float",
153
)
92
- response.raise_for_status()
93
- response_json = response.json()
154
95
- print("Embedding output:", response_json["data"][0]["embedding"])
155
+ print("Text embedding output:", response.data[0].embedding)
156
+
157
+
158
+model_example_map = {
159
+ "vlm2vec": run_vlm2vec,
160
+ "dse_qwen2_vl": run_dse_qwen2_vl,
161
+}
162
163
164
def parse_args():
@@ -103,29 +169,24 @@ def parse_args():
169
parser.add_argument(
170
"--model",
171
type=str,
106
- choices=["vlm2vec", "dse_qwen2_vl"],
172
+ choices=model_example_map.keys(),
173
required=True,
108
- help="Which model to call.",
174
+ help="The name of the embedding model.",
175
)
176
return parser.parse_args()
177
178
179
def main(args):
114
- if args.model == "vlm2vec":
115
- vlm2vec()
116
- elif args.model == "dse_qwen2_vl":
117
- dse_qwen2_vl(
118
- {
119
- "type": "image",
120
- "image_url": image_url,
121
- }
122
- )
123
- dse_qwen2_vl(
124
- {
125
- "type": "text",
126
- "content": "What is the weather like today?",
127
- }
128
- )
180
+ client = OpenAI(
181
+ # defaults to os.environ.get("OPENAI_API_KEY")
182
+ api_key=openai_api_key,
183
+ base_url=openai_api_base,
184
+ )
185
+
186
+ models = client.models.list()
187
+ model_id = models.data[0].id
188
+
189
+ model_example_map[args.model](client, model_id)
190
191
192
if __name__ == "__main__":
examples/template_vlm2vec_phi3v.jinja
renamed
examples/template_vlm2vec_qwen2vl.jinja
new
+15
@@ -0,0 +1,15 @@
1
+{%- if messages | length > 1 -%}
2
+ {{ raise_exception('Embedding models should only embed one message at a time') }}
3
+{%- endif -%}
4
+
5
+{% set vars = namespace(parts=[]) %}
6
+{%- for message in messages -%}
7
+ {%- for content in message['content'] -%}
8
+ {%- if content['type'] == 'text' -%}
9
+ {%- set vars.parts = vars.parts + [content['text']] %}
10
+ {%- elif content['type'] == 'image' -%}
11
+ {%- set vars.parts = vars.parts + ['<|image_pad|>'] %}
12
+ {%- endif -%}
13
+ {%- endfor -%}
14
+{%- endfor -%}
15
+{{ vars.parts | join(' ') }}
tests/entrypoints/pooling/openai/test_vision_embedding.py
+1
-1
@@ -14,7 +14,7 @@ from vllm.multimodal.utils import encode_image_base64, fetch_image
14
MODEL_NAME = "TIGER-Lab/VLM2Vec-Full"
15
MAXIMUM_IMAGES = 2
16
17
-vlm2vec_jinja_path = VLLM_PATH / "examples/template_vlm2vec.jinja"
17
+vlm2vec_jinja_path = VLLM_PATH / "examples/template_vlm2vec_phi3v.jinja"
18
assert vlm2vec_jinja_path.exists()
19
20
# Test different image extensions (JPG/PNG) and formats (gray/RGB/RGBA)
tests/entrypoints/test_chat_utils.py
+2
-1
@@ -2468,7 +2468,8 @@ def test_resolve_content_format_fallbacks(model, expected_format):
2468
("template_falcon.jinja", "string"),
2469
("template_inkbot.jinja", "string"),
2470
("template_teleflm.jinja", "string"),
2471
- ("template_vlm2vec.jinja", "openai"),
2471
+ ("template_vlm2vec_phi3v.jinja", "openai"),
2472
+ ("template_vlm2vec_qwen2vl.jinja", "openai"),
2473
("tool_chat_template_granite_20b_fc.jinja", "string"),
2474
("tool_chat_template_hermes.jinja", "string"),
2475
("tool_chat_template_internlm2_tool.jinja", "string"),