A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 04:30 UTC
[Frontend] Online Pooling API (#11457)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Cyrus Leung committed
Dec 24, 2024 at 17:54 UTC
9edca6bf8fa81e2dc678be68e9cdcede572947c1
15 files changed
+809
-157
docs/source/models/generative_models.md
+3
-15
@@ -120,19 +120,7 @@ outputs = llm.chat(conversation, chat_template=custom_template)
120
121
## Online Inference
122
123
-Our [OpenAI Compatible Server](../serving/openai_compatible_server) can be used for online inference.
124
-Please click on the above link for more details on how to launch the server.
123
+Our [OpenAI Compatible Server](../serving/openai_compatible_server) provides endpoints that correspond to the offline APIs:
124
126
-### Completions API
127
-
128
-Our Completions API is similar to `LLM.generate` but only accepts text.
129
-It is compatible with [OpenAI Completions API](https://platform.openai.com/docs/api-reference/completions)
130
-so that you can use OpenAI client to interact with it.
131
-A code example can be found in [examples/openai_completion_client.py](https://github.com/vllm-project/vllm/blob/main/examples/openai_completion_client.py).
132
-
133
-### Chat API
134
-
135
-Our Chat API is similar to `LLM.chat`, accepting both text and [multi-modal inputs](#multimodal-inputs).
136
-It is compatible with [OpenAI Chat Completions API](https://platform.openai.com/docs/api-reference/chat)
137
-so that you can use OpenAI client to interact with it.
138
-A code example can be found in [examples/openai_chat_completion_client.py](https://github.com/vllm-project/vllm/blob/main/examples/openai_chat_completion_client.py).
125
+- [Completions API](#completions-api) is similar to `LLM.generate` but only accepts text.
126
+- [Chat API](#chat-api) is similar to `LLM.chat`, accepting both text and [multi-modal inputs](#multimodal-inputs) for models with a chat template.
docs/source/models/pooling_models.md
+4
-18
@@ -106,22 +106,8 @@ A code example can be found in [examples/offline_inference_scoring.py](https://g
106
107
## Online Inference
108
109
-Our [OpenAI Compatible Server](../serving/openai_compatible_server.md) can be used for online inference.
110
-Please click on the above link for more details on how to launch the server.
109
+Our [OpenAI Compatible Server](../serving/openai_compatible_server.md) provides endpoints that correspond to the offline APIs:
110
112
-### Embeddings API
113
-
114
-Our Embeddings API is similar to `LLM.embed`, accepting both text and [multi-modal inputs](#multimodal-inputs).
115
-
116
-The text-only API is compatible with [OpenAI Embeddings API](https://platform.openai.com/docs/api-reference/embeddings)
117
-so that you can use OpenAI client to interact with it.
118
-A code example can be found in [examples/openai_embedding_client.py](https://github.com/vllm-project/vllm/blob/main/examples/openai_embedding_client.py).
119
-
120
-The multi-modal API is an extension of the [OpenAI Embeddings API](https://platform.openai.com/docs/api-reference/embeddings)
121
-that incorporates [OpenAI Chat Completions API](https://platform.openai.com/docs/api-reference/chat),
122
-so it is not part of the OpenAI standard. Please see [](#multimodal-inputs) for more details on how to use it.
123
-
124
-### Score API
125
-
126
-Our Score API is similar to `LLM.score`.
127
-Please see [this page](#score-api) for more details on how to use it.
111
+- [Pooling API](#pooling-api) is similar to `LLM.encode`, being applicable to all types of pooling models.
112
+- [Embeddings API](#embeddings-api) is similar to `LLM.embed`, accepting both text and [multi-modal inputs](#multimodal-inputs) for embedding models.
113
+- [Score API](#score-api) is similar to `LLM.score` for cross-encoder models.
docs/source/serving/openai_compatible_server.md
+40
-8
@@ -42,6 +42,8 @@ In addition, we have the following custom APIs:
42
43
- [Tokenizer API](#tokenizer-api) (`/tokenize`, `/detokenize`)
44
- Applicable to any model with a tokenizer.
45
+- [Pooling API](#pooling-api) (`/pooling`)
46
+ - Applicable to all [pooling models](../models/pooling_models.md).
47
- [Score API](#score-api) (`/score`)
48
- Only applicable to [cross-encoder models](../models/pooling_models.md) (`--task score`).
49
@@ -179,7 +181,12 @@ The order of priorities is `command line > config file values > defaults`.
181
(completions-api)=
182
### Completions API
183
182
-Refer to [OpenAI's API reference](https://platform.openai.com/docs/api-reference/completions) for more details.
184
+Our Completions API is compatible with [OpenAI's Completions API](https://platform.openai.com/docs/api-reference/completions);
185
+you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.
186
+
187
+#### Code example
188
+
189
+See [examples/openai_completion_client.py](https://github.com/vllm-project/vllm/blob/main/examples/openai_completion_client.py).
190
191
#### Extra parameters
192
@@ -200,15 +207,20 @@ The following extra parameters are supported:
207
```
208
209
(chat-api)=
203
-### Chat Completions API
210
+### Chat API
211
205
-Refer to [OpenAI's API reference](https://platform.openai.com/docs/api-reference/chat) for more details.
212
+Our Chat API is compatible with [OpenAI's Chat Completions API](https://platform.openai.com/docs/api-reference/chat);
213
+you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.
214
215
We support both [Vision](https://platform.openai.com/docs/guides/vision)- and
216
[Audio](https://platform.openai.com/docs/guides/audio?audio-generation-quickstart-example=audio-in)-related parameters;
217
see our [Multimodal Inputs](../usage/multimodal_inputs.md) guide for more information.
218
- *Note: `image_url.detail` parameter is not supported.*
219
220
+#### Code example
221
+
222
+See [examples/openai_chat_completion_client.py](https://github.com/vllm-project/vllm/blob/main/examples/openai_chat_completion_client.py).
223
+
224
#### Extra parameters
225
226
The following [sampling parameters (click through to see documentation)](../dev/sampling_params.md) are supported.
@@ -230,15 +242,20 @@ The following extra parameters are supported:
242
(embeddings-api)=
243
### Embeddings API
244
233
-Refer to [OpenAI's API reference](https://platform.openai.com/docs/api-reference/embeddings) for more details.
245
+Our Embeddings API is compatible with [OpenAI's Embeddings API](https://platform.openai.com/docs/api-reference/embeddings);
246
+you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.
247
235
-If the model has a [chat template](#chat-template), you can replace `inputs` with a list of `messages` (same schema as [Chat Completions API](#chat-api))
248
+If the model has a [chat template](#chat-template), you can replace `inputs` with a list of `messages` (same schema as [Chat API](#chat-api))
249
which will be treated as a single prompt to the model.
250
251
```{tip}
239
-This enables multi-modal inputs to be passed to embedding models, see [this page](../usage/multimodal_inputs.md) for details.
252
+This enables multi-modal inputs to be passed to embedding models, see [this page](#multimodal-inputs) for details.
253
```
254
255
+#### Code example
256
+
257
+See [examples/openai_embedding_client.py](https://github.com/vllm-project/vllm/blob/main/examples/openai_embedding_client.py).
258
+
259
#### Extra parameters
260
261
The following [pooling parameters (click through to see documentation)](../dev/pooling_params.md) are supported.
@@ -268,20 +285,35 @@ For chat-like input (i.e. if `messages` is passed), these extra parameters are s
285
(tokenizer-api)=
286
### Tokenizer API
287
271
-The Tokenizer API is a simple wrapper over [HuggingFace-style tokenizers](https://huggingface.co/docs/transformers/en/main_classes/tokenizer).
288
+Our Tokenizer API is a simple wrapper over [HuggingFace-style tokenizers](https://huggingface.co/docs/transformers/en/main_classes/tokenizer).
289
It consists of two endpoints:
290
291
- `/tokenize` corresponds to calling `tokenizer.encode()`.
292
- `/detokenize` corresponds to calling `tokenizer.decode()`.
293
294
+(pooling-api)=
295
+### Pooling API
296
+
297
+Our Pooling API encodes input prompts using a [pooling model](../models/pooling_models.md) and returns the corresponding hidden states.
298
+
299
+The input format is the same as [Embeddings API](#embeddings-api), but the output data can contain an arbitrary nested list, not just a 1-D list of floats.
300
+
301
+#### Code example
302
+
303
+See [examples/openai_pooling_client.py](https://github.com/vllm-project/vllm/blob/main/examples/openai_pooling_client.py).
304
+
305
(score-api)=
306
### Score API
307
280
-The Score API applies a cross-encoder model to predict scores for sentence pairs.
308
+Our Score API applies a cross-encoder model to predict scores for sentence pairs.
309
Usually, the score for a sentence pair refers to the similarity between two sentences, on a scale of 0 to 1.
310
311
You can find the documentation for these kind of models at [sbert.net](https://www.sbert.net/docs/package_reference/cross_encoder/cross_encoder.html).
312
313
+#### Code example
314
+
315
+See [examples/openai_cross_encoder_score.py](https://github.com/vllm-project/vllm/blob/main/examples/openai_cross_encoder_score.py).
316
+
317
#### Single inference
318
319
You can pass a string to both `text_1` and `text_2`, forming a single sentence pair.
examples/openai_cross_encoder_score.py
+1
-1
@@ -20,9 +20,9 @@ if __name__ == "__main__":
20
parser.add_argument("--host", type=str, default="localhost")
21
parser.add_argument("--port", type=int, default=8000)
22
parser.add_argument("--model", type=str, default="BAAI/bge-reranker-v2-m3")
23
+
24
args = parser.parse_args()
25
api_url = f"http://{args.host}:{args.port}/score"
25
-
26
model_name = args.model
27
28
text_1 = "What is the capital of Brazil?"
examples/openai_pooling_client.py
new
+51
@@ -0,0 +1,51 @@
1
+"""
2
+Example online usage of Pooling API.
3
+
4
+Run `vllm serve <model> --task <embed|classify|reward|score>`
5
+to start up the server in vLLM.
6
+"""
7
+import argparse
8
+import pprint
9
+
10
+import requests
11
+
12
+
13
+def post_http_request(prompt: dict, api_url: str) -> requests.Response:
14
+ headers = {"User-Agent": "Test Client"}
15
+ response = requests.post(api_url, headers=headers, json=prompt)
16
+ return response
17
+
18
+
19
+if __name__ == "__main__":
20
+ parser = argparse.ArgumentParser()
21
+ parser.add_argument("--host", type=str, default="localhost")
22
+ parser.add_argument("--port", type=int, default=8000)
23
+ parser.add_argument("--model",
24
+ type=str,
25
+ default="jason9693/Qwen2.5-1.5B-apeach")
26
+
27
+ args = parser.parse_args()
28
+ api_url = f"http://{args.host}:{args.port}/pooling"
29
+ model_name = args.model
30
+
31
+ # Input like Completions API
32
+ prompt = {"model": model_name, "input": "vLLM is great!"}
33
+ pooling_response = post_http_request(prompt=prompt, api_url=api_url)
34
+ print("Pooling Response:")
35
+ pprint.pprint(pooling_response.json())
36
+
37
+ # Input like Chat API
38
+ prompt = {
39
+ "model":
40
+ model_name,
41
+ "messages": [{
42
+ "role": "user",
43
+ "content": [{
44
+ "type": "text",
45
+ "text": "vLLM is great!"
46
+ }],
47
+ }]
48
+ }
49
+ pooling_response = post_http_request(prompt=prompt, api_url=api_url)
50
+ print("Pooling Response:")
51
+ pprint.pprint(pooling_response.json())
tests/entrypoints/openai/test_embedding.py
+46
-22
@@ -6,6 +6,7 @@ import pytest
6
import pytest_asyncio
7
import requests
8
9
+from vllm.entrypoints.openai.protocol import EmbeddingResponse
10
from vllm.transformers_utils.tokenizer import get_tokenizer
11
12
from ...utils import RemoteOpenAIServer
@@ -17,6 +18,8 @@ DUMMY_CHAT_TEMPLATE = """{% for message in messages %}{{message['role'] + ': ' +
18
@pytest.fixture(scope="module")
19
def server():
20
args = [
21
+ "--task",
22
+ "embed",
23
# use half precision for speed and memory savings in CI environment
24
"--dtype",
25
"bfloat16",
@@ -45,11 +48,14 @@ async def test_single_embedding(client: openai.AsyncOpenAI, model_name: str):
48
]
49
50
# test single embedding
48
- embeddings = await client.embeddings.create(
51
+ embedding_response = await client.embeddings.create(
52
model=model_name,
53
input=input_texts,
54
encoding_format="float",
55
)
56
+ embeddings = EmbeddingResponse.model_validate(
57
+ embedding_response.model_dump(mode="json"))
58
+
59
assert embeddings.id is not None
60
assert len(embeddings.data) == 1
61
assert len(embeddings.data[0].embedding) == 4096
@@ -59,11 +65,14 @@ async def test_single_embedding(client: openai.AsyncOpenAI, model_name: str):
65
66
# test using token IDs
67
input_tokens = [1, 1, 1, 1, 1]
62
- embeddings = await client.embeddings.create(
68
+ embedding_response = await client.embeddings.create(
69
model=model_name,
70
input=input_tokens,
71
encoding_format="float",
72
)
73
+ embeddings = EmbeddingResponse.model_validate(
74
+ embedding_response.model_dump(mode="json"))
75
+
76
assert embeddings.id is not None
77
assert len(embeddings.data) == 1
78
assert len(embeddings.data[0].embedding) == 4096
@@ -80,11 +89,14 @@ async def test_batch_embedding(client: openai.AsyncOpenAI, model_name: str):
89
"The cat sat on the mat.", "A feline was resting on a rug.",
90
"Stars twinkle brightly in the night sky."
91
]
83
- embeddings = await client.embeddings.create(
92
+ embedding_response = await client.embeddings.create(
93
model=model_name,
94
input=input_texts,
95
encoding_format="float",
96
)
97
+ embeddings = EmbeddingResponse.model_validate(
98
+ embedding_response.model_dump(mode="json"))
99
+
100
assert embeddings.id is not None
101
assert len(embeddings.data) == 3
102
assert len(embeddings.data[0].embedding) == 4096
@@ -95,11 +107,14 @@ async def test_batch_embedding(client: openai.AsyncOpenAI, model_name: str):
107
# test List[List[int]]
108
input_tokens = [[4, 5, 7, 9, 20], [15, 29, 499], [24, 24, 24, 24, 24],
109
[25, 32, 64, 77]]
98
- embeddings = await client.embeddings.create(
110
+ embedding_response = await client.embeddings.create(
111
model=model_name,
112
input=input_tokens,
113
encoding_format="float",
114
)
115
+ embeddings = EmbeddingResponse.model_validate(
116
+ embedding_response.model_dump(mode="json"))
117
+
118
assert embeddings.id is not None
119
assert len(embeddings.data) == 4
120
assert len(embeddings.data[0].embedding) == 4096
@@ -124,14 +139,16 @@ async def test_conversation_embedding(server: RemoteOpenAIServer,
139
"content": "Stars twinkle brightly in the night sky.",
140
}]
141
127
- chat_response = requests.post(server.url_for("v1/embeddings"),
128
- json={
129
- "model": model_name,
130
- "messages": messages,
131
- "encoding_format": "float",
132
- })
142
+ chat_response = requests.post(
143
+ server.url_for("v1/embeddings"),
144
+ json={
145
+ "model": model_name,
146
+ "messages": messages,
147
+ "encoding_format": "float",
148
+ },
149
+ )
150
chat_response.raise_for_status()
134
- chat_embeddings = chat_response.json()
151
+ chat_embeddings = EmbeddingResponse.model_validate(chat_response.json())
152
153
tokenizer = get_tokenizer(tokenizer_name=model_name, tokenizer_mode="fast")
154
prompt = tokenizer.apply_chat_template(
@@ -148,13 +165,15 @@ async def test_conversation_embedding(server: RemoteOpenAIServer,
165
# To be consistent with chat
166
extra_body={"add_special_tokens": False},
167
)
151
- completion_embeddings = completion_response.model_dump(mode="json")
168
+ completion_embeddings = EmbeddingResponse.model_validate(
169
+ completion_response.model_dump(mode="json"))
170
153
- assert chat_embeddings.pop("id") is not None
154
- assert completion_embeddings.pop("id") is not None
155
- assert chat_embeddings.pop("created") <= completion_embeddings.pop(
156
- "created")
157
- assert chat_embeddings == completion_embeddings
171
+ assert chat_embeddings.id is not None
172
+ assert completion_embeddings.id is not None
173
+ assert chat_embeddings.created <= completion_embeddings.created
174
+ assert chat_embeddings.model_dump(
175
+ exclude={"id", "created"}) == (completion_embeddings.model_dump(
176
+ exclude={"id", "created"}))
177
178
179
@pytest.mark.asyncio
@@ -204,10 +223,13 @@ async def test_single_embedding_truncation(client: openai.AsyncOpenAI,
223
]
224
225
# test single embedding
207
- embeddings = await client.embeddings.create(
226
+ embedding_response = await client.embeddings.create(
227
model=model_name,
228
input=input_texts,
229
extra_body={"truncate_prompt_tokens": 10})
230
+ embeddings = EmbeddingResponse.model_validate(
231
+ embedding_response.model_dump(mode="json"))
232
+
233
assert embeddings.id is not None
234
assert len(embeddings.data) == 1
235
assert len(embeddings.data[0].embedding) == 4096
@@ -219,10 +241,12 @@ async def test_single_embedding_truncation(client: openai.AsyncOpenAI,
241
1, 24428, 289, 18341, 26165, 285, 19323, 283, 289, 26789, 3871, 28728,
242
9901, 340, 2229, 385, 340, 315, 28741, 28804, 2
243
]
222
- embeddings = await client.embeddings.create(
244
+ embedding_response = await client.embeddings.create(
245
model=model_name,
246
input=input_tokens,
247
extra_body={"truncate_prompt_tokens": 10})
248
+ embeddings = EmbeddingResponse.model_validate(
249
+ embedding_response.model_dump(mode="json"))
250
251
assert embeddings.id is not None
252
assert len(embeddings.data) == 1
@@ -241,10 +265,10 @@ async def test_single_embedding_truncation_invalid(client: openai.AsyncOpenAI,
265
]
266
267
with pytest.raises(openai.BadRequestError):
244
- embeddings = await client.embeddings.create(
268
+ response = await client.embeddings.create(
269
model=model_name,
270
input=input_texts,
271
extra_body={"truncate_prompt_tokens": 8193})
248
- assert "error" in embeddings.object
272
+ assert "error" in response.object
273
assert "truncate_prompt_tokens value is greater than max_model_len. "\
250
- "Please, select a smaller truncation size." in embeddings.message
274
+ "Please, select a smaller truncation size." in response.message
tests/entrypoints/openai/test_pooling.py
new
+238
@@ -0,0 +1,238 @@
1
+import base64
2
+
3
+import numpy as np
4
+import pytest
5
+import requests
6
+
7
+from vllm.entrypoints.openai.protocol import PoolingResponse
8
+from vllm.transformers_utils.tokenizer import get_tokenizer
9
+
10
+from ...utils import RemoteOpenAIServer
11
+
12
+MODEL_NAME = "jason9693/Qwen2.5-1.5B-apeach"
13
+DUMMY_CHAT_TEMPLATE = """{% for message in messages %}{{message['role'] + ': ' + message['content'] + '\\n'}}{% endfor %}""" # noqa: E501
14
+
15
+
16
+@pytest.fixture(scope="module")
17
+def server():
18
+ args = [
19
+ "--task",
20
+ "classify",
21
+ # use half precision for speed and memory savings in CI environment
22
+ "--dtype",
23
+ "bfloat16",
24
+ "--enforce-eager",
25
+ "--max-model-len",
26
+ "8192",
27
+ "--chat-template",
28
+ DUMMY_CHAT_TEMPLATE,
29
+ ]
30
+
31
+ with RemoteOpenAIServer(MODEL_NAME, args) as remote_server:
32
+ yield remote_server
33
+
34
+
35
+@pytest.mark.asyncio
36
+@pytest.mark.parametrize("model_name", [MODEL_NAME])
37
+async def test_single_pooling(server: RemoteOpenAIServer, model_name: str):
38
+ input_texts = [
39
+ "The chef prepared a delicious meal.",
40
+ ]
41
+
42
+ # test single pooling
43
+ response = requests.post(
44
+ server.url_for("pooling"),
45
+ json={
46
+ "model": model_name,
47
+ "input": input_texts,
48
+ "encoding_format": "float"
49
+ },
50
+ )
51
+ response.raise_for_status()
52
+ poolings = PoolingResponse.model_validate(response.json())
53
+
54
+ assert poolings.id is not None
55
+ assert len(poolings.data) == 1
56
+ assert len(poolings.data[0].data) == 2
57
+ assert poolings.usage.completion_tokens == 0
58
+ assert poolings.usage.prompt_tokens == 7
59
+ assert poolings.usage.total_tokens == 7
60
+
61
+ # test using token IDs
62
+ input_tokens = [1, 1, 1, 1, 1]
63
+ response = requests.post(
64
+ server.url_for("pooling"),
65
+ json={
66
+ "model": model_name,
67
+ "input": input_tokens,
68
+ "encoding_format": "float"
69
+ },
70
+ )
71
+ response.raise_for_status()
72
+ poolings = PoolingResponse.model_validate(response.json())
73
+
74
+ assert poolings.id is not None
75
+ assert len(poolings.data) == 1
76
+ assert len(poolings.data[0].data) == 2
77
+ assert poolings.usage.completion_tokens == 0
78
+ assert poolings.usage.prompt_tokens == 5
79
+ assert poolings.usage.total_tokens == 5
80
+
81
+
82
+@pytest.mark.asyncio
83
+@pytest.mark.parametrize("model_name", [MODEL_NAME])
84
+async def test_batch_pooling(server: RemoteOpenAIServer, model_name: str):
85
+ # test List[str]
86
+ input_texts = [
87
+ "The cat sat on the mat.", "A feline was resting on a rug.",
88
+ "Stars twinkle brightly in the night sky."
89
+ ]
90
+ response = requests.post(
91
+ server.url_for("pooling"),
92
+ json={
93
+ "model": model_name,
94
+ "input": input_texts,
95
+ "encoding_format": "float"
96
+ },
97
+ )
98
+ response.raise_for_status()
99
+ poolings = PoolingResponse.model_validate(response.json())
100
+
101
+ assert poolings.id is not None
102
+ assert len(poolings.data) == 3
103
+ assert len(poolings.data[0].data) == 2
104
+ assert poolings.usage.completion_tokens == 0
105
+ assert poolings.usage.prompt_tokens == 25
106
+ assert poolings.usage.total_tokens == 25
107
+
108
+ # test List[List[int]]
109
+ input_tokens = [[4, 5, 7, 9, 20], [15, 29, 499], [24, 24, 24, 24, 24],
110
+ [25, 32, 64, 77]]
111
+ response = requests.post(
112
+ server.url_for("pooling"),
113
+ json={
114
+ "model": model_name,
115
+ "input": input_tokens,
116
+ "encoding_format": "float"
117
+ },
118
+ )
119
+ response.raise_for_status()
120
+ poolings = PoolingResponse.model_validate(response.json())
121
+
122
+ assert poolings.id is not None
123
+ assert len(poolings.data) == 4
124
+ assert len(poolings.data[0].data) == 2
125
+ assert poolings.usage.completion_tokens == 0
126
+ assert poolings.usage.prompt_tokens == 17
127
+ assert poolings.usage.total_tokens == 17
128
+
129
+
130
+@pytest.mark.asyncio
131
+@pytest.mark.parametrize("model_name", [MODEL_NAME])
132
+async def test_conversation_pooling(server: RemoteOpenAIServer,
133
+ model_name: str):
134
+ messages = [{
135
+ "role": "user",
136
+ "content": "The cat sat on the mat.",
137
+ }, {
138
+ "role": "assistant",
139
+ "content": "A feline was resting on a rug.",
140
+ }, {
141
+ "role": "user",
142
+ "content": "Stars twinkle brightly in the night sky.",
143
+ }]
144
+
145
+ chat_response = requests.post(
146
+ server.url_for("pooling"),
147
+ json={
148
+ "model": model_name,
149
+ "messages": messages,
150
+ "encoding_format": "float",
151
+ },
152
+ )
153
+ chat_response.raise_for_status()
154
+ chat_poolings = PoolingResponse.model_validate(chat_response.json())
155
+
156
+ tokenizer = get_tokenizer(tokenizer_name=model_name, tokenizer_mode="fast")
157
+ prompt = tokenizer.apply_chat_template(
158
+ messages,
159
+ chat_template=DUMMY_CHAT_TEMPLATE,
160
+ add_generation_prompt=True,
161
+ continue_final_message=False,
162
+ tokenize=False,
163
+ )
164
+ completions_response = requests.post(
165
+ server.url_for("pooling"),
166
+ json={
167
+ "model": model_name,
168
+ "input": prompt,
169
+ "encoding_format": "float",
170
+ # To be consistent with chat
171
+ "add_special_tokens": False,
172
+ },
173
+ )
174
+ completions_response.raise_for_status()
175
+ completion_poolings = PoolingResponse.model_validate(
176
+ completions_response.json())
177
+
178
+ assert chat_poolings.id is not None
179
+ assert completion_poolings.id is not None
180
+ assert chat_poolings.created <= completion_poolings.created
181
+ assert chat_poolings.model_dump(
182
+ exclude={"id", "created"}) == (completion_poolings.model_dump(
183
+ exclude={"id", "created"}))
184
+
185
+
186
+@pytest.mark.asyncio
187
+@pytest.mark.parametrize("model_name", [MODEL_NAME])
188
+async def test_batch_base64_pooling(server: RemoteOpenAIServer,
189
+ model_name: str):
190
+ input_texts = [
191
+ "Hello my name is",
192
+ "The best thing about vLLM is that it supports many different models"
193
+ ]
194
+
195
+ float_response = requests.post(
196
+ server.url_for("pooling"),
197
+ json={
198
+ "input": input_texts,
199
+ "model": model_name,
200
+ "encoding_format": "float",
201
+ },
202
+ )
203
+ float_response.raise_for_status()
204
+ responses_float = PoolingResponse.model_validate(float_response.json())
205
+
206
+ base64_response = requests.post(
207
+ server.url_for("pooling"),
208
+ json={
209
+ "input": input_texts,
210
+ "model": model_name,
211
+ "encoding_format": "base64",
212
+ },
213
+ )
214
+ base64_response.raise_for_status()
215
+ responses_base64 = PoolingResponse.model_validate(base64_response.json())
216
+
217
+ decoded_responses_base64_data = []
218
+ for data in responses_base64.data:
219
+ decoded_responses_base64_data.append(
220
+ np.frombuffer(base64.b64decode(data.data),
221
+ dtype="float32").tolist())
222
+
223
+ assert responses_float.data[0].data == decoded_responses_base64_data[0]
224
+ assert responses_float.data[1].data == decoded_responses_base64_data[1]
225
+
226
+ # Default response is float32 decoded from base64 by OpenAI Client
227
+ default_response = requests.post(
228
+ server.url_for("pooling"),
229
+ json={
230
+ "input": input_texts,
231
+ "model": model_name,
232
+ },
233
+ )
234
+ default_response.raise_for_status()
235
+ responses_default = PoolingResponse.model_validate(default_response.json())
236
+
237
+ assert responses_float.data[0].data == responses_default.data[0].data
238
+ assert responses_float.data[1].data == responses_default.data[1].data
tests/entrypoints/openai/test_vision_embedding.py
+17
-21
@@ -1,9 +1,9 @@
1
from typing import Dict
2
3
import pytest
4
-import pytest_asyncio
4
import requests
5
6
+from vllm.entrypoints.openai.protocol import EmbeddingResponse
7
from vllm.multimodal.utils import encode_image_base64, fetch_image
8
9
from ...utils import VLLM_PATH, RemoteOpenAIServer
@@ -46,12 +46,6 @@ def server():
46
yield remote_server
47
48
49
-@pytest_asyncio.fixture
50
-async def client(server):
51
- async with server.get_async_client() as async_client:
52
- yield async_client
53
-
54
-
49
@pytest.fixture(scope="session")
50
def base64_encoded_image() -> Dict[str, str]:
51
return {
@@ -82,18 +76,20 @@ async def test_image_embedding(server: RemoteOpenAIServer, model_name: str,
76
],
77
}]
78
85
- response = requests.post(server.url_for("v1/embeddings"),
86
- json={
87
- "model": model_name,
88
- "messages": messages,
89
- "encoding_format": "float"
90
- })
79
+ response = requests.post(
80
+ server.url_for("v1/embeddings"),
81
+ json={
82
+ "model": model_name,
83
+ "messages": messages,
84
+ "encoding_format": "float"
85
+ },
86
+ )
87
response.raise_for_status()
92
-
93
- embeddings = response.json()
94
- assert embeddings["id"] is not None
95
- assert len(embeddings["data"]) == 1
96
- assert len(embeddings["data"][0]["embedding"]) == 3072
97
- assert embeddings["usage"]["completion_tokens"] == 0
98
- assert embeddings["usage"]["prompt_tokens"] == 765
99
- assert embeddings["usage"]["total_tokens"] == 765
88
+ embeddings = EmbeddingResponse.model_validate(response.json())
89
+
90
+ assert embeddings.id is not None
91
+ assert len(embeddings.data) == 1
92
+ assert len(embeddings.data[0].embedding) == 3072
93
+ assert embeddings.usage.completion_tokens == 0
94
+ assert embeddings.usage.prompt_tokens == 765
95
+ assert embeddings.usage.total_tokens == 765
vllm/entrypoints/openai/api_server.py
+66
-7
@@ -45,8 +45,11 @@ from vllm.entrypoints.openai.protocol import (ChatCompletionRequest,
45
DetokenizeRequest,
46
DetokenizeResponse,
47
EmbeddingRequest,
48
- EmbeddingResponse, ErrorResponse,
48
+ EmbeddingResponse,
49
+ EmbeddingResponseData,
50
+ ErrorResponse,
51
LoadLoraAdapterRequest,
52
+ PoolingRequest, PoolingResponse,
53
ScoreRequest, ScoreResponse,
54
TokenizeRequest,
55
TokenizeResponse,
@@ -56,6 +59,7 @@ from vllm.entrypoints.openai.serving_chat import OpenAIServingChat
59
from vllm.entrypoints.openai.serving_completion import OpenAIServingCompletion
60
from vllm.entrypoints.openai.serving_embedding import OpenAIServingEmbedding
61
from vllm.entrypoints.openai.serving_engine import BaseModelPath, OpenAIServing
62
+from vllm.entrypoints.openai.serving_pooling import OpenAIServingPooling
63
from vllm.entrypoints.openai.serving_score import OpenAIServingScores
64
from vllm.entrypoints.openai.serving_tokenization import (
65
OpenAIServingTokenization)
@@ -284,6 +288,10 @@ def completion(request: Request) -> Optional[OpenAIServingCompletion]:
288
return request.app.state.openai_serving_completion
289
290
291
+def pooling(request: Request) -> Optional[OpenAIServingPooling]:
292
+ return request.app.state.openai_serving_pooling
293
+
294
+
295
def embedding(request: Request) -> Optional[OpenAIServingEmbedding]:
296
return request.app.state.openai_serving_embedding
297
@@ -395,10 +403,36 @@ async def create_completion(request: CompletionRequest, raw_request: Request):
403
async def create_embedding(request: EmbeddingRequest, raw_request: Request):
404
handler = embedding(raw_request)
405
if handler is None:
398
- return base(raw_request).create_error_response(
399
- message="The model does not support Embeddings API")
406
+ fallback_handler = pooling(raw_request)
407
+ if fallback_handler is None:
408
+ return base(raw_request).create_error_response(
409
+ message="The model does not support Embeddings API")
410
+
411
+ logger.warning(
412
+ "Embeddings API will become exclusive to embedding models "
413
+ "in a future release. To return the hidden states directly, "
414
+ "use the Pooling API (`/pooling`) instead.")
415
+
416
+ res = await fallback_handler.create_pooling(request, raw_request)
417
+ if isinstance(res, PoolingResponse):
418
+ generator = EmbeddingResponse(
419
+ id=res.id,
420
+ object=res.object,
421
+ created=res.created,
422
+ model=res.model,
423
+ data=[
424
+ EmbeddingResponseData(
425
+ index=d.index,
426
+ embedding=d.data, # type: ignore
427
+ ) for d in res.data
428
+ ],
429
+ usage=res.usage,
430
+ )
431
+ else:
432
+ generator = res
433
+ else:
434
+ generator = await handler.create_embedding(request, raw_request)
435
401
- generator = await handler.create_embedding(request, raw_request)
436
if isinstance(generator, ErrorResponse):
437
return JSONResponse(content=generator.model_dump(),
438
status_code=generator.code)
@@ -408,6 +442,24 @@ async def create_embedding(request: EmbeddingRequest, raw_request: Request):
442
assert_never(generator)
443
444
445
+@router.post("/pooling")
446
+@with_cancellation
447
+async def create_pooling(request: PoolingRequest, raw_request: Request):
448
+ handler = pooling(raw_request)
449
+ if handler is None:
450
+ return base(raw_request).create_error_response(
451
+ message="The model does not support Pooling API")
452
+
453
+ generator = await handler.create_pooling(request, raw_request)
454
+ if isinstance(generator, ErrorResponse):
455
+ return JSONResponse(content=generator.model_dump(),
456
+ status_code=generator.code)
457
+ elif isinstance(generator, PoolingResponse):
458
+ return JSONResponse(content=generator.model_dump())
459
+
460
+ assert_never(generator)
461
+
462
+
463
@router.post("/score")
464
@with_cancellation
465
async def create_score(request: ScoreRequest, raw_request: Request):
@@ -605,7 +657,7 @@ def init_app_state(
657
request_logger=request_logger,
658
return_tokens_as_token_ids=args.return_tokens_as_token_ids,
659
) if model_config.runner_type == "generate" else None
608
- state.openai_serving_embedding = OpenAIServingEmbedding(
660
+ state.openai_serving_pooling = OpenAIServingPooling(
661
engine_client,
662
model_config,
663
base_model_paths,
@@ -613,13 +665,20 @@ def init_app_state(
665
chat_template=resolved_chat_template,
666
chat_template_content_format=args.chat_template_content_format,
667
) if model_config.runner_type == "pooling" else None
668
+ state.openai_serving_embedding = OpenAIServingEmbedding(
669
+ engine_client,
670
+ model_config,
671
+ base_model_paths,
672
+ request_logger=request_logger,
673
+ chat_template=resolved_chat_template,
674
+ chat_template_content_format=args.chat_template_content_format,
675
+ ) if model_config.task == "embed" else None
676
state.openai_serving_scores = OpenAIServingScores(
677
engine_client,
678
model_config,
679
base_model_paths,
680
request_logger=request_logger
621
- ) if (model_config.runner_type == "pooling" \
622
- and model_config.is_cross_encoder) else None
681
+ ) if model_config.task == "score" else None
682
state.openai_serving_tokenization = OpenAIServingTokenization(
683
engine_client,
684
model_config,
vllm/entrypoints/openai/protocol.py
+19
@@ -963,6 +963,10 @@ class EmbeddingChatRequest(OpenAIBaseModel):
963
964
EmbeddingRequest = Union[EmbeddingCompletionRequest, EmbeddingChatRequest]
965
966
+PoolingCompletionRequest = EmbeddingCompletionRequest
967
+PoolingChatRequest = EmbeddingChatRequest
968
+PoolingRequest = Union[PoolingCompletionRequest, PoolingChatRequest]
969
+
970
971
class ScoreRequest(OpenAIBaseModel):
972
model: str
@@ -1058,6 +1062,21 @@ class EmbeddingResponse(OpenAIBaseModel):
1062
usage: UsageInfo
1063
1064
1065
+class PoolingResponseData(OpenAIBaseModel):
1066
+ index: int
1067
+ object: str = "pooling"
1068
+ data: Union[List[List[float]], List[float], str]
1069
+
1070
+
1071
+class PoolingResponse(OpenAIBaseModel):
1072
+ id: str = Field(default_factory=lambda: f"pool-{random_uuid()}")
1073
+ object: str = "list"
1074
+ created: int = Field(default_factory=lambda: int(time.time()))
1075
+ model: str
1076
+ data: List[PoolingResponseData]
1077
+ usage: UsageInfo
1078
+
1079
+
1080
class ScoreResponseData(OpenAIBaseModel):
1081
index: int
1082
object: str = "score"
vllm/entrypoints/openai/run_batch.py
+1
-1
@@ -232,7 +232,7 @@ async def main(args):
232
request_logger=request_logger,
233
chat_template=None,
234
chat_template_content_format="auto",
235
- ) if model_config.runner_type == "pooling" else None
235
+ ) if model_config.task == "embed" else None
236
237
tracker = BatchProgressTracker()
238
logger.info("Reading batch from %s...", args.input_file)
vllm/entrypoints/openai/serving_embedding.py
+45
-34
@@ -40,36 +40,6 @@ def _get_embedding(
40
assert_never(encoding_format)
41
42
43
-def request_output_to_embedding_response(
44
- final_res_batch: List[PoolingRequestOutput], request_id: str,
45
- created_time: int, model_name: str,
46
- encoding_format: Literal["float", "base64"]) -> EmbeddingResponse:
47
- data: List[EmbeddingResponseData] = []
48
- num_prompt_tokens = 0
49
- for idx, final_res in enumerate(final_res_batch):
50
- embedding_res = EmbeddingRequestOutput.from_base(final_res)
51
- prompt_token_ids = final_res.prompt_token_ids
52
-
53
- embedding = _get_embedding(embedding_res.outputs, encoding_format)
54
- embedding_data = EmbeddingResponseData(index=idx, embedding=embedding)
55
- data.append(embedding_data)
56
-
57
- num_prompt_tokens += len(prompt_token_ids)
58
-
59
- usage = UsageInfo(
60
- prompt_tokens=num_prompt_tokens,
61
- total_tokens=num_prompt_tokens,
62
- )
63
-
64
- return EmbeddingResponse(
65
- id=request_id,
66
- created=created_time,
67
- model=model_name,
68
- data=data,
69
- usage=usage,
70
- )
71
-
72
-
43
class OpenAIServingEmbedding(OpenAIServing):
44
45
def __init__(
@@ -114,7 +84,7 @@ class OpenAIServingEmbedding(OpenAIServing):
84
85
model_name = request.model
86
request_id = f"embd-{self._base_request_id(raw_request)}"
117
- created_time = int(time.monotonic())
87
+ created_time = int(time.time())
88
89
truncate_prompt_tokens = None
90
@@ -218,9 +188,13 @@ class OpenAIServingEmbedding(OpenAIServing):
188
final_res_batch_checked = cast(List[PoolingRequestOutput],
189
final_res_batch)
190
221
- response = request_output_to_embedding_response(
222
- final_res_batch_checked, request_id, created_time, model_name,
223
- encoding_format)
191
+ response = self.request_output_to_embedding_response(
192
+ final_res_batch_checked,
193
+ request_id,
194
+ created_time,
195
+ model_name,
196
+ encoding_format,
197
+ )
198
except asyncio.CancelledError:
199
return self.create_error_response("Client disconnected")
200
except ValueError as e:
@@ -228,3 +202,40 @@ class OpenAIServingEmbedding(OpenAIServing):
202
return self.create_error_response(str(e))
203
204
return response
205
+
206
+ def request_output_to_embedding_response(
207
+ self,
208
+ final_res_batch: List[PoolingRequestOutput],
209
+ request_id: str,
210
+ created_time: int,
211
+ model_name: str,
212
+ encoding_format: Literal["float", "base64"],
213
+ ) -> EmbeddingResponse:
214
+ items: List[EmbeddingResponseData] = []
215
+ num_prompt_tokens = 0
216
+
217
+ for idx, final_res in enumerate(final_res_batch):
218
+ embedding_res = EmbeddingRequestOutput.from_base(final_res)
219
+
220
+ item = EmbeddingResponseData(
221
+ index=idx,
222
+ embedding=_get_embedding(embedding_res.outputs,
223
+ encoding_format),
224
+ )
225
+ prompt_token_ids = final_res.prompt_token_ids
226
+
227
+ items.append(item)
228
+ num_prompt_tokens += len(prompt_token_ids)
229
+
230
+ usage = UsageInfo(
231
+ prompt_tokens=num_prompt_tokens,
232
+ total_tokens=num_prompt_tokens,
233
+ )
234
+
235
+ return EmbeddingResponse(
236
+ id=request_id,
237
+ created=created_time,
238
+ model=model_name,
239
+ data=items,
240
+ usage=usage,
241
+ )
vllm/entrypoints/openai/serving_pooling.py
new
+234
@@ -0,0 +1,234 @@
1
+import asyncio
2
+import base64
3
+import time
4
+from typing import AsyncGenerator, Final, List, Literal, Optional, Union, cast
5
+
6
+import numpy as np
7
+from fastapi import Request
8
+from typing_extensions import assert_never
9
+
10
+from vllm.config import ModelConfig
11
+from vllm.engine.protocol import EngineClient
12
+from vllm.entrypoints.chat_utils import ChatTemplateContentFormatOption
13
+from vllm.entrypoints.logger import RequestLogger
14
+from vllm.entrypoints.openai.protocol import (ErrorResponse,
15
+ PoolingChatRequest,
16
+ PoolingRequest, PoolingResponse,
17
+ PoolingResponseData, UsageInfo)
18
+from vllm.entrypoints.openai.serving_engine import BaseModelPath, OpenAIServing
19
+from vllm.logger import init_logger
20
+from vllm.outputs import PoolingOutput, PoolingRequestOutput
21
+from vllm.utils import merge_async_iterators
22
+
23
+logger = init_logger(__name__)
24
+
25
+
26
+def _get_data(
27
+ output: PoolingOutput,
28
+ encoding_format: Literal["float", "base64"],
29
+) -> Union[List[float], str]:
30
+ if encoding_format == "float":
31
+ return output.data.tolist()
32
+ elif encoding_format == "base64":
33
+ # Force to use float32 for base64 encoding
34
+ # to match the OpenAI python client behavior
35
+ pooling_bytes = np.array(output.data, dtype="float32").tobytes()
36
+ return base64.b64encode(pooling_bytes).decode("utf-8")
37
+
38
+ assert_never(encoding_format)
39
+
40
+
41
+class OpenAIServingPooling(OpenAIServing):
42
+
43
+ def __init__(
44
+ self,
45
+ engine_client: EngineClient,
46
+ model_config: ModelConfig,
47
+ base_model_paths: List[BaseModelPath],
48
+ *,
49
+ request_logger: Optional[RequestLogger],
50
+ chat_template: Optional[str],
51
+ chat_template_content_format: ChatTemplateContentFormatOption,
52
+ ) -> None:
53
+ super().__init__(engine_client=engine_client,
54
+ model_config=model_config,
55
+ base_model_paths=base_model_paths,
56
+ lora_modules=None,
57
+ prompt_adapters=None,
58
+ request_logger=request_logger)
59
+
60
+ self.chat_template = chat_template
61
+ self.chat_template_content_format: Final = chat_template_content_format
62
+
63
+ async def create_pooling(
64
+ self,
65
+ request: PoolingRequest,
66
+ raw_request: Optional[Request] = None,
67
+ ) -> Union[PoolingResponse, ErrorResponse]:
68
+ """
69
+ See https://platform.openai.com/docs/api-reference/embeddings/create
70
+ for the API specification. This API mimics the OpenAI Embedding API.
71
+ """
72
+ error_check_ret = await self._check_model(request)
73
+ if error_check_ret is not None:
74
+ return error_check_ret
75
+
76
+ encoding_format = request.encoding_format
77
+ if request.dimensions is not None:
78
+ return self.create_error_response(
79
+ "dimensions is currently not supported")
80
+
81
+ model_name = request.model
82
+ request_id = f"pool-{self._base_request_id(raw_request)}"
83
+ created_time = int(time.time())
84
+
85
+ truncate_prompt_tokens = None
86
+
87
+ if request.truncate_prompt_tokens is not None:
88
+ if request.truncate_prompt_tokens <= self.max_model_len:
89
+ truncate_prompt_tokens = request.truncate_prompt_tokens
90
+ else:
91
+ return self.create_error_response(
92
+ "truncate_prompt_tokens value is "
93
+ "greater than max_model_len."
94
+ " Please, select a smaller truncation size.")
95
+
96
+ try:
97
+ (
98
+ lora_request,
99
+ prompt_adapter_request,
100
+ ) = self._maybe_get_adapters(request)
101
+
102
+ tokenizer = await self.engine_client.get_tokenizer(lora_request)
103
+
104
+ if prompt_adapter_request is not None:
105
+ raise NotImplementedError("Prompt adapter is not supported "
106
+ "for pooling models")
107
+
108
+ if isinstance(request, PoolingChatRequest):
109
+ (
110
+ _,
111
+ request_prompts,
112
+ engine_prompts,
113
+ ) = await self._preprocess_chat(
114
+ request,
115
+ tokenizer,
116
+ request.messages,
117
+ chat_template=request.chat_template or self.chat_template,
118
+ chat_template_content_format=self.
119
+ chat_template_content_format,
120
+ # In pooling requests, we are not generating tokens,
121
+ # so there is no need to append extra tokens to the input
122
+ add_generation_prompt=False,
123
+ continue_final_message=False,
124
+ truncate_prompt_tokens=truncate_prompt_tokens,
125
+ add_special_tokens=request.add_special_tokens,
126
+ )
127
+ else:
128
+ (request_prompts,
129
+ engine_prompts) = await self._preprocess_completion(
130
+ request,
131
+ tokenizer,
132
+ request.input,
133
+ truncate_prompt_tokens=truncate_prompt_tokens,
134
+ add_special_tokens=request.add_special_tokens,
135
+ )
136
+ except ValueError as e:
137
+ logger.exception("Error in preprocessing prompt inputs")
138
+ return self.create_error_response(str(e))
139
+
140
+ # Schedule the request and get the result generator.
141
+ generators: List[AsyncGenerator[PoolingRequestOutput, None]] = []
142
+ try:
143
+ pooling_params = request.to_pooling_params()
144
+
145
+ for i, engine_prompt in enumerate(engine_prompts):
146
+ request_id_item = f"{request_id}-{i}"
147
+
148
+ self._log_inputs(request_id_item,
149
+ request_prompts[i],
150
+ params=pooling_params,
151
+ lora_request=lora_request,
152
+ prompt_adapter_request=prompt_adapter_request)
153
+
154
+ trace_headers = (None if raw_request is None else await
155
+ self._get_trace_headers(raw_request.headers))
156
+
157
+ generator = self.engine_client.encode(
158
+ engine_prompt,
159
+ pooling_params,
160
+ request_id_item,
161
+ lora_request=lora_request,
162
+ trace_headers=trace_headers,
163
+ priority=request.priority,
164
+ )
165
+
166
+ generators.append(generator)
167
+ except ValueError as e:
168
+ # TODO: Use a vllm-specific Validation Error
169
+ return self.create_error_response(str(e))
170
+
171
+ result_generator = merge_async_iterators(*generators)
172
+
173
+ num_prompts = len(engine_prompts)
174
+
175
+ # Non-streaming response
176
+ final_res_batch: List[Optional[PoolingRequestOutput]]
177
+ final_res_batch = [None] * num_prompts
178
+ try:
179
+ async for i, res in result_generator:
180
+ final_res_batch[i] = res
181
+
182
+ assert all(final_res is not None for final_res in final_res_batch)
183
+
184
+ final_res_batch_checked = cast(List[PoolingRequestOutput],
185
+ final_res_batch)
186
+
187
+ response = self.request_output_to_pooling_response(
188
+ final_res_batch_checked,
189
+ request_id,
190
+ created_time,
191
+ model_name,
192
+ encoding_format,
193
+ )
194
+ except asyncio.CancelledError:
195
+ return self.create_error_response("Client disconnected")
196
+ except ValueError as e:
197
+ # TODO: Use a vllm-specific Validation Error
198
+ return self.create_error_response(str(e))
199
+
200
+ return response
201
+
202
+ def request_output_to_pooling_response(
203
+ self,
204
+ final_res_batch: List[PoolingRequestOutput],
205
+ request_id: str,
206
+ created_time: int,
207
+ model_name: str,
208
+ encoding_format: Literal["float", "base64"],
209
+ ) -> PoolingResponse:
210
+ items: List[PoolingResponseData] = []
211
+ num_prompt_tokens = 0
212
+
213
+ for idx, final_res in enumerate(final_res_batch):
214
+ item = PoolingResponseData(
215
+ index=idx,
216
+ data=_get_data(final_res.outputs, encoding_format),
217
+ )
218
+ prompt_token_ids = final_res.prompt_token_ids
219
+
220
+ items.append(item)
221
+ num_prompt_tokens += len(prompt_token_ids)
222
+
223
+ usage = UsageInfo(
224
+ prompt_tokens=num_prompt_tokens,
225
+ total_tokens=num_prompt_tokens,
226
+ )
227
+
228
+ return PoolingResponse(
229
+ id=request_id,
230
+ created=created_time,
231
+ model=model_name,
232
+ data=items,
233
+ usage=usage,
234
+ )
vllm/entrypoints/openai/serving_score.py
+42
-29
@@ -20,32 +20,6 @@ from vllm.utils import make_async, merge_async_iterators
20
logger = init_logger(__name__)
21
22
23
-def request_output_to_score_response(
24
- final_res_batch: List[PoolingRequestOutput], request_id: str,
25
- created_time: int, model_name: str) -> ScoreResponse:
26
- data: List[ScoreResponseData] = []
27
- num_prompt_tokens = 0
28
- for idx, final_res in enumerate(final_res_batch):
29
- classify_res = ScoringRequestOutput.from_base(final_res)
30
-
31
- score_data = ScoreResponseData(index=idx,
32
- score=classify_res.outputs.score)
33
- data.append(score_data)
34
-
35
- usage = UsageInfo(
36
- prompt_tokens=num_prompt_tokens,
37
- total_tokens=num_prompt_tokens,
38
- )
39
-
40
- return ScoreResponse(
41
- id=request_id,
42
- created=created_time,
43
- model=model_name,
44
- data=data,
45
- usage=usage,
46
- )
47
-
48
-
23
def make_pairs(text_1: Union[List[str], str], text_2: Union[List[str],
24
str]) -> List:
25
if isinstance(text_1, (str, dict)):
@@ -103,7 +77,7 @@ class OpenAIServingScores(OpenAIServing):
77
78
model_name = request.model
79
request_id = f"score-{self._base_request_id(raw_request)}"
106
- created_time = int(time.monotonic())
80
+ created_time = int(time.time())
81
truncate_prompt_tokens = request.truncate_prompt_tokens
82
83
request_prompts = []
@@ -203,8 +177,12 @@ class OpenAIServingScores(OpenAIServing):
177
final_res_batch_checked = cast(List[PoolingRequestOutput],
178
final_res_batch)
179
206
- response = request_output_to_score_response(
207
- final_res_batch_checked, request_id, created_time, model_name)
180
+ response = self.request_output_to_score_response(
181
+ final_res_batch_checked,
182
+ request_id,
183
+ created_time,
184
+ model_name,
185
+ )
186
except asyncio.CancelledError:
187
return self.create_error_response("Client disconnected")
188
except ValueError as e:
@@ -212,3 +190,38 @@ class OpenAIServingScores(OpenAIServing):
190
return self.create_error_response(str(e))
191
192
return response
193
+
194
+ def request_output_to_score_response(
195
+ self,
196
+ final_res_batch: List[PoolingRequestOutput],
197
+ request_id: str,
198
+ created_time: int,
199
+ model_name: str,
200
+ ) -> ScoreResponse:
201
+ items: List[ScoreResponseData] = []
202
+ num_prompt_tokens = 0
203
+
204
+ for idx, final_res in enumerate(final_res_batch):
205
+ classify_res = ScoringRequestOutput.from_base(final_res)
206
+
207
+ item = ScoreResponseData(
208
+ index=idx,
209
+ score=classify_res.outputs.score,
210
+ )
211
+ prompt_token_ids = final_res.prompt_token_ids
212
+
213
+ items.append(item)
214
+ num_prompt_tokens += len(prompt_token_ids)
215
+
216
+ usage = UsageInfo(
217
+ prompt_tokens=num_prompt_tokens,
218
+ total_tokens=num_prompt_tokens,
219
+ )
220
+
221
+ return ScoreResponse(
222
+ id=request_id,
223
+ created=created_time,
224
+ model=model_name,
225
+ data=items,
226
+ usage=usage,
227
+ )
vllm/outputs.py
+2
-1
@@ -355,7 +355,8 @@ class PoolingRequestOutput(Generic[_O]):
355
pooled_data = seq_group.pooled_data
356
assert pooled_data is not None
357
358
- output = PoolingOutput(pooled_data)
358
+ data = pooled_data.to(dtype=torch.float32, device="cpu")
359
+ output = PoolingOutput(data)
360
prompt_token_ids = seq_group.prompt_token_ids
361
finished = seq_group.is_finished()
362