@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 9edca6bf8f
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 04:30 UTC

[Frontend] Online Pooling API (#11457)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Cyrus Leung committed Dec 24, 2024 at 17:54 UTC 9edca6bf8fa81e2dc678be68e9cdcede572947c1
15 files changed +809 -157
docs/source/models/generative_models.md
+3 -15
@@ -120,19 +120,7 @@ outputs = llm.chat(conversation, chat_template=custom_template)
120
121 ## Online Inference
122
123 -Our [OpenAI Compatible Server](../serving/openai_compatible_server) can be used for online inference.
124 -Please click on the above link for more details on how to launch the server.
123 +Our [OpenAI Compatible Server](../serving/openai_compatible_server) provides endpoints that correspond to the offline APIs:
124
126 -### Completions API
127 -
128 -Our Completions API is similar to `LLM.generate` but only accepts text.
129 -It is compatible with [OpenAI Completions API](https://platform.openai.com/docs/api-reference/completions)
130 -so that you can use OpenAI client to interact with it.
131 -A code example can be found in [examples/openai_completion_client.py](https://github.com/vllm-project/vllm/blob/main/examples/openai_completion_client.py).
132 -
133 -### Chat API
134 -
135 -Our Chat API is similar to `LLM.chat`, accepting both text and [multi-modal inputs](#multimodal-inputs).
136 -It is compatible with [OpenAI Chat Completions API](https://platform.openai.com/docs/api-reference/chat)
137 -so that you can use OpenAI client to interact with it.
138 -A code example can be found in [examples/openai_chat_completion_client.py](https://github.com/vllm-project/vllm/blob/main/examples/openai_chat_completion_client.py).
125 +- [Completions API](#completions-api) is similar to `LLM.generate` but only accepts text.
126 +- [Chat API](#chat-api) is similar to `LLM.chat`, accepting both text and [multi-modal inputs](#multimodal-inputs) for models with a chat template.
docs/source/models/pooling_models.md
+4 -18
@@ -106,22 +106,8 @@ A code example can be found in [examples/offline_inference_scoring.py](https://g
106
107 ## Online Inference
108
109 -Our [OpenAI Compatible Server](../serving/openai_compatible_server.md) can be used for online inference.
110 -Please click on the above link for more details on how to launch the server.
109 +Our [OpenAI Compatible Server](../serving/openai_compatible_server.md) provides endpoints that correspond to the offline APIs:
110
112 -### Embeddings API
113 -
114 -Our Embeddings API is similar to `LLM.embed`, accepting both text and [multi-modal inputs](#multimodal-inputs).
115 -
116 -The text-only API is compatible with [OpenAI Embeddings API](https://platform.openai.com/docs/api-reference/embeddings)
117 -so that you can use OpenAI client to interact with it.
118 -A code example can be found in [examples/openai_embedding_client.py](https://github.com/vllm-project/vllm/blob/main/examples/openai_embedding_client.py).
119 -
120 -The multi-modal API is an extension of the [OpenAI Embeddings API](https://platform.openai.com/docs/api-reference/embeddings)
121 -that incorporates [OpenAI Chat Completions API](https://platform.openai.com/docs/api-reference/chat),
122 -so it is not part of the OpenAI standard. Please see [](#multimodal-inputs) for more details on how to use it.
123 -
124 -### Score API
125 -
126 -Our Score API is similar to `LLM.score`.
127 -Please see [this page](#score-api) for more details on how to use it.
111 +- [Pooling API](#pooling-api) is similar to `LLM.encode`, being applicable to all types of pooling models.
112 +- [Embeddings API](#embeddings-api) is similar to `LLM.embed`, accepting both text and [multi-modal inputs](#multimodal-inputs) for embedding models.
113 +- [Score API](#score-api) is similar to `LLM.score` for cross-encoder models.
docs/source/serving/openai_compatible_server.md
+40 -8
@@ -42,6 +42,8 @@ In addition, we have the following custom APIs:
42
43 - [Tokenizer API](#tokenizer-api) (`/tokenize`, `/detokenize`)
44 - Applicable to any model with a tokenizer.
45 +- [Pooling API](#pooling-api) (`/pooling`)
46 + - Applicable to all [pooling models](../models/pooling_models.md).
47 - [Score API](#score-api) (`/score`)
48 - Only applicable to [cross-encoder models](../models/pooling_models.md) (`--task score`).
49
@@ -179,7 +181,12 @@ The order of priorities is `command line > config file values > defaults`.
181 (completions-api)=
182 ### Completions API
183
182 -Refer to [OpenAI's API reference](https://platform.openai.com/docs/api-reference/completions) for more details.
184 +Our Completions API is compatible with [OpenAI's Completions API](https://platform.openai.com/docs/api-reference/completions);
185 +you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.
186 +
187 +#### Code example
188 +
189 +See [examples/openai_completion_client.py](https://github.com/vllm-project/vllm/blob/main/examples/openai_completion_client.py).
190
191 #### Extra parameters
192
@@ -200,15 +207,20 @@ The following extra parameters are supported:
207 ```
208
209 (chat-api)=
203 -### Chat Completions API
210 +### Chat API
211
205 -Refer to [OpenAI's API reference](https://platform.openai.com/docs/api-reference/chat) for more details.
212 +Our Chat API is compatible with [OpenAI's Chat Completions API](https://platform.openai.com/docs/api-reference/chat);
213 +you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.
214
215 We support both [Vision](https://platform.openai.com/docs/guides/vision)- and
216 [Audio](https://platform.openai.com/docs/guides/audio?audio-generation-quickstart-example=audio-in)-related parameters;
217 see our [Multimodal Inputs](../usage/multimodal_inputs.md) guide for more information.
218 - *Note: `image_url.detail` parameter is not supported.*
219
220 +#### Code example
221 +
222 +See [examples/openai_chat_completion_client.py](https://github.com/vllm-project/vllm/blob/main/examples/openai_chat_completion_client.py).
223 +
224 #### Extra parameters
225
226 The following [sampling parameters (click through to see documentation)](../dev/sampling_params.md) are supported.
@@ -230,15 +242,20 @@ The following extra parameters are supported:
242 (embeddings-api)=
243 ### Embeddings API
244
233 -Refer to [OpenAI's API reference](https://platform.openai.com/docs/api-reference/embeddings) for more details.
245 +Our Embeddings API is compatible with [OpenAI's Embeddings API](https://platform.openai.com/docs/api-reference/embeddings);
246 +you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.
247
235 -If the model has a [chat template](#chat-template), you can replace `inputs` with a list of `messages` (same schema as [Chat Completions API](#chat-api))
248 +If the model has a [chat template](#chat-template), you can replace `inputs` with a list of `messages` (same schema as [Chat API](#chat-api))
249 which will be treated as a single prompt to the model.
250
251 ```{tip}
239 -This enables multi-modal inputs to be passed to embedding models, see [this page](../usage/multimodal_inputs.md) for details.
252 +This enables multi-modal inputs to be passed to embedding models, see [this page](#multimodal-inputs) for details.
253 ```
254
255 +#### Code example
256 +
257 +See [examples/openai_embedding_client.py](https://github.com/vllm-project/vllm/blob/main/examples/openai_embedding_client.py).
258 +
259 #### Extra parameters
260
261 The following [pooling parameters (click through to see documentation)](../dev/pooling_params.md) are supported.
@@ -268,20 +285,35 @@ For chat-like input (i.e. if `messages` is passed), these extra parameters are s
285 (tokenizer-api)=
286 ### Tokenizer API
287
271 -The Tokenizer API is a simple wrapper over [HuggingFace-style tokenizers](https://huggingface.co/docs/transformers/en/main_classes/tokenizer).
288 +Our Tokenizer API is a simple wrapper over [HuggingFace-style tokenizers](https://huggingface.co/docs/transformers/en/main_classes/tokenizer).
289 It consists of two endpoints:
290
291 - `/tokenize` corresponds to calling `tokenizer.encode()`.
292 - `/detokenize` corresponds to calling `tokenizer.decode()`.
293
294 +(pooling-api)=
295 +### Pooling API
296 +
297 +Our Pooling API encodes input prompts using a [pooling model](../models/pooling_models.md) and returns the corresponding hidden states.
298 +
299 +The input format is the same as [Embeddings API](#embeddings-api), but the output data can contain an arbitrary nested list, not just a 1-D list of floats.
300 +
301 +#### Code example
302 +
303 +See [examples/openai_pooling_client.py](https://github.com/vllm-project/vllm/blob/main/examples/openai_pooling_client.py).
304 +
305 (score-api)=
306 ### Score API
307
280 -The Score API applies a cross-encoder model to predict scores for sentence pairs.
308 +Our Score API applies a cross-encoder model to predict scores for sentence pairs.
309 Usually, the score for a sentence pair refers to the similarity between two sentences, on a scale of 0 to 1.
310
311 You can find the documentation for these kind of models at [sbert.net](https://www.sbert.net/docs/package_reference/cross_encoder/cross_encoder.html).
312
313 +#### Code example
314 +
315 +See [examples/openai_cross_encoder_score.py](https://github.com/vllm-project/vllm/blob/main/examples/openai_cross_encoder_score.py).
316 +
317 #### Single inference
318
319 You can pass a string to both `text_1` and `text_2`, forming a single sentence pair.
examples/openai_cross_encoder_score.py
+1 -1
@@ -20,9 +20,9 @@ if __name__ == "__main__":
20 parser.add_argument("--host", type=str, default="localhost")
21 parser.add_argument("--port", type=int, default=8000)
22 parser.add_argument("--model", type=str, default="BAAI/bge-reranker-v2-m3")
23 +
24 args = parser.parse_args()
25 api_url = f"http://{args.host}:{args.port}/score"
25 -
26 model_name = args.model
27
28 text_1 = "What is the capital of Brazil?"
examples/openai_pooling_client.py new
+51
@@ -0,0 +1,51 @@
1 +"""
2 +Example online usage of Pooling API.
3 +
4 +Run `vllm serve <model> --task <embed|classify|reward|score>`
5 +to start up the server in vLLM.
6 +"""
7 +import argparse
8 +import pprint
9 +
10 +import requests
11 +
12 +
13 +def post_http_request(prompt: dict, api_url: str) -> requests.Response:
14 + headers = {"User-Agent": "Test Client"}
15 + response = requests.post(api_url, headers=headers, json=prompt)
16 + return response
17 +
18 +
19 +if __name__ == "__main__":
20 + parser = argparse.ArgumentParser()
21 + parser.add_argument("--host", type=str, default="localhost")
22 + parser.add_argument("--port", type=int, default=8000)
23 + parser.add_argument("--model",
24 + type=str,
25 + default="jason9693/Qwen2.5-1.5B-apeach")
26 +
27 + args = parser.parse_args()
28 + api_url = f"http://{args.host}:{args.port}/pooling"
29 + model_name = args.model
30 +
31 + # Input like Completions API
32 + prompt = {"model": model_name, "input": "vLLM is great!"}
33 + pooling_response = post_http_request(prompt=prompt, api_url=api_url)
34 + print("Pooling Response:")
35 + pprint.pprint(pooling_response.json())
36 +
37 + # Input like Chat API
38 + prompt = {
39 + "model":
40 + model_name,
41 + "messages": [{
42 + "role": "user",
43 + "content": [{
44 + "type": "text",
45 + "text": "vLLM is great!"
46 + }],
47 + }]
48 + }
49 + pooling_response = post_http_request(prompt=prompt, api_url=api_url)
50 + print("Pooling Response:")
51 + pprint.pprint(pooling_response.json())
tests/entrypoints/openai/test_embedding.py
+46 -22
@@ -6,6 +6,7 @@ import pytest
6 import pytest_asyncio
7 import requests
8
9 +from vllm.entrypoints.openai.protocol import EmbeddingResponse
10 from vllm.transformers_utils.tokenizer import get_tokenizer
11
12 from ...utils import RemoteOpenAIServer
@@ -17,6 +18,8 @@ DUMMY_CHAT_TEMPLATE = """{% for message in messages %}{{message['role'] + ': ' +
18 @pytest.fixture(scope="module")
19 def server():
20 args = [
21 + "--task",
22 + "embed",
23 # use half precision for speed and memory savings in CI environment
24 "--dtype",
25 "bfloat16",
@@ -45,11 +48,14 @@ async def test_single_embedding(client: openai.AsyncOpenAI, model_name: str):
48 ]
49
50 # test single embedding
48 - embeddings = await client.embeddings.create(
51 + embedding_response = await client.embeddings.create(
52 model=model_name,
53 input=input_texts,
54 encoding_format="float",
55 )
56 + embeddings = EmbeddingResponse.model_validate(
57 + embedding_response.model_dump(mode="json"))
58 +
59 assert embeddings.id is not None
60 assert len(embeddings.data) == 1
61 assert len(embeddings.data[0].embedding) == 4096
@@ -59,11 +65,14 @@ async def test_single_embedding(client: openai.AsyncOpenAI, model_name: str):
65
66 # test using token IDs
67 input_tokens = [1, 1, 1, 1, 1]
62 - embeddings = await client.embeddings.create(
68 + embedding_response = await client.embeddings.create(
69 model=model_name,
70 input=input_tokens,
71 encoding_format="float",
72 )
73 + embeddings = EmbeddingResponse.model_validate(
74 + embedding_response.model_dump(mode="json"))
75 +
76 assert embeddings.id is not None
77 assert len(embeddings.data) == 1
78 assert len(embeddings.data[0].embedding) == 4096
@@ -80,11 +89,14 @@ async def test_batch_embedding(client: openai.AsyncOpenAI, model_name: str):
89 "The cat sat on the mat.", "A feline was resting on a rug.",
90 "Stars twinkle brightly in the night sky."
91 ]
83 - embeddings = await client.embeddings.create(
92 + embedding_response = await client.embeddings.create(
93 model=model_name,
94 input=input_texts,
95 encoding_format="float",
96 )
97 + embeddings = EmbeddingResponse.model_validate(
98 + embedding_response.model_dump(mode="json"))
99 +
100 assert embeddings.id is not None
101 assert len(embeddings.data) == 3
102 assert len(embeddings.data[0].embedding) == 4096
@@ -95,11 +107,14 @@ async def test_batch_embedding(client: openai.AsyncOpenAI, model_name: str):
107 # test List[List[int]]
108 input_tokens = [[4, 5, 7, 9, 20], [15, 29, 499], [24, 24, 24, 24, 24],
109 [25, 32, 64, 77]]
98 - embeddings = await client.embeddings.create(
110 + embedding_response = await client.embeddings.create(
111 model=model_name,
112 input=input_tokens,
113 encoding_format="float",
114 )
115 + embeddings = EmbeddingResponse.model_validate(
116 + embedding_response.model_dump(mode="json"))
117 +
118 assert embeddings.id is not None
119 assert len(embeddings.data) == 4
120 assert len(embeddings.data[0].embedding) == 4096
@@ -124,14 +139,16 @@ async def test_conversation_embedding(server: RemoteOpenAIServer,
139 "content": "Stars twinkle brightly in the night sky.",
140 }]
141
127 - chat_response = requests.post(server.url_for("v1/embeddings"),
128 - json={
129 - "model": model_name,
130 - "messages": messages,
131 - "encoding_format": "float",
132 - })
142 + chat_response = requests.post(
143 + server.url_for("v1/embeddings"),
144 + json={
145 + "model": model_name,
146 + "messages": messages,
147 + "encoding_format": "float",
148 + },
149 + )
150 chat_response.raise_for_status()
134 - chat_embeddings = chat_response.json()
151 + chat_embeddings = EmbeddingResponse.model_validate(chat_response.json())
152
153 tokenizer = get_tokenizer(tokenizer_name=model_name, tokenizer_mode="fast")
154 prompt = tokenizer.apply_chat_template(
@@ -148,13 +165,15 @@ async def test_conversation_embedding(server: RemoteOpenAIServer,
165 # To be consistent with chat
166 extra_body={"add_special_tokens": False},
167 )
151 - completion_embeddings = completion_response.model_dump(mode="json")
168 + completion_embeddings = EmbeddingResponse.model_validate(
169 + completion_response.model_dump(mode="json"))
170
153 - assert chat_embeddings.pop("id") is not None
154 - assert completion_embeddings.pop("id") is not None
155 - assert chat_embeddings.pop("created") <= completion_embeddings.pop(
156 - "created")
157 - assert chat_embeddings == completion_embeddings
171 + assert chat_embeddings.id is not None
172 + assert completion_embeddings.id is not None
173 + assert chat_embeddings.created <= completion_embeddings.created
174 + assert chat_embeddings.model_dump(
175 + exclude={"id", "created"}) == (completion_embeddings.model_dump(
176 + exclude={"id", "created"}))
177
178
179 @pytest.mark.asyncio
@@ -204,10 +223,13 @@ async def test_single_embedding_truncation(client: openai.AsyncOpenAI,
223 ]
224
225 # test single embedding
207 - embeddings = await client.embeddings.create(
226 + embedding_response = await client.embeddings.create(
227 model=model_name,
228 input=input_texts,
229 extra_body={"truncate_prompt_tokens": 10})
230 + embeddings = EmbeddingResponse.model_validate(
231 + embedding_response.model_dump(mode="json"))
232 +
233 assert embeddings.id is not None
234 assert len(embeddings.data) == 1
235 assert len(embeddings.data[0].embedding) == 4096
@@ -219,10 +241,12 @@ async def test_single_embedding_truncation(client: openai.AsyncOpenAI,
241 1, 24428, 289, 18341, 26165, 285, 19323, 283, 289, 26789, 3871, 28728,
242 9901, 340, 2229, 385, 340, 315, 28741, 28804, 2
243 ]
222 - embeddings = await client.embeddings.create(
244 + embedding_response = await client.embeddings.create(
245 model=model_name,
246 input=input_tokens,
247 extra_body={"truncate_prompt_tokens": 10})
248 + embeddings = EmbeddingResponse.model_validate(
249 + embedding_response.model_dump(mode="json"))
250
251 assert embeddings.id is not None
252 assert len(embeddings.data) == 1
@@ -241,10 +265,10 @@ async def test_single_embedding_truncation_invalid(client: openai.AsyncOpenAI,
265 ]
266
267 with pytest.raises(openai.BadRequestError):
244 - embeddings = await client.embeddings.create(
268 + response = await client.embeddings.create(
269 model=model_name,
270 input=input_texts,
271 extra_body={"truncate_prompt_tokens": 8193})
248 - assert "error" in embeddings.object
272 + assert "error" in response.object
273 assert "truncate_prompt_tokens value is greater than max_model_len. "\
250 - "Please, select a smaller truncation size." in embeddings.message
274 + "Please, select a smaller truncation size." in response.message
tests/entrypoints/openai/test_pooling.py new
+238
@@ -0,0 +1,238 @@
1 +import base64
2 +
3 +import numpy as np
4 +import pytest
5 +import requests
6 +
7 +from vllm.entrypoints.openai.protocol import PoolingResponse
8 +from vllm.transformers_utils.tokenizer import get_tokenizer
9 +
10 +from ...utils import RemoteOpenAIServer
11 +
12 +MODEL_NAME = "jason9693/Qwen2.5-1.5B-apeach"
13 +DUMMY_CHAT_TEMPLATE = """{% for message in messages %}{{message['role'] + ': ' + message['content'] + '\\n'}}{% endfor %}""" # noqa: E501
14 +
15 +
16 +@pytest.fixture(scope="module")
17 +def server():
18 + args = [
19 + "--task",
20 + "classify",
21 + # use half precision for speed and memory savings in CI environment
22 + "--dtype",
23 + "bfloat16",
24 + "--enforce-eager",
25 + "--max-model-len",
26 + "8192",
27 + "--chat-template",
28 + DUMMY_CHAT_TEMPLATE,
29 + ]
30 +
31 + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server:
32 + yield remote_server
33 +
34 +
35 +@pytest.mark.asyncio
36 +@pytest.mark.parametrize("model_name", [MODEL_NAME])
37 +async def test_single_pooling(server: RemoteOpenAIServer, model_name: str):
38 + input_texts = [
39 + "The chef prepared a delicious meal.",
40 + ]
41 +
42 + # test single pooling
43 + response = requests.post(
44 + server.url_for("pooling"),
45 + json={
46 + "model": model_name,
47 + "input": input_texts,
48 + "encoding_format": "float"
49 + },
50 + )
51 + response.raise_for_status()
52 + poolings = PoolingResponse.model_validate(response.json())
53 +
54 + assert poolings.id is not None
55 + assert len(poolings.data) == 1
56 + assert len(poolings.data[0].data) == 2
57 + assert poolings.usage.completion_tokens == 0
58 + assert poolings.usage.prompt_tokens == 7
59 + assert poolings.usage.total_tokens == 7
60 +
61 + # test using token IDs
62 + input_tokens = [1, 1, 1, 1, 1]
63 + response = requests.post(
64 + server.url_for("pooling"),
65 + json={
66 + "model": model_name,
67 + "input": input_tokens,
68 + "encoding_format": "float"
69 + },
70 + )
71 + response.raise_for_status()
72 + poolings = PoolingResponse.model_validate(response.json())
73 +
74 + assert poolings.id is not None
75 + assert len(poolings.data) == 1
76 + assert len(poolings.data[0].data) == 2
77 + assert poolings.usage.completion_tokens == 0
78 + assert poolings.usage.prompt_tokens == 5
79 + assert poolings.usage.total_tokens == 5
80 +
81 +
82 +@pytest.mark.asyncio
83 +@pytest.mark.parametrize("model_name", [MODEL_NAME])
84 +async def test_batch_pooling(server: RemoteOpenAIServer, model_name: str):
85 + # test List[str]
86 + input_texts = [
87 + "The cat sat on the mat.", "A feline was resting on a rug.",
88 + "Stars twinkle brightly in the night sky."
89 + ]
90 + response = requests.post(
91 + server.url_for("pooling"),
92 + json={
93 + "model": model_name,
94 + "input": input_texts,
95 + "encoding_format": "float"
96 + },
97 + )
98 + response.raise_for_status()
99 + poolings = PoolingResponse.model_validate(response.json())
100 +
101 + assert poolings.id is not None
102 + assert len(poolings.data) == 3
103 + assert len(poolings.data[0].data) == 2
104 + assert poolings.usage.completion_tokens == 0
105 + assert poolings.usage.prompt_tokens == 25
106 + assert poolings.usage.total_tokens == 25
107 +
108 + # test List[List[int]]
109 + input_tokens = [[4, 5, 7, 9, 20], [15, 29, 499], [24, 24, 24, 24, 24],
110 + [25, 32, 64, 77]]
111 + response = requests.post(
112 + server.url_for("pooling"),
113 + json={
114 + "model": model_name,
115 + "input": input_tokens,
116 + "encoding_format": "float"
117 + },
118 + )
119 + response.raise_for_status()
120 + poolings = PoolingResponse.model_validate(response.json())
121 +
122 + assert poolings.id is not None
123 + assert len(poolings.data) == 4
124 + assert len(poolings.data[0].data) == 2
125 + assert poolings.usage.completion_tokens == 0
126 + assert poolings.usage.prompt_tokens == 17
127 + assert poolings.usage.total_tokens == 17
128 +
129 +
130 +@pytest.mark.asyncio
131 +@pytest.mark.parametrize("model_name", [MODEL_NAME])
132 +async def test_conversation_pooling(server: RemoteOpenAIServer,
133 + model_name: str):
134 + messages = [{
135 + "role": "user",
136 + "content": "The cat sat on the mat.",
137 + }, {
138 + "role": "assistant",
139 + "content": "A feline was resting on a rug.",
140 + }, {
141 + "role": "user",
142 + "content": "Stars twinkle brightly in the night sky.",
143 + }]
144 +
145 + chat_response = requests.post(
146 + server.url_for("pooling"),
147 + json={
148 + "model": model_name,
149 + "messages": messages,
150 + "encoding_format": "float",
151 + },
152 + )
153 + chat_response.raise_for_status()
154 + chat_poolings = PoolingResponse.model_validate(chat_response.json())
155 +
156 + tokenizer = get_tokenizer(tokenizer_name=model_name, tokenizer_mode="fast")
157 + prompt = tokenizer.apply_chat_template(
158 + messages,
159 + chat_template=DUMMY_CHAT_TEMPLATE,
160 + add_generation_prompt=True,
161 + continue_final_message=False,
162 + tokenize=False,
163 + )
164 + completions_response = requests.post(
165 + server.url_for("pooling"),
166 + json={
167 + "model": model_name,
168 + "input": prompt,
169 + "encoding_format": "float",
170 + # To be consistent with chat
171 + "add_special_tokens": False,
172 + },
173 + )
174 + completions_response.raise_for_status()
175 + completion_poolings = PoolingResponse.model_validate(
176 + completions_response.json())
177 +
178 + assert chat_poolings.id is not None
179 + assert completion_poolings.id is not None
180 + assert chat_poolings.created <= completion_poolings.created
181 + assert chat_poolings.model_dump(
182 + exclude={"id", "created"}) == (completion_poolings.model_dump(
183 + exclude={"id", "created"}))
184 +
185 +
186 +@pytest.mark.asyncio
187 +@pytest.mark.parametrize("model_name", [MODEL_NAME])
188 +async def test_batch_base64_pooling(server: RemoteOpenAIServer,
189 + model_name: str):
190 + input_texts = [
191 + "Hello my name is",
192 + "The best thing about vLLM is that it supports many different models"
193 + ]
194 +
195 + float_response = requests.post(
196 + server.url_for("pooling"),
197 + json={
198 + "input": input_texts,
199 + "model": model_name,
200 + "encoding_format": "float",
201 + },
202 + )
203 + float_response.raise_for_status()
204 + responses_float = PoolingResponse.model_validate(float_response.json())
205 +
206 + base64_response = requests.post(
207 + server.url_for("pooling"),
208 + json={
209 + "input": input_texts,
210 + "model": model_name,
211 + "encoding_format": "base64",
212 + },
213 + )
214 + base64_response.raise_for_status()
215 + responses_base64 = PoolingResponse.model_validate(base64_response.json())
216 +
217 + decoded_responses_base64_data = []
218 + for data in responses_base64.data:
219 + decoded_responses_base64_data.append(
220 + np.frombuffer(base64.b64decode(data.data),
221 + dtype="float32").tolist())
222 +
223 + assert responses_float.data[0].data == decoded_responses_base64_data[0]
224 + assert responses_float.data[1].data == decoded_responses_base64_data[1]
225 +
226 + # Default response is float32 decoded from base64 by OpenAI Client
227 + default_response = requests.post(
228 + server.url_for("pooling"),
229 + json={
230 + "input": input_texts,
231 + "model": model_name,
232 + },
233 + )
234 + default_response.raise_for_status()
235 + responses_default = PoolingResponse.model_validate(default_response.json())
236 +
237 + assert responses_float.data[0].data == responses_default.data[0].data
238 + assert responses_float.data[1].data == responses_default.data[1].data
tests/entrypoints/openai/test_vision_embedding.py
+17 -21
@@ -1,9 +1,9 @@
1 from typing import Dict
2
3 import pytest
4 -import pytest_asyncio
4 import requests
5
6 +from vllm.entrypoints.openai.protocol import EmbeddingResponse
7 from vllm.multimodal.utils import encode_image_base64, fetch_image
8
9 from ...utils import VLLM_PATH, RemoteOpenAIServer
@@ -46,12 +46,6 @@ def server():
46 yield remote_server
47
48
49 -@pytest_asyncio.fixture
50 -async def client(server):
51 - async with server.get_async_client() as async_client:
52 - yield async_client
53 -
54 -
49 @pytest.fixture(scope="session")
50 def base64_encoded_image() -> Dict[str, str]:
51 return {
@@ -82,18 +76,20 @@ async def test_image_embedding(server: RemoteOpenAIServer, model_name: str,
76 ],
77 }]
78
85 - response = requests.post(server.url_for("v1/embeddings"),
86 - json={
87 - "model": model_name,
88 - "messages": messages,
89 - "encoding_format": "float"
90 - })
79 + response = requests.post(
80 + server.url_for("v1/embeddings"),
81 + json={
82 + "model": model_name,
83 + "messages": messages,
84 + "encoding_format": "float"
85 + },
86 + )
87 response.raise_for_status()
92 -
93 - embeddings = response.json()
94 - assert embeddings["id"] is not None
95 - assert len(embeddings["data"]) == 1
96 - assert len(embeddings["data"][0]["embedding"]) == 3072
97 - assert embeddings["usage"]["completion_tokens"] == 0
98 - assert embeddings["usage"]["prompt_tokens"] == 765
99 - assert embeddings["usage"]["total_tokens"] == 765
88 + embeddings = EmbeddingResponse.model_validate(response.json())
89 +
90 + assert embeddings.id is not None
91 + assert len(embeddings.data) == 1
92 + assert len(embeddings.data[0].embedding) == 3072
93 + assert embeddings.usage.completion_tokens == 0
94 + assert embeddings.usage.prompt_tokens == 765
95 + assert embeddings.usage.total_tokens == 765
vllm/entrypoints/openai/api_server.py
+66 -7
@@ -45,8 +45,11 @@ from vllm.entrypoints.openai.protocol import (ChatCompletionRequest,
45 DetokenizeRequest,
46 DetokenizeResponse,
47 EmbeddingRequest,
48 - EmbeddingResponse, ErrorResponse,
48 + EmbeddingResponse,
49 + EmbeddingResponseData,
50 + ErrorResponse,
51 LoadLoraAdapterRequest,
52 + PoolingRequest, PoolingResponse,
53 ScoreRequest, ScoreResponse,
54 TokenizeRequest,
55 TokenizeResponse,
@@ -56,6 +59,7 @@ from vllm.entrypoints.openai.serving_chat import OpenAIServingChat
59 from vllm.entrypoints.openai.serving_completion import OpenAIServingCompletion
60 from vllm.entrypoints.openai.serving_embedding import OpenAIServingEmbedding
61 from vllm.entrypoints.openai.serving_engine import BaseModelPath, OpenAIServing
62 +from vllm.entrypoints.openai.serving_pooling import OpenAIServingPooling
63 from vllm.entrypoints.openai.serving_score import OpenAIServingScores
64 from vllm.entrypoints.openai.serving_tokenization import (
65 OpenAIServingTokenization)
@@ -284,6 +288,10 @@ def completion(request: Request) -> Optional[OpenAIServingCompletion]:
288 return request.app.state.openai_serving_completion
289
290
291 +def pooling(request: Request) -> Optional[OpenAIServingPooling]:
292 + return request.app.state.openai_serving_pooling
293 +
294 +
295 def embedding(request: Request) -> Optional[OpenAIServingEmbedding]:
296 return request.app.state.openai_serving_embedding
297
@@ -395,10 +403,36 @@ async def create_completion(request: CompletionRequest, raw_request: Request):
403 async def create_embedding(request: EmbeddingRequest, raw_request: Request):
404 handler = embedding(raw_request)
405 if handler is None:
398 - return base(raw_request).create_error_response(
399 - message="The model does not support Embeddings API")
406 + fallback_handler = pooling(raw_request)
407 + if fallback_handler is None:
408 + return base(raw_request).create_error_response(
409 + message="The model does not support Embeddings API")
410 +
411 + logger.warning(
412 + "Embeddings API will become exclusive to embedding models "
413 + "in a future release. To return the hidden states directly, "
414 + "use the Pooling API (`/pooling`) instead.")
415 +
416 + res = await fallback_handler.create_pooling(request, raw_request)
417 + if isinstance(res, PoolingResponse):
418 + generator = EmbeddingResponse(
419 + id=res.id,
420 + object=res.object,
421 + created=res.created,
422 + model=res.model,
423 + data=[
424 + EmbeddingResponseData(
425 + index=d.index,
426 + embedding=d.data, # type: ignore
427 + ) for d in res.data
428 + ],
429 + usage=res.usage,
430 + )
431 + else:
432 + generator = res
433 + else:
434 + generator = await handler.create_embedding(request, raw_request)
435
401 - generator = await handler.create_embedding(request, raw_request)
436 if isinstance(generator, ErrorResponse):
437 return JSONResponse(content=generator.model_dump(),
438 status_code=generator.code)
@@ -408,6 +442,24 @@ async def create_embedding(request: EmbeddingRequest, raw_request: Request):
442 assert_never(generator)
443
444
445 +@router.post("/pooling")
446 +@with_cancellation
447 +async def create_pooling(request: PoolingRequest, raw_request: Request):
448 + handler = pooling(raw_request)
449 + if handler is None:
450 + return base(raw_request).create_error_response(
451 + message="The model does not support Pooling API")
452 +
453 + generator = await handler.create_pooling(request, raw_request)
454 + if isinstance(generator, ErrorResponse):
455 + return JSONResponse(content=generator.model_dump(),
456 + status_code=generator.code)
457 + elif isinstance(generator, PoolingResponse):
458 + return JSONResponse(content=generator.model_dump())
459 +
460 + assert_never(generator)
461 +
462 +
463 @router.post("/score")
464 @with_cancellation
465 async def create_score(request: ScoreRequest, raw_request: Request):
@@ -605,7 +657,7 @@ def init_app_state(
657 request_logger=request_logger,
658 return_tokens_as_token_ids=args.return_tokens_as_token_ids,
659 ) if model_config.runner_type == "generate" else None
608 - state.openai_serving_embedding = OpenAIServingEmbedding(
660 + state.openai_serving_pooling = OpenAIServingPooling(
661 engine_client,
662 model_config,
663 base_model_paths,
@@ -613,13 +665,20 @@ def init_app_state(
665 chat_template=resolved_chat_template,
666 chat_template_content_format=args.chat_template_content_format,
667 ) if model_config.runner_type == "pooling" else None
668 + state.openai_serving_embedding = OpenAIServingEmbedding(
669 + engine_client,
670 + model_config,
671 + base_model_paths,
672 + request_logger=request_logger,
673 + chat_template=resolved_chat_template,
674 + chat_template_content_format=args.chat_template_content_format,
675 + ) if model_config.task == "embed" else None
676 state.openai_serving_scores = OpenAIServingScores(
677 engine_client,
678 model_config,
679 base_model_paths,
680 request_logger=request_logger
621 - ) if (model_config.runner_type == "pooling" \
622 - and model_config.is_cross_encoder) else None
681 + ) if model_config.task == "score" else None
682 state.openai_serving_tokenization = OpenAIServingTokenization(
683 engine_client,
684 model_config,
vllm/entrypoints/openai/protocol.py
+19
@@ -963,6 +963,10 @@ class EmbeddingChatRequest(OpenAIBaseModel):
963
964 EmbeddingRequest = Union[EmbeddingCompletionRequest, EmbeddingChatRequest]
965
966 +PoolingCompletionRequest = EmbeddingCompletionRequest
967 +PoolingChatRequest = EmbeddingChatRequest
968 +PoolingRequest = Union[PoolingCompletionRequest, PoolingChatRequest]
969 +
970
971 class ScoreRequest(OpenAIBaseModel):
972 model: str
@@ -1058,6 +1062,21 @@ class EmbeddingResponse(OpenAIBaseModel):
1062 usage: UsageInfo
1063
1064
1065 +class PoolingResponseData(OpenAIBaseModel):
1066 + index: int
1067 + object: str = "pooling"
1068 + data: Union[List[List[float]], List[float], str]
1069 +
1070 +
1071 +class PoolingResponse(OpenAIBaseModel):
1072 + id: str = Field(default_factory=lambda: f"pool-{random_uuid()}")
1073 + object: str = "list"
1074 + created: int = Field(default_factory=lambda: int(time.time()))
1075 + model: str
1076 + data: List[PoolingResponseData]
1077 + usage: UsageInfo
1078 +
1079 +
1080 class ScoreResponseData(OpenAIBaseModel):
1081 index: int
1082 object: str = "score"
vllm/entrypoints/openai/run_batch.py
+1 -1
@@ -232,7 +232,7 @@ async def main(args):
232 request_logger=request_logger,
233 chat_template=None,
234 chat_template_content_format="auto",
235 - ) if model_config.runner_type == "pooling" else None
235 + ) if model_config.task == "embed" else None
236
237 tracker = BatchProgressTracker()
238 logger.info("Reading batch from %s...", args.input_file)
vllm/entrypoints/openai/serving_embedding.py
+45 -34
@@ -40,36 +40,6 @@ def _get_embedding(
40 assert_never(encoding_format)
41
42
43 -def request_output_to_embedding_response(
44 - final_res_batch: List[PoolingRequestOutput], request_id: str,
45 - created_time: int, model_name: str,
46 - encoding_format: Literal["float", "base64"]) -> EmbeddingResponse:
47 - data: List[EmbeddingResponseData] = []
48 - num_prompt_tokens = 0
49 - for idx, final_res in enumerate(final_res_batch):
50 - embedding_res = EmbeddingRequestOutput.from_base(final_res)
51 - prompt_token_ids = final_res.prompt_token_ids
52 -
53 - embedding = _get_embedding(embedding_res.outputs, encoding_format)
54 - embedding_data = EmbeddingResponseData(index=idx, embedding=embedding)
55 - data.append(embedding_data)
56 -
57 - num_prompt_tokens += len(prompt_token_ids)
58 -
59 - usage = UsageInfo(
60 - prompt_tokens=num_prompt_tokens,
61 - total_tokens=num_prompt_tokens,
62 - )
63 -
64 - return EmbeddingResponse(
65 - id=request_id,
66 - created=created_time,
67 - model=model_name,
68 - data=data,
69 - usage=usage,
70 - )
71 -
72 -
43 class OpenAIServingEmbedding(OpenAIServing):
44
45 def __init__(
@@ -114,7 +84,7 @@ class OpenAIServingEmbedding(OpenAIServing):
84
85 model_name = request.model
86 request_id = f"embd-{self._base_request_id(raw_request)}"
117 - created_time = int(time.monotonic())
87 + created_time = int(time.time())
88
89 truncate_prompt_tokens = None
90
@@ -218,9 +188,13 @@ class OpenAIServingEmbedding(OpenAIServing):
188 final_res_batch_checked = cast(List[PoolingRequestOutput],
189 final_res_batch)
190
221 - response = request_output_to_embedding_response(
222 - final_res_batch_checked, request_id, created_time, model_name,
223 - encoding_format)
191 + response = self.request_output_to_embedding_response(
192 + final_res_batch_checked,
193 + request_id,
194 + created_time,
195 + model_name,
196 + encoding_format,
197 + )
198 except asyncio.CancelledError:
199 return self.create_error_response("Client disconnected")
200 except ValueError as e:
@@ -228,3 +202,40 @@ class OpenAIServingEmbedding(OpenAIServing):
202 return self.create_error_response(str(e))
203
204 return response
205 +
206 + def request_output_to_embedding_response(
207 + self,
208 + final_res_batch: List[PoolingRequestOutput],
209 + request_id: str,
210 + created_time: int,
211 + model_name: str,
212 + encoding_format: Literal["float", "base64"],
213 + ) -> EmbeddingResponse:
214 + items: List[EmbeddingResponseData] = []
215 + num_prompt_tokens = 0
216 +
217 + for idx, final_res in enumerate(final_res_batch):
218 + embedding_res = EmbeddingRequestOutput.from_base(final_res)
219 +
220 + item = EmbeddingResponseData(
221 + index=idx,
222 + embedding=_get_embedding(embedding_res.outputs,
223 + encoding_format),
224 + )
225 + prompt_token_ids = final_res.prompt_token_ids
226 +
227 + items.append(item)
228 + num_prompt_tokens += len(prompt_token_ids)
229 +
230 + usage = UsageInfo(
231 + prompt_tokens=num_prompt_tokens,
232 + total_tokens=num_prompt_tokens,
233 + )
234 +
235 + return EmbeddingResponse(
236 + id=request_id,
237 + created=created_time,
238 + model=model_name,
239 + data=items,
240 + usage=usage,
241 + )
vllm/entrypoints/openai/serving_pooling.py new
+234
@@ -0,0 +1,234 @@
1 +import asyncio
2 +import base64
3 +import time
4 +from typing import AsyncGenerator, Final, List, Literal, Optional, Union, cast
5 +
6 +import numpy as np
7 +from fastapi import Request
8 +from typing_extensions import assert_never
9 +
10 +from vllm.config import ModelConfig
11 +from vllm.engine.protocol import EngineClient
12 +from vllm.entrypoints.chat_utils import ChatTemplateContentFormatOption
13 +from vllm.entrypoints.logger import RequestLogger
14 +from vllm.entrypoints.openai.protocol import (ErrorResponse,
15 + PoolingChatRequest,
16 + PoolingRequest, PoolingResponse,
17 + PoolingResponseData, UsageInfo)
18 +from vllm.entrypoints.openai.serving_engine import BaseModelPath, OpenAIServing
19 +from vllm.logger import init_logger
20 +from vllm.outputs import PoolingOutput, PoolingRequestOutput
21 +from vllm.utils import merge_async_iterators
22 +
23 +logger = init_logger(__name__)
24 +
25 +
26 +def _get_data(
27 + output: PoolingOutput,
28 + encoding_format: Literal["float", "base64"],
29 +) -> Union[List[float], str]:
30 + if encoding_format == "float":
31 + return output.data.tolist()
32 + elif encoding_format == "base64":
33 + # Force to use float32 for base64 encoding
34 + # to match the OpenAI python client behavior
35 + pooling_bytes = np.array(output.data, dtype="float32").tobytes()
36 + return base64.b64encode(pooling_bytes).decode("utf-8")
37 +
38 + assert_never(encoding_format)
39 +
40 +
41 +class OpenAIServingPooling(OpenAIServing):
42 +
43 + def __init__(
44 + self,
45 + engine_client: EngineClient,
46 + model_config: ModelConfig,
47 + base_model_paths: List[BaseModelPath],
48 + *,
49 + request_logger: Optional[RequestLogger],
50 + chat_template: Optional[str],
51 + chat_template_content_format: ChatTemplateContentFormatOption,
52 + ) -> None:
53 + super().__init__(engine_client=engine_client,
54 + model_config=model_config,
55 + base_model_paths=base_model_paths,
56 + lora_modules=None,
57 + prompt_adapters=None,
58 + request_logger=request_logger)
59 +
60 + self.chat_template = chat_template
61 + self.chat_template_content_format: Final = chat_template_content_format
62 +
63 + async def create_pooling(
64 + self,
65 + request: PoolingRequest,
66 + raw_request: Optional[Request] = None,
67 + ) -> Union[PoolingResponse, ErrorResponse]:
68 + """
69 + See https://platform.openai.com/docs/api-reference/embeddings/create
70 + for the API specification. This API mimics the OpenAI Embedding API.
71 + """
72 + error_check_ret = await self._check_model(request)
73 + if error_check_ret is not None:
74 + return error_check_ret
75 +
76 + encoding_format = request.encoding_format
77 + if request.dimensions is not None:
78 + return self.create_error_response(
79 + "dimensions is currently not supported")
80 +
81 + model_name = request.model
82 + request_id = f"pool-{self._base_request_id(raw_request)}"
83 + created_time = int(time.time())
84 +
85 + truncate_prompt_tokens = None
86 +
87 + if request.truncate_prompt_tokens is not None:
88 + if request.truncate_prompt_tokens <= self.max_model_len:
89 + truncate_prompt_tokens = request.truncate_prompt_tokens
90 + else:
91 + return self.create_error_response(
92 + "truncate_prompt_tokens value is "
93 + "greater than max_model_len."
94 + " Please, select a smaller truncation size.")
95 +
96 + try:
97 + (
98 + lora_request,
99 + prompt_adapter_request,
100 + ) = self._maybe_get_adapters(request)
101 +
102 + tokenizer = await self.engine_client.get_tokenizer(lora_request)
103 +
104 + if prompt_adapter_request is not None:
105 + raise NotImplementedError("Prompt adapter is not supported "
106 + "for pooling models")
107 +
108 + if isinstance(request, PoolingChatRequest):
109 + (
110 + _,
111 + request_prompts,
112 + engine_prompts,
113 + ) = await self._preprocess_chat(
114 + request,
115 + tokenizer,
116 + request.messages,
117 + chat_template=request.chat_template or self.chat_template,
118 + chat_template_content_format=self.
119 + chat_template_content_format,
120 + # In pooling requests, we are not generating tokens,
121 + # so there is no need to append extra tokens to the input
122 + add_generation_prompt=False,
123 + continue_final_message=False,
124 + truncate_prompt_tokens=truncate_prompt_tokens,
125 + add_special_tokens=request.add_special_tokens,
126 + )
127 + else:
128 + (request_prompts,
129 + engine_prompts) = await self._preprocess_completion(
130 + request,
131 + tokenizer,
132 + request.input,
133 + truncate_prompt_tokens=truncate_prompt_tokens,
134 + add_special_tokens=request.add_special_tokens,
135 + )
136 + except ValueError as e:
137 + logger.exception("Error in preprocessing prompt inputs")
138 + return self.create_error_response(str(e))
139 +
140 + # Schedule the request and get the result generator.
141 + generators: List[AsyncGenerator[PoolingRequestOutput, None]] = []
142 + try:
143 + pooling_params = request.to_pooling_params()
144 +
145 + for i, engine_prompt in enumerate(engine_prompts):
146 + request_id_item = f"{request_id}-{i}"
147 +
148 + self._log_inputs(request_id_item,
149 + request_prompts[i],
150 + params=pooling_params,
151 + lora_request=lora_request,
152 + prompt_adapter_request=prompt_adapter_request)
153 +
154 + trace_headers = (None if raw_request is None else await
155 + self._get_trace_headers(raw_request.headers))
156 +
157 + generator = self.engine_client.encode(
158 + engine_prompt,
159 + pooling_params,
160 + request_id_item,
161 + lora_request=lora_request,
162 + trace_headers=trace_headers,
163 + priority=request.priority,
164 + )
165 +
166 + generators.append(generator)
167 + except ValueError as e:
168 + # TODO: Use a vllm-specific Validation Error
169 + return self.create_error_response(str(e))
170 +
171 + result_generator = merge_async_iterators(*generators)
172 +
173 + num_prompts = len(engine_prompts)
174 +
175 + # Non-streaming response
176 + final_res_batch: List[Optional[PoolingRequestOutput]]
177 + final_res_batch = [None] * num_prompts
178 + try:
179 + async for i, res in result_generator:
180 + final_res_batch[i] = res
181 +
182 + assert all(final_res is not None for final_res in final_res_batch)
183 +
184 + final_res_batch_checked = cast(List[PoolingRequestOutput],
185 + final_res_batch)
186 +
187 + response = self.request_output_to_pooling_response(
188 + final_res_batch_checked,
189 + request_id,
190 + created_time,
191 + model_name,
192 + encoding_format,
193 + )
194 + except asyncio.CancelledError:
195 + return self.create_error_response("Client disconnected")
196 + except ValueError as e:
197 + # TODO: Use a vllm-specific Validation Error
198 + return self.create_error_response(str(e))
199 +
200 + return response
201 +
202 + def request_output_to_pooling_response(
203 + self,
204 + final_res_batch: List[PoolingRequestOutput],
205 + request_id: str,
206 + created_time: int,
207 + model_name: str,
208 + encoding_format: Literal["float", "base64"],
209 + ) -> PoolingResponse:
210 + items: List[PoolingResponseData] = []
211 + num_prompt_tokens = 0
212 +
213 + for idx, final_res in enumerate(final_res_batch):
214 + item = PoolingResponseData(
215 + index=idx,
216 + data=_get_data(final_res.outputs, encoding_format),
217 + )
218 + prompt_token_ids = final_res.prompt_token_ids
219 +
220 + items.append(item)
221 + num_prompt_tokens += len(prompt_token_ids)
222 +
223 + usage = UsageInfo(
224 + prompt_tokens=num_prompt_tokens,
225 + total_tokens=num_prompt_tokens,
226 + )
227 +
228 + return PoolingResponse(
229 + id=request_id,
230 + created=created_time,
231 + model=model_name,
232 + data=items,
233 + usage=usage,
234 + )
vllm/entrypoints/openai/serving_score.py
+42 -29
@@ -20,32 +20,6 @@ from vllm.utils import make_async, merge_async_iterators
20 logger = init_logger(__name__)
21
22
23 -def request_output_to_score_response(
24 - final_res_batch: List[PoolingRequestOutput], request_id: str,
25 - created_time: int, model_name: str) -> ScoreResponse:
26 - data: List[ScoreResponseData] = []
27 - num_prompt_tokens = 0
28 - for idx, final_res in enumerate(final_res_batch):
29 - classify_res = ScoringRequestOutput.from_base(final_res)
30 -
31 - score_data = ScoreResponseData(index=idx,
32 - score=classify_res.outputs.score)
33 - data.append(score_data)
34 -
35 - usage = UsageInfo(
36 - prompt_tokens=num_prompt_tokens,
37 - total_tokens=num_prompt_tokens,
38 - )
39 -
40 - return ScoreResponse(
41 - id=request_id,
42 - created=created_time,
43 - model=model_name,
44 - data=data,
45 - usage=usage,
46 - )
47 -
48 -
23 def make_pairs(text_1: Union[List[str], str], text_2: Union[List[str],
24 str]) -> List:
25 if isinstance(text_1, (str, dict)):
@@ -103,7 +77,7 @@ class OpenAIServingScores(OpenAIServing):
77
78 model_name = request.model
79 request_id = f"score-{self._base_request_id(raw_request)}"
106 - created_time = int(time.monotonic())
80 + created_time = int(time.time())
81 truncate_prompt_tokens = request.truncate_prompt_tokens
82
83 request_prompts = []
@@ -203,8 +177,12 @@ class OpenAIServingScores(OpenAIServing):
177 final_res_batch_checked = cast(List[PoolingRequestOutput],
178 final_res_batch)
179
206 - response = request_output_to_score_response(
207 - final_res_batch_checked, request_id, created_time, model_name)
180 + response = self.request_output_to_score_response(
181 + final_res_batch_checked,
182 + request_id,
183 + created_time,
184 + model_name,
185 + )
186 except asyncio.CancelledError:
187 return self.create_error_response("Client disconnected")
188 except ValueError as e:
@@ -212,3 +190,38 @@ class OpenAIServingScores(OpenAIServing):
190 return self.create_error_response(str(e))
191
192 return response
193 +
194 + def request_output_to_score_response(
195 + self,
196 + final_res_batch: List[PoolingRequestOutput],
197 + request_id: str,
198 + created_time: int,
199 + model_name: str,
200 + ) -> ScoreResponse:
201 + items: List[ScoreResponseData] = []
202 + num_prompt_tokens = 0
203 +
204 + for idx, final_res in enumerate(final_res_batch):
205 + classify_res = ScoringRequestOutput.from_base(final_res)
206 +
207 + item = ScoreResponseData(
208 + index=idx,
209 + score=classify_res.outputs.score,
210 + )
211 + prompt_token_ids = final_res.prompt_token_ids
212 +
213 + items.append(item)
214 + num_prompt_tokens += len(prompt_token_ids)
215 +
216 + usage = UsageInfo(
217 + prompt_tokens=num_prompt_tokens,
218 + total_tokens=num_prompt_tokens,
219 + )
220 +
221 + return ScoreResponse(
222 + id=request_id,
223 + created=created_time,
224 + model=model_name,
225 + data=items,
226 + usage=usage,
227 + )
vllm/outputs.py
+2 -1
@@ -355,7 +355,8 @@ class PoolingRequestOutput(Generic[_O]):
355 pooled_data = seq_group.pooled_data
356 assert pooled_data is not None
357
358 - output = PoolingOutput(pooled_data)
358 + data = pooled_data.to(dtype=torch.float32, device="cpu")
359 + output = PoolingOutput(data)
360 prompt_token_ids = seq_group.prompt_token_ids
361 finished = seq_group.is_finished()
362