@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 249b88228d
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 19:30 UTC

[Frontend] Support embeddings in the run_batch API (#7132)

Co-authored-by: Simon Mo <simon.mo@hey.com>

Pooya Davoodi committed Aug 9, 2024 at 09:48 UTC 249b88228d1d371a5830c3394be010d51c7e5cbf
5 files changed +138 -24
examples/offline_inference_openai.md
+37 -4
@@ -10,7 +10,7 @@
10
11 Each line represents a separate request. See the [OpenAI package reference](https://platform.openai.com/docs/api-reference/batch/requestInput) for more details.
12
13 - **NOTE:** We currently only support to `/v1/chat/completions` endpoint (embeddings and completions coming soon).
13 + **NOTE:** We currently only support `/v1/chat/completions` and `/v1/embeddings` endpoints (completions coming soon).
14
15 ## Pre-requisites
16
@@ -21,7 +21,7 @@
21 - Get access to the gated model by [visiting the model card](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) and agreeing to the terms and conditions.
22
23
24 - ## Example: Running with a local file
24 + ## Example 1: Running with a local file
25
26 ### Step 1: Create your batch file
27
@@ -54,7 +54,7 @@ python -m vllm.entrypoints.openai.run_batch -i openai_example_batch.jsonl -o res
54 You should now have your results at `results.jsonl`. You can check your results by running `cat results.jsonl`
55
56 ```
57 -$ cat ../results.jsonl
57 +$ cat results.jsonl
58 {"id":"vllm-383d1c59835645aeb2e07d004d62a826","custom_id":"request-1","response":{"id":"cmpl-61c020e54b964d5a98fa7527bfcdd378","object":"chat.completion","created":1715633336,"model":"meta-llama/Meta-Llama-3-8B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"Hello! It's great to meet you! I'm here to help with any questions or tasks you may have. What's on your mind today?"},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":25,"total_tokens":56,"completion_tokens":31}},"error":null}
59 {"id":"vllm-42e3d09b14b04568afa3f1797751a267","custom_id":"request-2","response":{"id":"cmpl-f44d049f6b3a42d4b2d7850bb1e31bcc","object":"chat.completion","created":1715633336,"model":"meta-llama/Meta-Llama-3-8B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"*silence*"},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":27,"total_tokens":32,"completion_tokens":5}},"error":null}
60 ```
@@ -107,7 +107,7 @@ aws s3 cp openai_example_batch.jsonl s3://MY_BUCKET/MY_INPUT_FILE.jsonl
107
108 ### Step 2: Generate your presigned urls
109
110 -Presigned put urls can only be generated via the SDK. You can run the following python script to generate your presigned urls. Be sure to replace the `MY_BUCKET`, `MY_INPUT_FILE.jsonl`, and `MY_OUTPUT_FILE.jsonl` placeholders with your bucket and file names.
110 +Presigned urls can only be generated via the SDK. You can run the following python script to generate your presigned urls. Be sure to replace the `MY_BUCKET`, `MY_INPUT_FILE.jsonl`, and `MY_OUTPUT_FILE.jsonl` placeholders with your bucket and file names.
111
112 (The script is adapted from https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/python/example_code/s3/s3_basics/presigned_url.py)
113
@@ -170,3 +170,36 @@ Your results are now on S3. You can view them in your terminal by running
170 ```
171 aws s3 cp s3://MY_BUCKET/MY_OUTPUT_FILE.jsonl -
172 ```
173 +
174 +## Example 4: Using embeddings endpoint
175 +
176 +### Additional prerequisites
177 +
178 +* Ensure you are using `vllm >= 0.5.5`.
179 +
180 +### Step 1: Create your batch file
181 +
182 + Add embedding requests to your batch file. The following is an example:
183 +
184 + ```
185 + {"custom_id": "request-1", "method": "POST", "url": "/v1/embeddings", "body": {"model": "intfloat/e5-mistral-7b-instruct", "input": "You are a helpful assistant."}}
186 +{"custom_id": "request-2", "method": "POST", "url": "/v1/embeddings", "body": {"model": "intfloat/e5-mistral-7b-instruct", "input": "You are an unhelpful assistant."}}
187 +```
188 +
189 + You can even mix chat completion and embedding requests in the batch file, as long as the model you are using supports both chat completion and embeddings (note that all requests must use the same model).
190 +
191 +
192 + ### Step 2: Run the batch
193 +
194 +You can run the batch using the same command as in earlier examples.
195 +
196 +
197 +### Step 3: Check your results
198 +
199 +You can check your results by running `cat results.jsonl`
200 +
201 +```
202 +$ cat results.jsonl
203 +{"id":"vllm-db0f71f7dec244e6bce530e0b4ef908b","custom_id":"request-1","response":{"status_code":200,"request_id":"vllm-batch-3580bf4d4ae54d52b67eee266a6eab20","body":{"id":"embd-33ac2efa7996430184461f2e38529746","object":"list","created":444647,"model":"intfloat/e5-mistral-7b-instruct","data":[{"index":0,"object":"embedding","embedding":[0.016204833984375,0.0092010498046875,0.0018358230590820312,-0.0028228759765625,0.001422882080078125,-0.0031147003173828125,...]}],"usage":{"prompt_tokens":8,"total_tokens":8,"completion_tokens":0}}},"error":null}
204 +...```
205 +```
tests/entrypoints/openai/test_run_batch.py
+50 -2
@@ -7,13 +7,39 @@ from vllm.entrypoints.openai.protocol import BatchRequestOutput
7 # ruff: noqa: E501
8 INPUT_BATCH = """{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "NousResearch/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}
9 {"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "NousResearch/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}
10 +
11 {"custom_id": "request-3", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "NonExistModel", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}"""
12
13 INVALID_INPUT_BATCH = """{"invalid_field": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "NousResearch/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}
14 {"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "NousResearch/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}"""
15
16 +INPUT_EMBEDDING_BATCH = """{"custom_id": "request-1", "method": "POST", "url": "/v1/embeddings", "body": {"model": "intfloat/e5-mistral-7b-instruct", "input": "You are a helpful assistant."}}
17 +{"custom_id": "request-2", "method": "POST", "url": "/v1/embeddings", "body": {"model": "intfloat/e5-mistral-7b-instruct", "input": "You are an unhelpful assistant."}}
18 +
19 +{"custom_id": "request-3", "method": "POST", "url": "/v1/embeddings", "body": {"model": "intfloat/e5-mistral-7b-instruct", "input": "Hello world!"}}
20 +{"custom_id": "request-4", "method": "POST", "url": "/v1/embeddings", "body": {"model": "NonExistModel", "input": "Hello world!"}}"""
21 +
22
16 -def test_e2e():
23 +def test_empty_file():
24 + with tempfile.NamedTemporaryFile(
25 + "w") as input_file, tempfile.NamedTemporaryFile(
26 + "r") as output_file:
27 + input_file.write("")
28 + input_file.flush()
29 + proc = subprocess.Popen([
30 + sys.executable, "-m", "vllm.entrypoints.openai.run_batch", "-i",
31 + input_file.name, "-o", output_file.name, "--model",
32 + "intfloat/e5-mistral-7b-instruct"
33 + ], )
34 + proc.communicate()
35 + proc.wait()
36 + assert proc.returncode == 0, f"{proc=}"
37 +
38 + contents = output_file.read()
39 + assert contents.strip() == ""
40 +
41 +
42 +def test_completions():
43 with tempfile.NamedTemporaryFile(
44 "w") as input_file, tempfile.NamedTemporaryFile(
45 "r") as output_file:
@@ -35,7 +61,7 @@ def test_e2e():
61 BatchRequestOutput.model_validate_json(line)
62
63
38 -def test_e2e_invalid_input():
64 +def test_completions_invalid_input():
65 """
66 Ensure that we fail when the input doesn't conform to the openai api.
67 """
@@ -52,3 +78,25 @@ def test_e2e_invalid_input():
78 proc.communicate()
79 proc.wait()
80 assert proc.returncode != 0, f"{proc=}"
81 +
82 +
83 +def test_embeddings():
84 + with tempfile.NamedTemporaryFile(
85 + "w") as input_file, tempfile.NamedTemporaryFile(
86 + "r") as output_file:
87 + input_file.write(INPUT_EMBEDDING_BATCH)
88 + input_file.flush()
89 + proc = subprocess.Popen([
90 + sys.executable, "-m", "vllm.entrypoints.openai.run_batch", "-i",
91 + input_file.name, "-o", output_file.name, "--model",
92 + "intfloat/e5-mistral-7b-instruct"
93 + ], )
94 + proc.communicate()
95 + proc.wait()
96 + assert proc.returncode == 0, f"{proc=}"
97 +
98 + contents = output_file.read()
99 + for line in contents.strip().split("\n"):
100 + # Ensure that the output format conforms to the openai api.
101 + # Validation should throw if the schema is wrong.
102 + BatchRequestOutput.model_validate_json(line)
vllm/entrypoints/openai/protocol.py
+2 -2
@@ -672,7 +672,7 @@ class BatchRequestInput(OpenAIBaseModel):
672 url: str
673
674 # The parameters of the request.
675 - body: ChatCompletionRequest
675 + body: Union[ChatCompletionRequest, EmbeddingRequest]
676
677
678 class BatchResponseData(OpenAIBaseModel):
@@ -683,7 +683,7 @@ class BatchResponseData(OpenAIBaseModel):
683 request_id: str
684
685 # The body of the response.
686 - body: Optional[ChatCompletionResponse] = None
686 + body: Optional[Union[ChatCompletionResponse, EmbeddingResponse]] = None
687
688
689 class BatchRequestOutput(OpenAIBaseModel):
vllm/entrypoints/openai/run_batch.py
+37 -11
@@ -1,18 +1,21 @@
1 import asyncio
2 from io import StringIO
3 -from typing import Awaitable, List
3 +from typing import Awaitable, Callable, List
4
5 import aiohttp
6
7 from vllm.engine.arg_utils import AsyncEngineArgs, nullable_str
8 from vllm.engine.async_llm_engine import AsyncLLMEngine
9 from vllm.entrypoints.logger import RequestLogger
10 +# yapf: disable
11 from vllm.entrypoints.openai.protocol import (BatchRequestInput,
12 BatchRequestOutput,
13 BatchResponseData,
14 ChatCompletionResponse,
14 - ErrorResponse)
15 + EmbeddingResponse, ErrorResponse)
16 +# yapf: enable
17 from vllm.entrypoints.openai.serving_chat import OpenAIServingChat
18 +from vllm.entrypoints.openai.serving_embedding import OpenAIServingEmbedding
19 from vllm.logger import init_logger
20 from vllm.usage.usage_lib import UsageContext
21 from vllm.utils import FlexibleArgumentParser, random_uuid
@@ -82,27 +85,26 @@ async def write_file(path_or_url: str, data: str) -> None:
85 f.write(data)
86
87
85 -async def run_request(chat_serving: OpenAIServingChat,
88 +async def run_request(serving_engine_func: Callable,
89 request: BatchRequestInput) -> BatchRequestOutput:
87 - chat_request = request.body
88 - chat_response = await chat_serving.create_chat_completion(chat_request)
90 + response = await serving_engine_func(request.body)
91
90 - if isinstance(chat_response, ChatCompletionResponse):
92 + if isinstance(response, (ChatCompletionResponse, EmbeddingResponse)):
93 batch_output = BatchRequestOutput(
94 id=f"vllm-{random_uuid()}",
95 custom_id=request.custom_id,
96 response=BatchResponseData(
95 - body=chat_response, request_id=f"vllm-batch-{random_uuid()}"),
97 + body=response, request_id=f"vllm-batch-{random_uuid()}"),
98 error=None,
99 )
98 - elif isinstance(chat_response, ErrorResponse):
100 + elif isinstance(response, ErrorResponse):
101 batch_output = BatchRequestOutput(
102 id=f"vllm-{random_uuid()}",
103 custom_id=request.custom_id,
104 response=BatchResponseData(
103 - status_code=chat_response.code,
105 + status_code=response.code,
106 request_id=f"vllm-batch-{random_uuid()}"),
105 - error=chat_response,
107 + error=response,
108 )
109 else:
110 raise ValueError("Request must not be sent in stream mode")
@@ -128,6 +130,7 @@ async def main(args):
130 else:
131 request_logger = RequestLogger(max_log_len=args.max_log_len)
132
133 + # Create the openai serving objects.
134 openai_serving_chat = OpenAIServingChat(
135 engine,
136 model_config,
@@ -138,12 +141,35 @@ async def main(args):
141 request_logger=request_logger,
142 chat_template=None,
143 )
144 + openai_serving_embedding = OpenAIServingEmbedding(
145 + engine,
146 + model_config,
147 + served_model_names,
148 + request_logger=request_logger,
149 + )
150
151 # Submit all requests in the file to the engine "concurrently".
152 response_futures: List[Awaitable[BatchRequestOutput]] = []
153 for request_json in (await read_file(args.input_file)).strip().split("\n"):
154 + # Skip empty lines.
155 + request_json = request_json.strip()
156 + if not request_json:
157 + continue
158 +
159 request = BatchRequestInput.model_validate_json(request_json)
146 - response_futures.append(run_request(openai_serving_chat, request))
160 +
161 + # Determine the type of request and run it.
162 + if request.url == "/v1/chat/completions":
163 + response_futures.append(
164 + run_request(openai_serving_chat.create_chat_completion,
165 + request))
166 + elif request.url == "/v1/embeddings":
167 + response_futures.append(
168 + run_request(openai_serving_embedding.create_embedding,
169 + request))
170 + else:
171 + raise ValueError("Only /v1/chat/completions and /v1/embeddings are"
172 + "supported in the batch endpoint.")
173
174 responses = await asyncio.gather(*response_futures)
175
vllm/entrypoints/openai/serving_embedding.py
+12 -5
@@ -1,7 +1,8 @@
1 import asyncio
2 import base64
3 import time
4 -from typing import AsyncGenerator, AsyncIterator, List, Optional, Tuple, cast
4 +from typing import (AsyncGenerator, AsyncIterator, List, Optional, Tuple,
5 + Union, cast)
6
7 import numpy as np
8 from fastapi import Request
@@ -11,7 +12,8 @@ from vllm.engine.protocol import AsyncEngineClient
12 from vllm.entrypoints.logger import RequestLogger
13 from vllm.entrypoints.openai.protocol import (EmbeddingRequest,
14 EmbeddingResponse,
14 - EmbeddingResponseData, UsageInfo)
15 + EmbeddingResponseData,
16 + ErrorResponse, UsageInfo)
17 from vllm.entrypoints.openai.serving_engine import OpenAIServing
18 from vllm.logger import init_logger
19 from vllm.outputs import EmbeddingRequestOutput
@@ -71,8 +73,11 @@ class OpenAIServingEmbedding(OpenAIServing):
73 request_logger=request_logger)
74 self._check_embedding_mode(model_config.embedding_mode)
75
74 - async def create_embedding(self, request: EmbeddingRequest,
75 - raw_request: Request):
76 + async def create_embedding(
77 + self,
78 + request: EmbeddingRequest,
79 + raw_request: Optional[Request] = None
80 + ) -> Union[ErrorResponse, EmbeddingResponse]:
81 """Completion API similar to OpenAI's API.
82
83 See https://platform.openai.com/docs/api-reference/embeddings/create
@@ -140,7 +145,9 @@ class OpenAIServingEmbedding(OpenAIServing):
145
146 result_generator: AsyncIterator[Tuple[
147 int, EmbeddingRequestOutput]] = merge_async_iterators(
143 - *generators, is_cancelled=raw_request.is_disconnected)
148 + *generators,
149 + is_cancelled=raw_request.is_disconnected
150 + if raw_request else None)
151
152 # Non-streaming response
153 final_res_batch: List[Optional[EmbeddingRequestOutput]]