A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 01:45 UTC
[Frontend] Support OpenAI batch file format (#4794)
Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>
Alex Wu committed
May 15, 2024 at 19:13 UTC
52f8107cf2e5b3cc1a6a4a96c22b24505f02df01
7 files changed
+415
-3
examples/offline_inference_openai.md
new
+172
@@ -0,0 +1,172 @@
1
+# Offline Inference with the OpenAI Batch file format
2
+
3
+ **NOTE:** This is a guide to performing batch inference using the OpenAI batch file format, **NOT** the complete Batch (REST) API.
4
+
5
+ ## File Format
6
+
7
+ The OpenAI batch file format consists of a series of json objects on new lines.
8
+
9
+ [See here for an example file.](https://github.com/vllm-project/vllm/blob/main/examples/openai_example_batch.jsonl)
10
+
11
+ Each line represents a separate request. See the [OpenAI package reference](https://platform.openai.com/docs/api-reference/batch/requestInput) for more details.
12
+
13
+ **NOTE:** We currently only support to `/v1/chat/completions` endpoint (embeddings and completions coming soon).
14
+
15
+ ## Pre-requisites
16
+
17
+* Ensure you are using `vllm >= 0.4.3`. You can check by running `python -c "import vllm; print(vllm.__version__)"`.
18
+* The examples in this document use `meta-llama/Meta-Llama-3-8B-Instruct`.
19
+ - Create a [user access token](https://huggingface.co/docs/hub/en/security-tokens)
20
+ - Install the token on your machine (Run `huggingface-cli login`).
21
+ - Get access to the gated model by [visiting the model card](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) and agreeing to the terms and conditions.
22
+
23
+
24
+ ## Example: Running with a local file
25
+
26
+ ### Step 1: Create your batch file
27
+
28
+ To follow along with this example, you can download the example batch, or create your own batch file in your working directory.
29
+
30
+ ```
31
+ wget https://raw.githubusercontent.com/vllm-project/vllm/main/examples/openai_example_batch.jsonl
32
+ ```
33
+
34
+ Once you've created your batch file it should look like this
35
+
36
+ ```
37
+ $ cat openai_example_batch.jsonl
38
+{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}
39
+{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}
40
+ ```
41
+
42
+ ### Step 2: Run the batch
43
+
44
+The batch running tool is designed to be used from the command line.
45
+
46
+You can run the batch with the following command, which will write its results to a file called `results.jsonl`
47
+
48
+```
49
+python -m vllm.entrypoints.openai.run_batch -i openai_example_batch.jsonl -o results.jsonl --model meta-llama/Meta-Llama-3-8B-Instruct
50
+```
51
+
52
+### Step 3: Check your results
53
+
54
+You should now have your results at `results.jsonl`. You can check your results by running `cat results.jsonl`
55
+
56
+```
57
+$ cat ../results.jsonl
58
+{"id":"vllm-383d1c59835645aeb2e07d004d62a826","custom_id":"request-1","response":{"id":"cmpl-61c020e54b964d5a98fa7527bfcdd378","object":"chat.completion","created":1715633336,"model":"meta-llama/Meta-Llama-3-8B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"Hello! It's great to meet you! I'm here to help with any questions or tasks you may have. What's on your mind today?"},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":25,"total_tokens":56,"completion_tokens":31}},"error":null}
59
+{"id":"vllm-42e3d09b14b04568afa3f1797751a267","custom_id":"request-2","response":{"id":"cmpl-f44d049f6b3a42d4b2d7850bb1e31bcc","object":"chat.completion","created":1715633336,"model":"meta-llama/Meta-Llama-3-8B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"*silence*"},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":27,"total_tokens":32,"completion_tokens":5}},"error":null}
60
+```
61
+
62
+## Example 2: Using remote files
63
+
64
+The batch runner supports remote input and output urls that are accessible via http/https.
65
+
66
+For example, to run against our example input file located at `https://raw.githubusercontent.com/vllm-project/vllm/main/examples/openai_example_batch.jsonl`, you can run
67
+
68
+```
69
+python -m vllm.entrypoints.openai.run_batch -i https://raw.githubusercontent.com/vllm-project/vllm/main/examples/openai_example_batch.jsonl -o results.jsonl --model meta-llama/Meta-Llama-3-8B-Instruct
70
+```
71
+
72
+## Example 3: Integrating with AWS S3
73
+
74
+To integrate with cloud blob storage, we recommend using presigned urls.
75
+
76
+[Learn more about S3 presigned urls here]
77
+
78
+### Additional prerequisites
79
+
80
+* [Create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html).
81
+* The `awscli` package (Run `pip install awscli`) to configure your credentials and interactively use s3.
82
+ - [Configure your credentials](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html).
83
+* The `boto3` python package (Run `pip install boto3`) to generate presigned urls.
84
+
85
+### Step 1: Upload your input script
86
+
87
+To follow along with this example, you can download the example batch, or create your own batch file in your working directory.
88
+
89
+ ```
90
+ wget https://raw.githubusercontent.com/vllm-project/vllm/main/examples/openai_example_batch.jsonl
91
+ ```
92
+
93
+ Once you've created your batch file it should look like this
94
+
95
+ ```
96
+ $ cat openai_example_batch.jsonl
97
+{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}
98
+{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}
99
+ ```
100
+
101
+Now upload your batch file to your S3 bucket.
102
+
103
+```
104
+aws s3 cp openai_example_batch.jsonl s3://MY_BUCKET/MY_INPUT_FILE.jsonl
105
+```
106
+
107
+
108
+### Step 2: Generate your presigned urls
109
+
110
+Presigned put urls can only be generated via the SDK. You can run the following python script to generate your presigned urls. Be sure to replace the `MY_BUCKET`, `MY_INPUT_FILE.jsonl`, and `MY_OUTPUT_FILE.jsonl` placeholders with your bucket and file names.
111
+
112
+(The script is adapted from https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/python/example_code/s3/s3_basics/presigned_url.py)
113
+
114
+```
115
+import boto3
116
+from botocore.exceptions import ClientError
117
+
118
+def generate_presigned_url(s3_client, client_method, method_parameters, expires_in):
119
+ """
120
+ Generate a presigned Amazon S3 URL that can be used to perform an action.
121
+
122
+ :param s3_client: A Boto3 Amazon S3 client.
123
+ :param client_method: The name of the client method that the URL performs.
124
+ :param method_parameters: The parameters of the specified client method.
125
+ :param expires_in: The number of seconds the presigned URL is valid for.
126
+ :return: The presigned URL.
127
+ """
128
+ try:
129
+ url = s3_client.generate_presigned_url(
130
+ ClientMethod=client_method, Params=method_parameters, ExpiresIn=expires_in
131
+ )
132
+ except ClientError:
133
+ raise
134
+ return url
135
+
136
+
137
+s3_client = boto3.client("s3")
138
+input_url = generate_presigned_url(
139
+ s3_client, "get_object", {"Bucket": "MY_BUCKET", "Key": "MY_INPUT_FILE.jsonl"}, 3600
140
+)
141
+output_url = generate_presigned_url(
142
+ s3_client, "put_object", {"Bucket": "MY_BUCKET", "Key": "MY_OUTPUT_FILE.jsonl"}, 3600
143
+)
144
+print(f"{input_url=}")
145
+print(f"{output_url=}")
146
+```
147
+
148
+This script should output
149
+
150
+```
151
+input_url='https://s3.us-west-2.amazonaws.com/MY_BUCKET/MY_INPUT_FILE.jsonl?AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST&Signature=abcdefghijklmnopqrstuvwxyz12345&Expires=1715800091'
152
+output_url='https://s3.us-west-2.amazonaws.com/MY_BUCKET/MY_OUTPUT_FILE.jsonl?AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST&Signature=abcdefghijklmnopqrstuvwxyz12345&Expires=1715800091'
153
+```
154
+
155
+### Step 3: Run the batch runner using your presigned urls
156
+
157
+You can now run the batch runner, using the urls generated in the previous section.
158
+
159
+```
160
+python -m vllm.entrypoints.openai.run_batch \
161
+ -i "https://s3.us-west-2.amazonaws.com/MY_BUCKET/MY_INPUT_FILE.jsonl?AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST&Signature=abcdefghijklmnopqrstuvwxyz12345&Expires=1715800091" \
162
+ -o "https://s3.us-west-2.amazonaws.com/MY_BUCKET/MY_OUTPUT_FILE.jsonl?AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST&Signature=abcdefghijklmnopqrstuvwxyz12345&Expires=1715800091" \
163
+ --model --model meta-llama/Meta-Llama-3-8B-Instruct
164
+```
165
+
166
+### Step 4: View your results
167
+
168
+Your results are now on S3. You can view them in your terminal by running
169
+
170
+```
171
+aws s3 cp s3://MY_BUCKET/MY_OUTPUT_FILE.jsonl -
172
+```
examples/openi_example_batch.jsonl
new
+2
@@ -0,0 +1,2 @@
1
+{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}
2
+{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}
requirements-common.txt
+1
@@ -8,6 +8,7 @@ py-cpuinfo
8
transformers >= 4.40.0 # Required for StarCoder2 & Llava, Llama 3.
9
tokenizers >= 0.19.1 # Required for Llama 3.
10
fastapi
11
+aiohttp
12
openai
13
uvicorn[standard]
14
pydantic >= 2.0 # Required for OpenAI server.
tests/entrypoints/test_openai_run_batch.py
new
+53
@@ -0,0 +1,53 @@
1
+import subprocess
2
+import sys
3
+import tempfile
4
+
5
+from vllm.entrypoints.openai.protocol import BatchRequestOutput
6
+
7
+# ruff: noqa: E501
8
+INPUT_BATCH = """{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "NousResearch/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}
9
+{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "NousResearch/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}"""
10
+
11
+INVALID_INPUT_BATCH = """{"invalid_field": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "NousResearch/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}
12
+{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "NousResearch/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}"""
13
+
14
+
15
+def test_e2e():
16
+ with tempfile.NamedTemporaryFile(
17
+ "w") as input_file, tempfile.NamedTemporaryFile(
18
+ "r") as output_file:
19
+ input_file.write(INPUT_BATCH)
20
+ input_file.flush()
21
+ proc = subprocess.Popen([
22
+ sys.executable, "-m", "vllm.entrypoints.openai.run_batch", "-i",
23
+ input_file.name, "-o", output_file.name, "--model",
24
+ "NousResearch/Meta-Llama-3-8B-Instruct"
25
+ ], )
26
+ proc.communicate()
27
+ proc.wait()
28
+ assert proc.returncode == 0, f"{proc=}"
29
+
30
+ contents = output_file.read()
31
+ for line in contents.strip().split("\n"):
32
+ # Ensure that the output format conforms to the openai api.
33
+ # Validation should throw if the schema is wrong.
34
+ BatchRequestOutput.model_validate_json(line)
35
+
36
+
37
+def test_e2e_invalid_input():
38
+ """
39
+ Ensure that we fail when the input doesn't conform to the openai api.
40
+ """
41
+ with tempfile.NamedTemporaryFile(
42
+ "w") as input_file, tempfile.NamedTemporaryFile(
43
+ "r") as output_file:
44
+ input_file.write(INVALID_INPUT_BATCH)
45
+ input_file.flush()
46
+ proc = subprocess.Popen([
47
+ sys.executable, "-m", "vllm.entrypoints.openai.run_batch", "-i",
48
+ input_file.name, "-o", output_file.name, "--model",
49
+ "NousResearch/Meta-Llama-3-8B-Instruct"
50
+ ], )
51
+ proc.communicate()
52
+ proc.wait()
53
+ assert proc.returncode != 0, f"{proc=}"
vllm/entrypoints/openai/protocol.py
+41
@@ -526,3 +526,44 @@ class ChatCompletionStreamResponse(OpenAIBaseModel):
526
model: str
527
choices: List[ChatCompletionResponseStreamChoice]
528
usage: Optional[UsageInfo] = Field(default=None)
529
+
530
+
531
+class BatchRequestInput(OpenAIBaseModel):
532
+ """
533
+ The per-line object of the batch input file.
534
+
535
+ NOTE: Currently only the `/v1/chat/completions` endpoint is supported.
536
+ """
537
+
538
+ # A developer-provided per-request id that will be used to match outputs to
539
+ # inputs. Must be unique for each request in a batch.
540
+ custom_id: str
541
+
542
+ # The HTTP method to be used for the request. Currently only POST is
543
+ # supported.
544
+ method: str
545
+
546
+ # The OpenAI API relative URL to be used for the request. Currently
547
+ # /v1/chat/completions is supported.
548
+ url: str
549
+
550
+ # The parameteters of the request.
551
+ body: Union[ChatCompletionRequest, ]
552
+
553
+
554
+class BatchRequestOutput(OpenAIBaseModel):
555
+ """
556
+ The per-line object of the batch output and error files
557
+ """
558
+
559
+ id: str
560
+
561
+ # A developer-provided per-request id that will be used to match outputs to
562
+ # inputs.
563
+ custom_id: str
564
+
565
+ response: Optional[ChatCompletionResponse]
566
+
567
+ # For requests that failed with a non-HTTP error, this will contain more
568
+ # information on the cause of the failure.
569
+ error: Optional[Any]
vllm/entrypoints/openai/run_batch.py
new
+141
@@ -0,0 +1,141 @@
1
+import argparse
2
+import asyncio
3
+import sys
4
+from io import StringIO
5
+
6
+import aiohttp
7
+
8
+import vllm
9
+from vllm.engine.arg_utils import AsyncEngineArgs, nullable_str
10
+from vllm.engine.async_llm_engine import AsyncLLMEngine
11
+from vllm.entrypoints.openai.protocol import (BatchRequestInput,
12
+ BatchRequestOutput,
13
+ ChatCompletionResponse)
14
+from vllm.entrypoints.openai.serving_chat import OpenAIServingChat
15
+from vllm.logger import init_logger
16
+from vllm.usage.usage_lib import UsageContext
17
+from vllm.utils import random_uuid
18
+
19
+logger = init_logger(__name__)
20
+
21
+
22
+def parse_args():
23
+ parser = argparse.ArgumentParser(
24
+ description="vLLM OpenAI-Compatible batch runner.")
25
+ parser.add_argument(
26
+ "-i",
27
+ "--input-file",
28
+ required=True,
29
+ type=str,
30
+ help=
31
+ "The path or url to a single input file. Currently supports local file "
32
+ "paths, or the http protocol (http or https). If a URL is specified, "
33
+ "the file should be available via HTTP GET.")
34
+ parser.add_argument(
35
+ "-o",
36
+ "--output-file",
37
+ required=True,
38
+ type=str,
39
+ help="The path or url to a single output file. Currently supports "
40
+ "local file paths, or web (http or https) urls. If a URL is specified,"
41
+ " the file should be available via HTTP PUT.")
42
+ parser.add_argument("--response-role",
43
+ type=nullable_str,
44
+ default="assistant",
45
+ help="The role name to return if "
46
+ "`request.add_generation_prompt=true`.")
47
+
48
+ parser = AsyncEngineArgs.add_cli_args(parser)
49
+ return parser.parse_args()
50
+
51
+
52
+async def read_file(path_or_url: str) -> str:
53
+ if path_or_url.startswith("http://") or path_or_url.startswith("https://"):
54
+ async with aiohttp.ClientSession() as session, \
55
+ session.get(path_or_url) as resp:
56
+ return await resp.text()
57
+ else:
58
+ with open(path_or_url, "r") as f:
59
+ return f.read()
60
+
61
+
62
+async def write_file(path_or_url: str, data: str) -> None:
63
+ if path_or_url.startswith("http://") or path_or_url.startswith("https://"):
64
+ async with aiohttp.ClientSession() as session, \
65
+ session.put(path_or_url, data=data.encode("utf-8")):
66
+ pass
67
+ else:
68
+ # We should make this async, but as long as this is always run as a
69
+ # standalone program, blocking the event loop won't effect performance
70
+ # in this particular case.
71
+ with open(path_or_url, "w") as f:
72
+ f.write(data)
73
+
74
+
75
+async def run_request(chat_serving: OpenAIServingChat,
76
+ request: BatchRequestInput) -> BatchRequestOutput:
77
+ chat_request = request.body
78
+ chat_response = await chat_serving.create_chat_completion(chat_request)
79
+ if isinstance(chat_response, ChatCompletionResponse):
80
+ batch_output = BatchRequestOutput(
81
+ id=f"vllm-{random_uuid()}",
82
+ custom_id=request.custom_id,
83
+ response=chat_response,
84
+ error=None,
85
+ )
86
+ else:
87
+ batch_output = BatchRequestOutput(
88
+ id=f"vllm-{random_uuid()}",
89
+ custom_id=request.custom_id,
90
+ response=None,
91
+ error=chat_response,
92
+ )
93
+ return batch_output
94
+
95
+
96
+async def main(args):
97
+ if args.served_model_name is not None:
98
+ served_model_names = args.served_model_name
99
+ else:
100
+ served_model_names = [args.model]
101
+
102
+ engine_args = AsyncEngineArgs.from_cli_args(args)
103
+ engine = AsyncLLMEngine.from_engine_args(
104
+ engine_args, usage_context=UsageContext.OPENAI_API_SERVER)
105
+
106
+ # When using single vLLM without engine_use_ray
107
+ model_config = await engine.get_model_config()
108
+
109
+ openai_serving_chat = OpenAIServingChat(
110
+ engine,
111
+ model_config,
112
+ served_model_names,
113
+ args.response_role,
114
+ )
115
+
116
+ # Submit all requests in the file to the engine "concurrently".
117
+ response_futures = []
118
+ for request_json in (await read_file(args.input_file)).strip().split("\n"):
119
+ request = BatchRequestInput.model_validate_json(request_json)
120
+ response_futures.append(run_request(openai_serving_chat, request))
121
+
122
+ responses = await asyncio.gather(*response_futures)
123
+
124
+ output_buffer = StringIO()
125
+ for response in responses:
126
+ print(response.model_dump_json(), file=output_buffer)
127
+
128
+ output_buffer.seek(0)
129
+ await write_file(args.output_file, output_buffer.read().strip())
130
+
131
+ # Temporary workaround for https://github.com/vllm-project/vllm/issues/4789
132
+ sys.exit(0)
133
+
134
+
135
+if __name__ == "__main__":
136
+ args = parse_args()
137
+
138
+ logger.info("vLLM API server version %s", vllm.__version__)
139
+ logger.info("args: %s", args)
140
+
141
+ asyncio.run(main(args))
vllm/entrypoints/openai/serving_chat.py
+5
-3
@@ -119,7 +119,9 @@ class OpenAIServingChat(OpenAIServing):
119
return self._parse_chat_message_content_parts(role, content)
120
121
async def create_chat_completion(
122
- self, request: ChatCompletionRequest, raw_request: Request
122
+ self,
123
+ request: ChatCompletionRequest,
124
+ raw_request: Optional[Request] = None
125
) -> Union[ErrorResponse, AsyncGenerator[str, None],
126
ChatCompletionResponse]:
127
"""Completion API similar to OpenAI's API.
@@ -337,7 +339,7 @@ class OpenAIServingChat(OpenAIServing):
339
yield "data: [DONE]\n\n"
340
341
async def chat_completion_full_generator(
340
- self, request: ChatCompletionRequest, raw_request: Request,
342
+ self, request: ChatCompletionRequest, raw_request: Optional[Request],
343
result_generator: AsyncIterator[RequestOutput], request_id: str,
344
conversation: List[ConversationMessage]
345
) -> Union[ErrorResponse, ChatCompletionResponse]:
@@ -347,7 +349,7 @@ class OpenAIServingChat(OpenAIServing):
349
final_res: Optional[RequestOutput] = None
350
351
async for res in result_generator:
350
- if await raw_request.is_disconnected():
352
+ if raw_request is not None and await raw_request.is_disconnected():
353
# Abort the request if the client disconnects.
354
await self.engine.abort(request_id)
355
return self.create_error_response("Client disconnected")