@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 52f8107cf2
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 01:45 UTC

[Frontend] Support OpenAI batch file format (#4794)

Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>

Alex Wu committed May 15, 2024 at 19:13 UTC 52f8107cf2e5b3cc1a6a4a96c22b24505f02df01
7 files changed +415 -3
examples/offline_inference_openai.md new
+172
@@ -0,0 +1,172 @@
1 +# Offline Inference with the OpenAI Batch file format
2 +
3 + **NOTE:** This is a guide to performing batch inference using the OpenAI batch file format, **NOT** the complete Batch (REST) API.
4 +
5 + ## File Format
6 +
7 + The OpenAI batch file format consists of a series of json objects on new lines.
8 +
9 + [See here for an example file.](https://github.com/vllm-project/vllm/blob/main/examples/openai_example_batch.jsonl)
10 +
11 + Each line represents a separate request. See the [OpenAI package reference](https://platform.openai.com/docs/api-reference/batch/requestInput) for more details.
12 +
13 + **NOTE:** We currently only support to `/v1/chat/completions` endpoint (embeddings and completions coming soon).
14 +
15 + ## Pre-requisites
16 +
17 +* Ensure you are using `vllm >= 0.4.3`. You can check by running `python -c "import vllm; print(vllm.__version__)"`.
18 +* The examples in this document use `meta-llama/Meta-Llama-3-8B-Instruct`.
19 + - Create a [user access token](https://huggingface.co/docs/hub/en/security-tokens)
20 + - Install the token on your machine (Run `huggingface-cli login`).
21 + - Get access to the gated model by [visiting the model card](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) and agreeing to the terms and conditions.
22 +
23 +
24 + ## Example: Running with a local file
25 +
26 + ### Step 1: Create your batch file
27 +
28 + To follow along with this example, you can download the example batch, or create your own batch file in your working directory.
29 +
30 + ```
31 + wget https://raw.githubusercontent.com/vllm-project/vllm/main/examples/openai_example_batch.jsonl
32 + ```
33 +
34 + Once you've created your batch file it should look like this
35 +
36 + ```
37 + $ cat openai_example_batch.jsonl
38 +{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}
39 +{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}
40 + ```
41 +
42 + ### Step 2: Run the batch
43 +
44 +The batch running tool is designed to be used from the command line.
45 +
46 +You can run the batch with the following command, which will write its results to a file called `results.jsonl`
47 +
48 +```
49 +python -m vllm.entrypoints.openai.run_batch -i openai_example_batch.jsonl -o results.jsonl --model meta-llama/Meta-Llama-3-8B-Instruct
50 +```
51 +
52 +### Step 3: Check your results
53 +
54 +You should now have your results at `results.jsonl`. You can check your results by running `cat results.jsonl`
55 +
56 +```
57 +$ cat ../results.jsonl
58 +{"id":"vllm-383d1c59835645aeb2e07d004d62a826","custom_id":"request-1","response":{"id":"cmpl-61c020e54b964d5a98fa7527bfcdd378","object":"chat.completion","created":1715633336,"model":"meta-llama/Meta-Llama-3-8B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"Hello! It's great to meet you! I'm here to help with any questions or tasks you may have. What's on your mind today?"},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":25,"total_tokens":56,"completion_tokens":31}},"error":null}
59 +{"id":"vllm-42e3d09b14b04568afa3f1797751a267","custom_id":"request-2","response":{"id":"cmpl-f44d049f6b3a42d4b2d7850bb1e31bcc","object":"chat.completion","created":1715633336,"model":"meta-llama/Meta-Llama-3-8B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"*silence*"},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":27,"total_tokens":32,"completion_tokens":5}},"error":null}
60 +```
61 +
62 +## Example 2: Using remote files
63 +
64 +The batch runner supports remote input and output urls that are accessible via http/https.
65 +
66 +For example, to run against our example input file located at `https://raw.githubusercontent.com/vllm-project/vllm/main/examples/openai_example_batch.jsonl`, you can run
67 +
68 +```
69 +python -m vllm.entrypoints.openai.run_batch -i https://raw.githubusercontent.com/vllm-project/vllm/main/examples/openai_example_batch.jsonl -o results.jsonl --model meta-llama/Meta-Llama-3-8B-Instruct
70 +```
71 +
72 +## Example 3: Integrating with AWS S3
73 +
74 +To integrate with cloud blob storage, we recommend using presigned urls.
75 +
76 +[Learn more about S3 presigned urls here]
77 +
78 +### Additional prerequisites
79 +
80 +* [Create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html).
81 +* The `awscli` package (Run `pip install awscli`) to configure your credentials and interactively use s3.
82 + - [Configure your credentials](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html).
83 +* The `boto3` python package (Run `pip install boto3`) to generate presigned urls.
84 +
85 +### Step 1: Upload your input script
86 +
87 +To follow along with this example, you can download the example batch, or create your own batch file in your working directory.
88 +
89 + ```
90 + wget https://raw.githubusercontent.com/vllm-project/vllm/main/examples/openai_example_batch.jsonl
91 + ```
92 +
93 + Once you've created your batch file it should look like this
94 +
95 + ```
96 + $ cat openai_example_batch.jsonl
97 +{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}
98 +{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}
99 + ```
100 +
101 +Now upload your batch file to your S3 bucket.
102 +
103 +```
104 +aws s3 cp openai_example_batch.jsonl s3://MY_BUCKET/MY_INPUT_FILE.jsonl
105 +```
106 +
107 +
108 +### Step 2: Generate your presigned urls
109 +
110 +Presigned put urls can only be generated via the SDK. You can run the following python script to generate your presigned urls. Be sure to replace the `MY_BUCKET`, `MY_INPUT_FILE.jsonl`, and `MY_OUTPUT_FILE.jsonl` placeholders with your bucket and file names.
111 +
112 +(The script is adapted from https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/python/example_code/s3/s3_basics/presigned_url.py)
113 +
114 +```
115 +import boto3
116 +from botocore.exceptions import ClientError
117 +
118 +def generate_presigned_url(s3_client, client_method, method_parameters, expires_in):
119 + """
120 + Generate a presigned Amazon S3 URL that can be used to perform an action.
121 +
122 + :param s3_client: A Boto3 Amazon S3 client.
123 + :param client_method: The name of the client method that the URL performs.
124 + :param method_parameters: The parameters of the specified client method.
125 + :param expires_in: The number of seconds the presigned URL is valid for.
126 + :return: The presigned URL.
127 + """
128 + try:
129 + url = s3_client.generate_presigned_url(
130 + ClientMethod=client_method, Params=method_parameters, ExpiresIn=expires_in
131 + )
132 + except ClientError:
133 + raise
134 + return url
135 +
136 +
137 +s3_client = boto3.client("s3")
138 +input_url = generate_presigned_url(
139 + s3_client, "get_object", {"Bucket": "MY_BUCKET", "Key": "MY_INPUT_FILE.jsonl"}, 3600
140 +)
141 +output_url = generate_presigned_url(
142 + s3_client, "put_object", {"Bucket": "MY_BUCKET", "Key": "MY_OUTPUT_FILE.jsonl"}, 3600
143 +)
144 +print(f"{input_url=}")
145 +print(f"{output_url=}")
146 +```
147 +
148 +This script should output
149 +
150 +```
151 +input_url='https://s3.us-west-2.amazonaws.com/MY_BUCKET/MY_INPUT_FILE.jsonl?AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST&Signature=abcdefghijklmnopqrstuvwxyz12345&Expires=1715800091'
152 +output_url='https://s3.us-west-2.amazonaws.com/MY_BUCKET/MY_OUTPUT_FILE.jsonl?AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST&Signature=abcdefghijklmnopqrstuvwxyz12345&Expires=1715800091'
153 +```
154 +
155 +### Step 3: Run the batch runner using your presigned urls
156 +
157 +You can now run the batch runner, using the urls generated in the previous section.
158 +
159 +```
160 +python -m vllm.entrypoints.openai.run_batch \
161 + -i "https://s3.us-west-2.amazonaws.com/MY_BUCKET/MY_INPUT_FILE.jsonl?AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST&Signature=abcdefghijklmnopqrstuvwxyz12345&Expires=1715800091" \
162 + -o "https://s3.us-west-2.amazonaws.com/MY_BUCKET/MY_OUTPUT_FILE.jsonl?AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST&Signature=abcdefghijklmnopqrstuvwxyz12345&Expires=1715800091" \
163 + --model --model meta-llama/Meta-Llama-3-8B-Instruct
164 +```
165 +
166 +### Step 4: View your results
167 +
168 +Your results are now on S3. You can view them in your terminal by running
169 +
170 +```
171 +aws s3 cp s3://MY_BUCKET/MY_OUTPUT_FILE.jsonl -
172 +```
examples/openi_example_batch.jsonl new
+2
@@ -0,0 +1,2 @@
1 +{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}
2 +{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}
requirements-common.txt
+1
@@ -8,6 +8,7 @@ py-cpuinfo
8 transformers >= 4.40.0 # Required for StarCoder2 & Llava, Llama 3.
9 tokenizers >= 0.19.1 # Required for Llama 3.
10 fastapi
11 +aiohttp
12 openai
13 uvicorn[standard]
14 pydantic >= 2.0 # Required for OpenAI server.
tests/entrypoints/test_openai_run_batch.py new
+53
@@ -0,0 +1,53 @@
1 +import subprocess
2 +import sys
3 +import tempfile
4 +
5 +from vllm.entrypoints.openai.protocol import BatchRequestOutput
6 +
7 +# ruff: noqa: E501
8 +INPUT_BATCH = """{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "NousResearch/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}
9 +{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "NousResearch/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}"""
10 +
11 +INVALID_INPUT_BATCH = """{"invalid_field": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "NousResearch/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}
12 +{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "NousResearch/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}"""
13 +
14 +
15 +def test_e2e():
16 + with tempfile.NamedTemporaryFile(
17 + "w") as input_file, tempfile.NamedTemporaryFile(
18 + "r") as output_file:
19 + input_file.write(INPUT_BATCH)
20 + input_file.flush()
21 + proc = subprocess.Popen([
22 + sys.executable, "-m", "vllm.entrypoints.openai.run_batch", "-i",
23 + input_file.name, "-o", output_file.name, "--model",
24 + "NousResearch/Meta-Llama-3-8B-Instruct"
25 + ], )
26 + proc.communicate()
27 + proc.wait()
28 + assert proc.returncode == 0, f"{proc=}"
29 +
30 + contents = output_file.read()
31 + for line in contents.strip().split("\n"):
32 + # Ensure that the output format conforms to the openai api.
33 + # Validation should throw if the schema is wrong.
34 + BatchRequestOutput.model_validate_json(line)
35 +
36 +
37 +def test_e2e_invalid_input():
38 + """
39 + Ensure that we fail when the input doesn't conform to the openai api.
40 + """
41 + with tempfile.NamedTemporaryFile(
42 + "w") as input_file, tempfile.NamedTemporaryFile(
43 + "r") as output_file:
44 + input_file.write(INVALID_INPUT_BATCH)
45 + input_file.flush()
46 + proc = subprocess.Popen([
47 + sys.executable, "-m", "vllm.entrypoints.openai.run_batch", "-i",
48 + input_file.name, "-o", output_file.name, "--model",
49 + "NousResearch/Meta-Llama-3-8B-Instruct"
50 + ], )
51 + proc.communicate()
52 + proc.wait()
53 + assert proc.returncode != 0, f"{proc=}"
vllm/entrypoints/openai/protocol.py
+41
@@ -526,3 +526,44 @@ class ChatCompletionStreamResponse(OpenAIBaseModel):
526 model: str
527 choices: List[ChatCompletionResponseStreamChoice]
528 usage: Optional[UsageInfo] = Field(default=None)
529 +
530 +
531 +class BatchRequestInput(OpenAIBaseModel):
532 + """
533 + The per-line object of the batch input file.
534 +
535 + NOTE: Currently only the `/v1/chat/completions` endpoint is supported.
536 + """
537 +
538 + # A developer-provided per-request id that will be used to match outputs to
539 + # inputs. Must be unique for each request in a batch.
540 + custom_id: str
541 +
542 + # The HTTP method to be used for the request. Currently only POST is
543 + # supported.
544 + method: str
545 +
546 + # The OpenAI API relative URL to be used for the request. Currently
547 + # /v1/chat/completions is supported.
548 + url: str
549 +
550 + # The parameteters of the request.
551 + body: Union[ChatCompletionRequest, ]
552 +
553 +
554 +class BatchRequestOutput(OpenAIBaseModel):
555 + """
556 + The per-line object of the batch output and error files
557 + """
558 +
559 + id: str
560 +
561 + # A developer-provided per-request id that will be used to match outputs to
562 + # inputs.
563 + custom_id: str
564 +
565 + response: Optional[ChatCompletionResponse]
566 +
567 + # For requests that failed with a non-HTTP error, this will contain more
568 + # information on the cause of the failure.
569 + error: Optional[Any]
vllm/entrypoints/openai/run_batch.py new
+141
@@ -0,0 +1,141 @@
1 +import argparse
2 +import asyncio
3 +import sys
4 +from io import StringIO
5 +
6 +import aiohttp
7 +
8 +import vllm
9 +from vllm.engine.arg_utils import AsyncEngineArgs, nullable_str
10 +from vllm.engine.async_llm_engine import AsyncLLMEngine
11 +from vllm.entrypoints.openai.protocol import (BatchRequestInput,
12 + BatchRequestOutput,
13 + ChatCompletionResponse)
14 +from vllm.entrypoints.openai.serving_chat import OpenAIServingChat
15 +from vllm.logger import init_logger
16 +from vllm.usage.usage_lib import UsageContext
17 +from vllm.utils import random_uuid
18 +
19 +logger = init_logger(__name__)
20 +
21 +
22 +def parse_args():
23 + parser = argparse.ArgumentParser(
24 + description="vLLM OpenAI-Compatible batch runner.")
25 + parser.add_argument(
26 + "-i",
27 + "--input-file",
28 + required=True,
29 + type=str,
30 + help=
31 + "The path or url to a single input file. Currently supports local file "
32 + "paths, or the http protocol (http or https). If a URL is specified, "
33 + "the file should be available via HTTP GET.")
34 + parser.add_argument(
35 + "-o",
36 + "--output-file",
37 + required=True,
38 + type=str,
39 + help="The path or url to a single output file. Currently supports "
40 + "local file paths, or web (http or https) urls. If a URL is specified,"
41 + " the file should be available via HTTP PUT.")
42 + parser.add_argument("--response-role",
43 + type=nullable_str,
44 + default="assistant",
45 + help="The role name to return if "
46 + "`request.add_generation_prompt=true`.")
47 +
48 + parser = AsyncEngineArgs.add_cli_args(parser)
49 + return parser.parse_args()
50 +
51 +
52 +async def read_file(path_or_url: str) -> str:
53 + if path_or_url.startswith("http://") or path_or_url.startswith("https://"):
54 + async with aiohttp.ClientSession() as session, \
55 + session.get(path_or_url) as resp:
56 + return await resp.text()
57 + else:
58 + with open(path_or_url, "r") as f:
59 + return f.read()
60 +
61 +
62 +async def write_file(path_or_url: str, data: str) -> None:
63 + if path_or_url.startswith("http://") or path_or_url.startswith("https://"):
64 + async with aiohttp.ClientSession() as session, \
65 + session.put(path_or_url, data=data.encode("utf-8")):
66 + pass
67 + else:
68 + # We should make this async, but as long as this is always run as a
69 + # standalone program, blocking the event loop won't effect performance
70 + # in this particular case.
71 + with open(path_or_url, "w") as f:
72 + f.write(data)
73 +
74 +
75 +async def run_request(chat_serving: OpenAIServingChat,
76 + request: BatchRequestInput) -> BatchRequestOutput:
77 + chat_request = request.body
78 + chat_response = await chat_serving.create_chat_completion(chat_request)
79 + if isinstance(chat_response, ChatCompletionResponse):
80 + batch_output = BatchRequestOutput(
81 + id=f"vllm-{random_uuid()}",
82 + custom_id=request.custom_id,
83 + response=chat_response,
84 + error=None,
85 + )
86 + else:
87 + batch_output = BatchRequestOutput(
88 + id=f"vllm-{random_uuid()}",
89 + custom_id=request.custom_id,
90 + response=None,
91 + error=chat_response,
92 + )
93 + return batch_output
94 +
95 +
96 +async def main(args):
97 + if args.served_model_name is not None:
98 + served_model_names = args.served_model_name
99 + else:
100 + served_model_names = [args.model]
101 +
102 + engine_args = AsyncEngineArgs.from_cli_args(args)
103 + engine = AsyncLLMEngine.from_engine_args(
104 + engine_args, usage_context=UsageContext.OPENAI_API_SERVER)
105 +
106 + # When using single vLLM without engine_use_ray
107 + model_config = await engine.get_model_config()
108 +
109 + openai_serving_chat = OpenAIServingChat(
110 + engine,
111 + model_config,
112 + served_model_names,
113 + args.response_role,
114 + )
115 +
116 + # Submit all requests in the file to the engine "concurrently".
117 + response_futures = []
118 + for request_json in (await read_file(args.input_file)).strip().split("\n"):
119 + request = BatchRequestInput.model_validate_json(request_json)
120 + response_futures.append(run_request(openai_serving_chat, request))
121 +
122 + responses = await asyncio.gather(*response_futures)
123 +
124 + output_buffer = StringIO()
125 + for response in responses:
126 + print(response.model_dump_json(), file=output_buffer)
127 +
128 + output_buffer.seek(0)
129 + await write_file(args.output_file, output_buffer.read().strip())
130 +
131 + # Temporary workaround for https://github.com/vllm-project/vllm/issues/4789
132 + sys.exit(0)
133 +
134 +
135 +if __name__ == "__main__":
136 + args = parse_args()
137 +
138 + logger.info("vLLM API server version %s", vllm.__version__)
139 + logger.info("args: %s", args)
140 +
141 + asyncio.run(main(args))
vllm/entrypoints/openai/serving_chat.py
+5 -3
@@ -119,7 +119,9 @@ class OpenAIServingChat(OpenAIServing):
119 return self._parse_chat_message_content_parts(role, content)
120
121 async def create_chat_completion(
122 - self, request: ChatCompletionRequest, raw_request: Request
122 + self,
123 + request: ChatCompletionRequest,
124 + raw_request: Optional[Request] = None
125 ) -> Union[ErrorResponse, AsyncGenerator[str, None],
126 ChatCompletionResponse]:
127 """Completion API similar to OpenAI's API.
@@ -337,7 +339,7 @@ class OpenAIServingChat(OpenAIServing):
339 yield "data: [DONE]\n\n"
340
341 async def chat_completion_full_generator(
340 - self, request: ChatCompletionRequest, raw_request: Request,
342 + self, request: ChatCompletionRequest, raw_request: Optional[Request],
343 result_generator: AsyncIterator[RequestOutput], request_id: str,
344 conversation: List[ConversationMessage]
345 ) -> Union[ErrorResponse, ChatCompletionResponse]:
@@ -347,7 +349,7 @@ class OpenAIServingChat(OpenAIServing):
349 final_res: Optional[RequestOutput] = None
350
351 async for res in result_generator:
350 - if await raw_request.is_disconnected():
352 + if raw_request is not None and await raw_request.is_disconnected():
353 # Abort the request if the client disconnects.
354 await self.engine.abort(request_id)
355 return self.create_error_response("Client disconnected")