@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 276 lines 13.1 KB
Rendered Raw
1 # Offline Inference with the OpenAI Batch file format
2
3 ```{important}
4 This is a guide to performing batch inference using the OpenAI batch file format, **not** the complete Batch (REST) API.
5 ```
6
7 ## File Format
8
9 The OpenAI batch file format consists of a series of json objects on new lines.
10
11 [See here for an example file.](https://github.com/vllm-project/vllm/blob/main/examples/features/openai_batch/openai_example_batch.jsonl)
12
13 Each line represents a separate request. See the [OpenAI package reference](https://platform.openai.com/docs/api-reference/batch/requestInput) for more details.
14
15 ```{note}
16 We currently support `/v1/chat/completions`, `/v1/embeddings`, and `/v1/score` endpoints (completions coming soon).
17 ```
18
19 ## Pre-requisites
20
21 * The examples in this document use `meta-llama/Meta-Llama-3-8B-Instruct`.
22 * Create a [user access token](https://huggingface.co/docs/hub/en/security-tokens)
23 * Install the token on your machine (Run `hf auth login`).
24 * Get access to the gated model by [visiting the model card](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) and agreeing to the terms and conditions.
25
26 ## Example 1: Running with a local file
27
28 ### Step 1: Create your batch file
29
30 To follow along with this example, you can download the example batch, or create your own batch file in your working directory.
31
32 ```bash
33 wget https://raw.githubusercontent.com/vllm-project/vllm/main/examples/features/openai_batch/openai_example_batch.jsonl
34 ```
35
36 Once you've created your batch file it should look like this
37
38 ```bash
39 cat examples/features/openai_batch/openai_example_batch.jsonl
40 {"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_completion_tokens": 1000}}
41 {"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_completion_tokens": 1000}}
42 ```
43
44 ### Step 2: Run the batch
45
46 The batch running tool is designed to be used from the command line.
47
48 You can run the batch with the following command, which will write its results to a file called `results.jsonl`
49
50 ```bash
51 python -m vllm.entrypoints.launchers.run_batch \
52 -i examples/features/openai_batch/openai_example_batch.jsonl \
53 -o results.jsonl \
54 --model meta-llama/Meta-Llama-3-8B-Instruct
55 ```
56
57 or use command-line:
58
59 ```bash
60 vllm run-batch \
61 -i examples/features/openai_batch/openai_example_batch.jsonl \
62 -o results.jsonl \
63 --model meta-llama/Meta-Llama-3-8B-Instruct
64 ```
65
66 ### Step 3: Check your results
67
68 You should now have your results at `results.jsonl`. You can check your results by running `cat results.jsonl`
69
70 ```bash
71 cat results.jsonl
72 {"id":"vllm-383d1c59835645aeb2e07d004d62a826","custom_id":"request-1","response":{"id":"cmpl-61c020e54b964d5a98fa7527bfcdd378","object":"chat.completion","created":1715633336,"model":"meta-llama/Meta-Llama-3-8B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"Hello! It's great to meet you! I'm here to help with any questions or tasks you may have. What's on your mind today?"},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":25,"total_tokens":56,"completion_tokens":31}},"error":null}
73 {"id":"vllm-42e3d09b14b04568afa3f1797751a267","custom_id":"request-2","response":{"id":"cmpl-f44d049f6b3a42d4b2d7850bb1e31bcc","object":"chat.completion","created":1715633336,"model":"meta-llama/Meta-Llama-3-8B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"*silence*"},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":27,"total_tokens":32,"completion_tokens":5}},"error":null}
74 ```
75
76 ## Example 2: Using remote files
77
78 The batch runner supports remote input and output urls that are accessible via http/https.
79
80 For example, to run against our example input file located at `https://raw.githubusercontent.com/vllm-project/vllm/main/examples/features/openai_batch/openai_example_batch.jsonl`, you can run
81
82 ```bash
83 python -m vllm.entrypoints.launchers.run_batch \
84 -i https://raw.githubusercontent.com/vllm-project/vllm/main/examples/features/openai_batch/openai_example_batch.jsonl \
85 -o results.jsonl \
86 --model meta-llama/Meta-Llama-3-8B-Instruct
87 ```
88
89 or use command-line:
90
91 ```bash
92 vllm run-batch \
93 -i https://raw.githubusercontent.com/vllm-project/vllm/main/examples/features/openai_batch/openai_example_batch.jsonl \
94 -o results.jsonl \
95 --model meta-llama/Meta-Llama-3-8B-Instruct
96 ```
97
98 ## Example 3: Integrating with AWS S3
99
100 To integrate with cloud blob storage, we recommend using presigned urls.
101
102 [Learn more about S3 presigned urls here]
103
104 ### Additional prerequisites
105
106 * [Create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html).
107 * The `awscli` package (Run `pip install awscli`) to configure your credentials and interactively use s3.
108 * [Configure your credentials](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html).
109 * The `boto3` python package (Run `pip install boto3`) to generate presigned urls.
110
111 ### Step 1: Upload your input script
112
113 To follow along with this example, you can download the example batch, or create your own batch file in your working directory.
114
115 ```bash
116 wget https://raw.githubusercontent.com/vllm-project/vllm/main/examples/features/openai_batch/openai_example_batch.jsonl
117 ```
118
119 Once you've created your batch file it should look like this
120
121 ```bash
122 cat examples/features/openai_batch/openai_example_batch.jsonl
123 {"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_completion_tokens": 1000}}
124 {"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_completion_tokens": 1000}}
125 ```
126
127 Now upload your batch file to your S3 bucket.
128
129 ```bash
130 aws s3 cp examples/features/openai_batch/openai_example_batch.jsonl s3://MY_BUCKET/MY_INPUT_FILE.jsonl
131 ```
132
133 ### Step 2: Generate your presigned urls
134
135 Presigned urls can only be generated via the SDK. You can run the following python script to generate your presigned urls. Be sure to replace the `MY_BUCKET`, `MY_INPUT_FILE.jsonl`, and `MY_OUTPUT_FILE.jsonl` placeholders with your bucket and file names.
136
137 (The script is adapted from <https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/python/example_code/s3/s3_basics/presigned_url.py>)
138
139 ```python
140 import boto3
141 from botocore.exceptions import ClientError
142
143 def generate_presigned_url(s3_client, client_method, method_parameters, expires_in):
144 """
145 Generate a presigned Amazon S3 URL that can be used to perform an action.
146
147 :param s3_client: A Boto3 Amazon S3 client.
148 :param client_method: The name of the client method that the URL performs.
149 :param method_parameters: The parameters of the specified client method.
150 :param expires_in: The number of seconds the presigned URL is valid for.
151 :return: The presigned URL.
152 """
153 try:
154 url = s3_client.generate_presigned_url(
155 ClientMethod=client_method,
156 Params=method_parameters,
157 ExpiresIn=expires_in,
158 )
159 except ClientError:
160 raise
161 return url
162
163
164 s3_client = boto3.client("s3")
165 input_url = generate_presigned_url(
166 s3_client,
167 "get_object",
168 {"Bucket": "MY_BUCKET", "Key": "MY_INPUT_FILE.jsonl"},
169 expires_in=3600,
170 )
171 output_url = generate_presigned_url(
172 s3_client,
173 "put_object",
174 {"Bucket": "MY_BUCKET", "Key": "MY_OUTPUT_FILE.jsonl"},
175 expires_in=3600,
176 )
177 print(f"{input_url=}")
178 print(f"{output_url=}")
179 ```
180
181 This script should output
182
183 ```text
184 input_url='https://s3.us-west-2.amazonaws.com/MY_BUCKET/MY_INPUT_FILE.jsonl?AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST&Signature=abcdefghijklmnopqrstuvwxyz12345&Expires=1715800091'
185 output_url='https://s3.us-west-2.amazonaws.com/MY_BUCKET/MY_OUTPUT_FILE.jsonl?AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST&Signature=abcdefghijklmnopqrstuvwxyz12345&Expires=1715800091'
186 ```
187
188 ### Step 3: Run the batch runner using your presigned urls
189
190 You can now run the batch runner, using the urls generated in the previous section.
191
192 ```bash
193 python -m vllm.entrypoints.launchers.run_batch \
194 -i "https://s3.us-west-2.amazonaws.com/MY_BUCKET/MY_INPUT_FILE.jsonl?AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST&Signature=abcdefghijklmnopqrstuvwxyz12345&Expires=1715800091" \
195 -o "https://s3.us-west-2.amazonaws.com/MY_BUCKET/MY_OUTPUT_FILE.jsonl?AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST&Signature=abcdefghijklmnopqrstuvwxyz12345&Expires=1715800091" \
196 --model meta-llama/Meta-Llama-3-8B-Instruct
197 ```
198
199 or use command-line:
200
201 ```bash
202 vllm run-batch \
203 -i "https://s3.us-west-2.amazonaws.com/MY_BUCKET/MY_INPUT_FILE.jsonl?AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST&Signature=abcdefghijklmnopqrstuvwxyz12345&Expires=1715800091" \
204 -o "https://s3.us-west-2.amazonaws.com/MY_BUCKET/MY_OUTPUT_FILE.jsonl?AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST&Signature=abcdefghijklmnopqrstuvwxyz12345&Expires=1715800091" \
205 --model meta-llama/Meta-Llama-3-8B-Instruct
206 ```
207
208 ### Step 4: View your results
209
210 Your results are now on S3. You can view them in your terminal by running
211
212 ```bash
213 aws s3 cp s3://MY_BUCKET/MY_OUTPUT_FILE.jsonl -
214 ```
215
216 ## Example 4: Using embeddings endpoint
217
218 ### Additional prerequisites
219
220 * Ensure you are using `vllm >= 0.5.5`.
221
222 ### Step 1: Create your batch file
223
224 Add embedding requests to your batch file. The following is an example:
225
226 ```text
227 {"custom_id": "request-1", "method": "POST", "url": "/v1/embeddings", "body": {"model": "intfloat/e5-mistral-7b-instruct", "input": "You are a helpful assistant."}}
228 {"custom_id": "request-2", "method": "POST", "url": "/v1/embeddings", "body": {"model": "intfloat/e5-mistral-7b-instruct", "input": "You are an unhelpful assistant."}}
229 ```
230
231 You can even mix chat completion and embedding requests in the batch file, as long as the model you are using supports both chat completion and embeddings (note that all requests must use the same model).
232
233 ### Step 2: Run the batch
234
235 You can run the batch using the same command as in earlier examples.
236
237 ### Step 3: Check your results
238
239 You can check your results by running `cat results.jsonl`
240
241 ```bash
242 cat results.jsonl
243 {"id":"vllm-db0f71f7dec244e6bce530e0b4ef908b","custom_id":"request-1","response":{"status_code":200,"request_id":"vllm-batch-3580bf4d4ae54d52b67eee266a6eab20","body":{"id":"embd-33ac2efa7996430184461f2e38529746","object":"list","created":444647,"model":"intfloat/e5-mistral-7b-instruct","data":[{"index":0,"object":"embedding","embedding":[0.016204833984375,0.0092010498046875,0.0018358230590820312,-0.0028228759765625,0.001422882080078125,-0.0031147003173828125,...]}],"usage":{"prompt_tokens":8,"total_tokens":8,"completion_tokens":0}}},"error":null}
244 ...
245 ```
246
247 ## Example 5: Using score endpoint
248
249 ### Additional prerequisites
250
251 * Ensure you are using `vllm >= 0.7.0`.
252
253 ### Step 1: Create your batch file
254
255 Add score requests to your batch file. The following is an example:
256
257 ```text
258 {"custom_id": "request-1", "method": "POST", "url": "/v1/score", "body": {"model": "BAAI/bge-reranker-v2-m3", "queries": "What is the capital of France?", "documents": ["The capital of Brazil is Brasilia.", "The capital of France is Paris."]}}
259 {"custom_id": "request-2", "method": "POST", "url": "/v1/score", "body": {"model": "BAAI/bge-reranker-v2-m3", "queries": "What is the capital of France?", "documents": ["The capital of Brazil is Brasilia.", "The capital of France is Paris."]}}
260 ```
261
262 You can mix chat completion, embedding, and score requests in the batch file, as long as the model you are using supports them all (note that all requests must use the same model).
263
264 ### Step 2: Run the batch
265
266 You can run the batch using the same command as in earlier examples.
267
268 ### Step 3: Check your results
269
270 You can check your results by running `cat results.jsonl`
271
272 ```bash
273 cat results.jsonl
274 {"id":"vllm-f87c5c4539184f618e555744a2965987","custom_id":"request-1","response":{"status_code":200,"request_id":"vllm-batch-806ab64512e44071b37d3f7ccd291413","body":{"id":"score-4ee45236897b4d29907d49b01298cdb1","object":"list","created":1737847944,"model":"BAAI/bge-reranker-v2-m3","data":[{"index":0,"object":"score","score":0.0010900497436523438},{"index":1,"object":"score","score":1.0}],"usage":{"prompt_tokens":37,"total_tokens":37,"completion_tokens":0,"prompt_tokens_details":null}}},"error":null}
275 {"id":"vllm-41990c51a26d4fac8419077f12871099","custom_id":"request-2","response":{"status_code":200,"request_id":"vllm-batch-73ce66379026482699f81974e14e1e99","body":{"id":"score-13f2ffe6ba40460fbf9f7f00ad667d75","object":"list","created":1737847944,"model":"BAAI/bge-reranker-v2-m3","data":[{"index":0,"object":"score","score":0.001094818115234375},{"index":1,"object":"score","score":1.0}],"usage":{"prompt_tokens":37,"total_tokens":37,"completion_tokens":0,"prompt_tokens_details":null}}},"error":null}
276 ```