A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 23:15 UTC
[CI/Build][Misc] Add CI that benchmarks vllm performance on those PRs with `perf-benchmarks` label (#5073)
Co-authored-by: simon-mo <simon.mo@hey.com>
Kuntai Du committed
Jun 13, 2024 at 22:36 UTC
319ad7f1d386699e94f629341c9988a926821f24
13 files changed
+880
-41
.buildkite/nightly-benchmarks/README.md
new
+98
@@ -0,0 +1,98 @@
1
+# vLLM benchmark suite
2
+
3
+## Introduction
4
+
5
+This directory contains the performance benchmarking CI for vllm.
6
+The goal is to help developers know the impact of their PRs on the performance of vllm.
7
+
8
+This benchmark will be *triggered* upon:
9
+- A PR being merged into vllm.
10
+- Every commit for those PRs with `perf-benchmarks` label.
11
+
12
+**Benchmarking Coverage**: latency, throughput and fix-qps serving on A100 (the support for more GPUs is comming later), with different models.
13
+
14
+**Benchmarking Duration**: about 1hr.
15
+
16
+## Configuring the workload for the quick benchmark
17
+
18
+The workload of the quick benchmark contains two parts: latency tests in `latency-tests.json`, throughput tests in `throughput-tests.json` and serving tests in `serving-tests.json`.
19
+
20
+### Latency test
21
+
22
+Here is an example of one test inside `latency-tests.json`:
23
+
24
+```json
25
+[
26
+ ...
27
+ {
28
+ "test_name": "latency_llama8B_tp1",
29
+ "parameters": {
30
+ "model": "meta-llama/Meta-Llama-3-8B",
31
+ "tensor_parallel_size": 1,
32
+ "load_format": "dummy",
33
+ "num_iters_warmup": 5,
34
+ "num_iters": 15
35
+ }
36
+ },
37
+ ...
38
+]
39
+```
40
+
41
+In this example:
42
+- The `test_name` attributes is a unique identifier for the test. In `latency-tests.json`, it must start with `latency_`.
43
+- The `parameters` attribute control the command line arguments to be used for `benchmark_latency.py`. Note that please use underline `_` instead of the dash `-` when specifying the command line arguments, and `run-benchmarks-suite.sh` will convert the underline to dash when feeding the arguments to `benchmark_latency.py`. For example, the corresponding command line arguments for `benchmark_latency.py` will be `--model meta-llama/Meta-Llama-3-8B --tensor-parallel-size 1 --load-format dummy --num-iters-warmup 5 --num-iters 15`
44
+
45
+Note that the performance numbers are highly sensitive to the value of the parameters. Please make sure the parameters are set correctly.
46
+
47
+WARNING: The benchmarking script will save json results by itself, so please do not configure `--output-json` parameter in the json file.
48
+
49
+
50
+### Throughput test
51
+The tests are specified in `throughput-tests.json`. The syntax is similar to `latency-tests.json`, except for that the parameters will be fed forward to `benchmark_throughput.py`.
52
+
53
+The number of this test is also stable -- a slight change on the value of this number might vary the performance numbers by a lot.
54
+
55
+### Serving test
56
+We test the throughput by using `benchmark_serving.py` with request rate = inf to cover the online serving overhead. The corresponding parameters are in `serving-tests.json`, and here is an example:
57
+
58
+```
59
+[
60
+ ...
61
+ {
62
+ "test_name": "serving_llama8B_tp1_sharegpt",
63
+ "qps_list": [1, 4, 16, "inf"],
64
+ "server_parameters": {
65
+ "model": "meta-llama/Meta-Llama-3-8B",
66
+ "tensor_parallel_size": 1,
67
+ "swap_space": 16,
68
+ "disable_log_stats": "",
69
+ "disable_log_requests": "",
70
+ "load_format": "dummy"
71
+ },
72
+ "client_parameters": {
73
+ "model": "meta-llama/Meta-Llama-3-8B",
74
+ "backend": "vllm",
75
+ "dataset_name": "sharegpt",
76
+ "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
77
+ "num_prompts": 200
78
+ }
79
+ },
80
+ ...
81
+]
82
+```
83
+
84
+Inside this example:
85
+- The `test_name` attribute is also a unique identifier for the test. It must start with `serving_`.
86
+- The `server-parameters` includes the command line arguments for vLLM server.
87
+- The `client-parameters` includes the command line arguments for `benchmark_serving.py`.
88
+- The `qps_list` controls the list of qps for test. It will be used to configure the `--request-rate` parameter in `benchmark_serving.py`
89
+
90
+The number of this test is less stable compared to the delay and latency benchmarks (due to randomized sharegpt dataset sampling inside `benchmark_serving.py`), but a large change on this number (e.g. 5% change) still vary the output greatly.
91
+
92
+WARNING: The benchmarking script will save json results by itself, so please do not configure `--save-results` or other results-saving-related parameters in `serving-tests.json`.
93
+
94
+## Visualizing the results
95
+The `convert-results-json-to-markdown.py` helps you put the benchmarking results inside a markdown table.
96
+You can find the result presented as a table inside the `buildkite/performance-benchmark` job page.
97
+If you do not see the table, please wait till the benchmark finish running.
98
+The JSON file is also attached within each buildkite job for further analysis.
\ No newline at end of file
.buildkite/nightly-benchmarks/benchmark-pipeline.yaml
new
+61
@@ -0,0 +1,61 @@
1
+steps:
2
+ - label: "Wait for container to be ready"
3
+ agents:
4
+ queue: A100
5
+ plugins:
6
+ - kubernetes:
7
+ podSpec:
8
+ containers:
9
+ - image: badouralix/curl-jq
10
+ command:
11
+ - sh
12
+ - .buildkite/nightly-benchmarks/scripts/wait-for-image.sh
13
+ - wait
14
+ - label: "A100 Benchmark"
15
+ agents:
16
+ queue: A100
17
+ plugins:
18
+ - kubernetes:
19
+ podSpec:
20
+ containers:
21
+ - image: public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT
22
+ command:
23
+ - bash .buildkite/nightly-benchmarks/run-benchmarks-suite.sh
24
+ resources:
25
+ limits:
26
+ nvidia.com/gpu: 8
27
+ volumeMounts:
28
+ - name: devshm
29
+ mountPath: /dev/shm
30
+ env:
31
+ - name: VLLM_USAGE_SOURCE
32
+ value: ci-test
33
+ - name: HF_TOKEN
34
+ valueFrom:
35
+ secretKeyRef:
36
+ name: hf-token-secret
37
+ key: token
38
+ nodeSelector:
39
+ nvidia.com/gpu.product: NVIDIA-A100-SXM4-80GB
40
+ volumes:
41
+ - name: devshm
42
+ emptyDir:
43
+ medium: Memory
44
+ # - label: "H100: NVIDIA SMI"
45
+ # agents:
46
+ # queue: H100
47
+ # plugins:
48
+ # - docker#v5.11.0:
49
+ # image: public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT
50
+ # command:
51
+ # - bash
52
+ # - .buildkite/nightly-benchmarks/run-benchmarks-suite.sh
53
+ # mount-buildkite-agent: true
54
+ # propagate-environment: true
55
+ # propagate-uid-gid: false
56
+ # ipc: host
57
+ # gpus: all
58
+ # environment:
59
+ # - VLLM_USAGE_SOURCE
60
+ # - HF_TOKEN
61
+
.buildkite/nightly-benchmarks/kickoff-pipeline.sh
+2
-1
@@ -1,5 +1,6 @@
1
#!/usr/bin/env bash
2
3
+# NOTE(simon): this script runs inside a buildkite agent with CPU only access.
4
set -euo pipefail
5
6
# Install system packages
@@ -23,4 +24,4 @@ if [ "$BUILDKITE_PULL_REQUEST" != "false" ]; then
24
fi
25
26
# Upload sample.yaml
26
-buildkite-agent pipeline upload .buildkite/nightly-benchmarks/sample.yaml
27
+buildkite-agent pipeline upload .buildkite/nightly-benchmarks/benchmark-pipeline.yaml
.buildkite/nightly-benchmarks/latency-tests.json
new
+32
@@ -0,0 +1,32 @@
1
+[
2
+ {
3
+ "test_name": "latency_llama8B_tp1",
4
+ "parameters": {
5
+ "model": "meta-llama/Meta-Llama-3-8B",
6
+ "tensor_parallel_size": 1,
7
+ "load_format": "dummy",
8
+ "num_iters_warmup": 5,
9
+ "num_iters": 15
10
+ }
11
+ },
12
+ {
13
+ "test_name": "latency_llama70B_tp4",
14
+ "parameters": {
15
+ "model": "meta-llama/Meta-Llama-3-70B-Instruct",
16
+ "tensor_parallel_size": 4,
17
+ "load_format": "dummy",
18
+ "num-iters-warmup": 5,
19
+ "num-iters": 15
20
+ }
21
+ },
22
+ {
23
+ "test_name": "latency_mixtral8x7B_tp2",
24
+ "parameters": {
25
+ "model": "mistralai/Mixtral-8x7B-Instruct-v0.1",
26
+ "tensor_parallel_size": 2,
27
+ "load_format": "dummy",
28
+ "num-iters-warmup": 5,
29
+ "num-iters": 15
30
+ }
31
+ }
32
+]
.buildkite/nightly-benchmarks/run-benchmarks-suite.sh
new
+358
@@ -0,0 +1,358 @@
1
+#!/bin/bash
2
+
3
+# This script should be run inside the CI process
4
+# This script assumes that we are already inside the vllm/ directory
5
+# Benchmarking results will be available inside vllm/benchmarks/results/
6
+
7
+# Do not set -e, as the mixtral 8x22B model tends to crash occasionally
8
+# and we still want to see other benchmarking results even when mixtral crashes.
9
+set -o pipefail
10
+
11
+check_gpus() {
12
+ # check the number of GPUs and GPU type.
13
+ declare -g gpu_count=$(nvidia-smi --list-gpus | wc -l)
14
+ if [[ $gpu_count -gt 0 ]]; then
15
+ echo "GPU found."
16
+ else
17
+ echo "Need at least 1 GPU to run benchmarking."
18
+ exit 1
19
+ fi
20
+ declare -g gpu_type=$(echo $(nvidia-smi --query-gpu=name --format=csv,noheader) | awk '{print $2}')
21
+ echo "GPU type is $gpu_type"
22
+}
23
+
24
+check_hf_token() {
25
+ # check if HF_TOKEN is available and valid
26
+ if [[ -z "$HF_TOKEN" ]]; then
27
+ echo "Error: HF_TOKEN is not set."
28
+ exit 1
29
+ elif [[ ! "$HF_TOKEN" =~ ^hf_ ]]; then
30
+ echo "Error: HF_TOKEN does not start with 'hf_'."
31
+ exit 1
32
+ else
33
+ echo "HF_TOKEN is set and valid."
34
+ fi
35
+}
36
+
37
+json2args() {
38
+ # transforms the JSON string to command line args, and '_' is replaced to '-'
39
+ # example:
40
+ # input: { "model": "meta-llama/Llama-2-7b-chat-hf", "tensor_parallel_size": 1 }
41
+ # output: --model meta-llama/Llama-2-7b-chat-hf --tensor-parallel-size 1
42
+ local json_string=$1
43
+ local args=$(
44
+ echo "$json_string" | jq -r '
45
+ to_entries |
46
+ map("--" + (.key | gsub("_"; "-")) + " " + (.value | tostring)) |
47
+ join(" ")
48
+ '
49
+ )
50
+ echo "$args"
51
+}
52
+
53
+wait_for_server() {
54
+ # wait for vllm server to start
55
+ # return 1 if vllm server crashes
56
+ timeout 1200 bash -c '
57
+ until curl localhost:8000/v1/completions; do
58
+ sleep 1
59
+ done' && return 0 || return 1
60
+}
61
+
62
+kill_gpu_processes() {
63
+ # kill all processes on GPU.
64
+ pids=$(nvidia-smi --query-compute-apps=pid --format=csv,noheader)
65
+ if [ -z "$pids" ]; then
66
+ echo "No GPU processes found."
67
+ else
68
+ for pid in $pids; do
69
+ kill -9 "$pid"
70
+ echo "Killed process with PID: $pid"
71
+ done
72
+
73
+ echo "All GPU processes have been killed."
74
+ fi
75
+
76
+ # waiting for GPU processes to be fully killed
77
+ sleep 10
78
+
79
+ # remove vllm config file
80
+ rm -rf ~/.config/vllm
81
+
82
+ # Print the GPU memory usage
83
+ # so that we know if all GPU processes are killed.
84
+ gpu_memory_usage=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits -i 0)
85
+ # The memory usage should be 0 MB.
86
+ echo "GPU 0 Memory Usage: $gpu_memory_usage MB"
87
+}
88
+
89
+upload_to_buildkite() {
90
+ # upload the benchmarking results to buildkite
91
+
92
+ # if the agent binary is not found, skip uploading the results, exit 0
93
+ if [ ! -f /workspace/buildkite-agent ]; then
94
+ echo "buildkite-agent binary not found. Skip uploading the results."
95
+ return 0
96
+ fi
97
+ /workspace/buildkite-agent annotate --style "info" --context "benchmark-results" < $RESULTS_FOLDER/benchmark_results.md
98
+ /workspace/buildkite-agent artifact upload "$RESULTS_FOLDER/*"
99
+}
100
+
101
+run_latency_tests() {
102
+ # run latency tests using `benchmark_latency.py`
103
+ # $1: a json file specifying latency test cases
104
+
105
+ local latency_test_file
106
+ latency_test_file=$1
107
+
108
+ # Iterate over latency tests
109
+ jq -c '.[]' "$latency_test_file" | while read -r params; do
110
+ # get the test name, and append the GPU type back to it.
111
+ test_name=$(echo "$params" | jq -r '.test_name')
112
+ if [[ ! "$test_name" =~ ^latency_ ]]; then
113
+ echo "In latency-test.json, test_name must start with \"latency_\"."
114
+ exit 1
115
+ fi
116
+
117
+ # if TEST_SELECTOR is set, only run the test cases that match the selector
118
+ if [[ -n "$TEST_SELECTOR" ]] && [[ ! "$test_name" =~ $TEST_SELECTOR ]]; then
119
+ echo "Skip test case $test_name."
120
+ continue
121
+ fi
122
+
123
+ # get arguments
124
+ latency_params=$(echo "$params" | jq -r '.parameters')
125
+ latency_args=$(json2args "$latency_params")
126
+
127
+ # check if there is enough GPU to run the test
128
+ tp=$(echo "$latency_params" | jq -r '.tensor_parallel_size')
129
+ if [[ $gpu_count -lt $tp ]]; then
130
+ echo "Required tensor-parallel-size $tp but only $gpu_count GPU found. Skip testcase $testname."
131
+ continue
132
+ fi
133
+
134
+ latency_command="python3 benchmark_latency.py \
135
+ --output-json $RESULTS_FOLDER/${test_name}.json \
136
+ $latency_args"
137
+
138
+ echo "Running test case $test_name"
139
+ echo "Latency command: $latency_command"
140
+
141
+ # recoding benchmarking command ang GPU command
142
+ jq_output=$(jq -n \
143
+ --arg latency "$latency_command" \
144
+ --arg gpu "$gpu_type" \
145
+ '{
146
+ latency_command: $latency,
147
+ gpu_type: $gpu
148
+ }')
149
+ echo "$jq_output" > "$RESULTS_FOLDER/$test_name.commands"
150
+
151
+ # run the benchmark
152
+ eval "$latency_command"
153
+
154
+ kill_gpu_processes
155
+
156
+ done
157
+}
158
+
159
+
160
+run_throughput_tests() {
161
+ # run throughput tests using `benchmark_throughput.py`
162
+ # $1: a json file specifying throughput test cases
163
+
164
+ local throughput_test_file
165
+ throughput_test_file=$1
166
+
167
+ # Iterate over throughput tests
168
+ jq -c '.[]' "$throughput_test_file" | while read -r params; do
169
+ # get the test name, and append the GPU type back to it.
170
+ test_name=$(echo "$params" | jq -r '.test_name')
171
+ if [[ ! "$test_name" =~ ^throughput_ ]]; then
172
+ echo "In throughput-test.json, test_name must start with \"throughput_\"."
173
+ exit 1
174
+ fi
175
+
176
+ # if TEST_SELECTOR is set, only run the test cases that match the selector
177
+ if [[ -n "$TEST_SELECTOR" ]] && [[ ! "$test_name" =~ $TEST_SELECTOR ]]; then
178
+ echo "Skip test case $test_name."
179
+ continue
180
+ fi
181
+
182
+ # get arguments
183
+ throughput_params=$(echo "$params" | jq -r '.parameters')
184
+ throughput_args=$(json2args "$throughput_params")
185
+
186
+ # check if there is enough GPU to run the test
187
+ tp=$(echo $throughput_params | jq -r '.tensor_parallel_size')
188
+ if [[ $gpu_count -lt $tp ]]; then
189
+ echo "Required tensor-parallel-size $tp but only $gpu_count GPU found. Skip testcase $testname."
190
+ continue
191
+ fi
192
+
193
+ throughput_command="python3 benchmark_throughput.py \
194
+ --output-json $RESULTS_FOLDER/${test_name}.json \
195
+ $throughput_args"
196
+
197
+ echo "Running test case $test_name"
198
+ echo "Throughput command: $throughput_command"
199
+ # recoding benchmarking command ang GPU command
200
+ jq_output=$(jq -n \
201
+ --arg command "$throughput_command" \
202
+ --arg gpu "$gpu_type" \
203
+ '{
204
+ throughput_command: $command,
205
+ gpu_type: $gpu
206
+ }')
207
+ echo "$jq_output" > "$RESULTS_FOLDER/$test_name.commands"
208
+
209
+ # run the benchmark
210
+ eval "$throughput_command"
211
+
212
+ kill_gpu_processes
213
+
214
+ done
215
+}
216
+
217
+run_serving_tests() {
218
+ # run serving tests using `benchmark_serving.py`
219
+ # $1: a json file specifying serving test cases
220
+
221
+ local serving_test_file
222
+ serving_test_file=$1
223
+
224
+ # Iterate over serving tests
225
+ jq -c '.[]' "$serving_test_file" | while read -r params; do
226
+ # get the test name, and append the GPU type back to it.
227
+ test_name=$(echo "$params" | jq -r '.test_name')
228
+ if [[ ! "$test_name" =~ ^serving_ ]]; then
229
+ echo "In serving-test.json, test_name must start with \"serving_\"."
230
+ exit 1
231
+ fi
232
+
233
+ # if TEST_SELECTOR is set, only run the test cases that match the selector
234
+ if [[ -n "$TEST_SELECTOR" ]] && [[ ! "$test_name" =~ $TEST_SELECTOR ]]; then
235
+ echo "Skip test case $test_name."
236
+ continue
237
+ fi
238
+
239
+
240
+ # get client and server arguments
241
+ server_params=$(echo "$params" | jq -r '.server_parameters')
242
+ client_params=$(echo "$params" | jq -r '.client_parameters')
243
+ server_args=$(json2args "$server_params")
244
+ client_args=$(json2args "$client_params")
245
+ qps_list=$(echo "$params" | jq -r '.qps_list')
246
+ qps_list=$(echo "$qps_list" | jq -r '.[] | @sh')
247
+ echo "Running over qps list $qps_list"
248
+
249
+ # check if there is enough GPU to run the test
250
+ tp=$(echo "$server_params" | jq -r '.tensor_parallel_size')
251
+ if [[ $gpu_count -lt $tp ]]; then
252
+ echo "Required tensor-parallel-size $tp but only $gpu_count GPU found. Skip testcase $testname."
253
+ continue
254
+ fi
255
+
256
+ # check if server model and client model is aligned
257
+ server_model=$(echo "$server_params" | jq -r '.model')
258
+ client_model=$(echo "$client_params" | jq -r '.model')
259
+ if [[ $server_model != "$client_model" ]]; then
260
+ echo "Server model and client model must be the same. Skip testcase $testname."
261
+ continue
262
+ fi
263
+
264
+ server_command="python3 \
265
+ -m vllm.entrypoints.openai.api_server \
266
+ $server_args"
267
+
268
+ # run the server
269
+ echo "Running test case $test_name"
270
+ echo "Server command: $server_command"
271
+ eval "$server_command" &
272
+
273
+ # wait until the server is alive
274
+ wait_for_server
275
+ if [ $? -eq 0 ]; then
276
+ echo ""
277
+ echo "vllm server is up and running."
278
+ else
279
+ echo ""
280
+ echo "vllm failed to start within the timeout period."
281
+ fi
282
+
283
+ # iterate over different QPS
284
+ for qps in $qps_list; do
285
+ # remove the surrounding single quote from qps
286
+ if [[ "$qps" == *"inf"* ]]; then
287
+ echo "qps was $qps"
288
+ qps="inf"
289
+ echo "now qps is $qps"
290
+ fi
291
+
292
+ new_test_name=$test_name"_qps_"$qps
293
+
294
+ client_command="python3 benchmark_serving.py \
295
+ --save-result \
296
+ --result-dir $RESULTS_FOLDER \
297
+ --result-filename ${new_test_name}.json \
298
+ --request-rate $qps \
299
+ $client_args"
300
+
301
+ echo "Running test case $test_name with qps $qps"
302
+ echo "Client command: $client_command"
303
+
304
+ eval "$client_command"
305
+
306
+ # record the benchmarking commands
307
+ jq_output=$(jq -n \
308
+ --arg server "$server_command" \
309
+ --arg client "$client_command" \
310
+ --arg gpu "$gpu_type" \
311
+ '{
312
+ server_command: $server,
313
+ client_command: $client,
314
+ gpu_type: $gpu
315
+ }')
316
+ echo "$jq_output" > "$RESULTS_FOLDER/${new_test_name}.commands"
317
+
318
+ done
319
+
320
+ # clean up
321
+ kill_gpu_processes
322
+ done
323
+}
324
+
325
+main() {
326
+ check_gpus
327
+ check_hf_token
328
+
329
+ # dependencies
330
+ (which wget && which curl) || (apt-get update && apt-get install -y wget curl)
331
+ (which jq) || (apt-get update && apt-get -y install jq)
332
+
333
+ # get the current IP address, required by benchmark_serving.py
334
+ export VLLM_HOST_IP=$(hostname -I | awk '{print $1}')
335
+ # turn of the reporting of the status of each request, to clean up the terminal output
336
+ export VLLM_LOG_LEVEL="WARNING"
337
+
338
+ # prepare for benchmarking
339
+ cd benchmarks || exit 1
340
+ wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
341
+ declare -g RESULTS_FOLDER=results/
342
+ mkdir -p $RESULTS_FOLDER
343
+ QUICK_BENCHMARK_ROOT=../.buildkite/nightly-benchmarks/
344
+
345
+ # benchmarking
346
+ run_serving_tests $QUICK_BENCHMARK_ROOT/serving-tests.json
347
+ run_latency_tests $QUICK_BENCHMARK_ROOT/latency-tests.json
348
+ run_throughput_tests $QUICK_BENCHMARK_ROOT/throughput-tests.json
349
+
350
+
351
+ # postprocess benchmarking results
352
+ pip install tabulate pandas
353
+ python3 $QUICK_BENCHMARK_ROOT/scripts/convert-results-json-to-markdown.py
354
+
355
+ upload_to_buildkite
356
+}
357
+
358
+main "$@"
.buildkite/nightly-benchmarks/sample.yaml
deleted
-39
@@ -1,39 +0,0 @@
1
-steps:
2
- # NOTE(simon): You can create separate blocks for different jobs
3
- - label: "A100: NVIDIA SMI"
4
- agents:
5
- queue: A100
6
- plugins:
7
- - kubernetes:
8
- podSpec:
9
- containers:
10
- # - image: us-central1-docker.pkg.dev/vllm-405802/vllm-ci-test-repo/vllm-test:$BUILDKITE_COMMIT
11
- # TODO(simon): check latest main branch or use the PR image.
12
- - image: us-central1-docker.pkg.dev/vllm-405802/vllm-ci-test-repo/vllm-test:45c35f0d58f4508bf43bd6af1d3d0d0ec0c915e6
13
- command:
14
- - bash -c 'nvidia-smi && nvidia-smi topo -m && pwd && ls'
15
- resources:
16
- limits:
17
- nvidia.com/gpu: 8
18
- volumeMounts:
19
- - name: devshm
20
- mountPath: /dev/shm
21
- nodeSelector:
22
- nvidia.com/gpu.product: NVIDIA-A100-SXM4-80GB
23
- volumes:
24
- - name: devshm
25
- emptyDir:
26
- medium: Memory
27
- # TODO(simon): bring H100 online
28
- # - label: "H100: NVIDIA SMI"
29
- # agents:
30
- # queue: H100
31
- # plugins:
32
- # - docker#v5.11.0:
33
- # image: us-central1-docker.pkg.dev/vllm-405802/vllm-ci-test-repo/vllm-test:45c35f0d58f4508bf43bd6af1d3d0d0ec0c915e6
34
- # command:
35
- # - bash -c 'nvidia-smi && nvidia-smi topo -m'
36
- # propagate-environment: true
37
- # ipc: host
38
- # gpus: all
39
-
.buildkite/nightly-benchmarks/scripts/convert-results-json-to-markdown.py
new
+155
@@ -0,0 +1,155 @@
1
+import json
2
+from pathlib import Path
3
+
4
+import pandas as pd
5
+from tabulate import tabulate
6
+
7
+results_folder = Path("results/")
8
+
9
+# latency results and the keys that will be printed into markdown
10
+latency_results = []
11
+latency_column_mapping = {
12
+ "test_name": "Test name",
13
+ "gpu_type": "GPU",
14
+ "avg_latency": "Average latency (s)",
15
+ "P10": "P10 (s)",
16
+ "P25": "P25 (s)",
17
+ "P50": "P50 (s)",
18
+ "P75": "P75 (s)",
19
+ "P90": "P90 (s)",
20
+}
21
+
22
+# thoughput tests and the keys that will be printed into markdown
23
+throughput_results = []
24
+throughput_results_column_mapping = {
25
+ "test_name": "Test name",
26
+ "gpu_type": "GPU",
27
+ "num_requests": "# of req.",
28
+ "total_num_tokens": "Total # of tokens",
29
+ "elapsed_time": "Elapsed time (s)",
30
+ "requests_per_second": "Tput (req/s)",
31
+ "tokens_per_second": "Tput (tok/s)",
32
+}
33
+
34
+# serving results and the keys that will be printed into markdown
35
+serving_results = []
36
+serving_column_mapping = {
37
+ "test_name": "Test name",
38
+ "gpu_type": "GPU",
39
+ "completed": "# of req.",
40
+ "request_throughput": "Tput (req/s)",
41
+ "input_throughput": "Input Tput (tok/s)",
42
+ "output_throughput": "Output Tput (tok/s)",
43
+ "mean_ttft_ms": "Mean TTFT (ms)",
44
+ # do not say TTFT again to avoid the table getting too wide
45
+ "median_ttft_ms": "Median",
46
+ "p99_ttft_ms": "P99",
47
+ "mean_tpot_ms": "Mean TPOT (ms)",
48
+ "median_tpot_ms": "Median",
49
+ "p99_tpot_ms": "P99",
50
+ "mean_itl_ms": "Mean ITL (ms)",
51
+ "median_itl_ms": "Median",
52
+ "p99_itl_ms": "P99",
53
+}
54
+
55
+for test_file in results_folder.glob("*.json"):
56
+
57
+ with open(test_file, "r") as f:
58
+ raw_result = json.loads(f.read())
59
+
60
+ if "serving" in str(test_file):
61
+ # this result is generated via `benchmark_serving.py`
62
+
63
+ # attach the benchmarking command to raw_result
64
+ with open(test_file.with_suffix(".commands"), "r") as f:
65
+ command = json.loads(f.read())
66
+ raw_result.update(command)
67
+
68
+ # update the test name of this result
69
+ raw_result.update({"test_name": test_file.stem})
70
+
71
+ # add the result to raw_result
72
+ serving_results.append(raw_result)
73
+ continue
74
+
75
+ elif "latency" in f.name:
76
+ # this result is generated via `benchmark_latency.py`
77
+
78
+ # attach the benchmarking command to raw_result
79
+ with open(test_file.with_suffix(".commands"), "r") as f:
80
+ command = json.loads(f.read())
81
+ raw_result.update(command)
82
+
83
+ # update the test name of this result
84
+ raw_result.update({"test_name": test_file.stem})
85
+
86
+ # get different percentiles
87
+ for perc in [10, 25, 50, 75, 90]:
88
+ raw_result.update(
89
+ {f"P{perc}": raw_result["percentiles"][str(perc)]})
90
+
91
+ # add the result to raw_result
92
+ latency_results.append(raw_result)
93
+ continue
94
+
95
+ elif "throughput" in f.name:
96
+ # this result is generated via `benchmark_throughput.py`
97
+
98
+ # attach the benchmarking command to raw_result
99
+ with open(test_file.with_suffix(".commands"), "r") as f:
100
+ command = json.loads(f.read())
101
+ raw_result.update(command)
102
+
103
+ # update the test name of this result
104
+ raw_result.update({"test_name": test_file.stem})
105
+
106
+ # add the result to raw_result
107
+ throughput_results.append(raw_result)
108
+ continue
109
+
110
+ print(f"Skipping {test_file}")
111
+
112
+latency_results = pd.DataFrame.from_dict(latency_results)
113
+serving_results = pd.DataFrame.from_dict(serving_results)
114
+throughput_results = pd.DataFrame.from_dict(throughput_results)
115
+
116
+# remapping the key, for visualization purpose
117
+if not latency_results.empty:
118
+ latency_results = latency_results[list(
119
+ latency_column_mapping.keys())].rename(columns=latency_column_mapping)
120
+if not serving_results.empty:
121
+ serving_results = serving_results[list(
122
+ serving_column_mapping.keys())].rename(columns=serving_column_mapping)
123
+if not throughput_results.empty:
124
+ throughput_results = throughput_results[list(
125
+ throughput_results_column_mapping.keys())].rename(
126
+ columns=throughput_results_column_mapping)
127
+
128
+# get markdown tables
129
+latency_md_table = tabulate(latency_results,
130
+ headers='keys',
131
+ tablefmt='pipe',
132
+ showindex=False)
133
+serving_md_table = tabulate(serving_results,
134
+ headers='keys',
135
+ tablefmt='pipe',
136
+ showindex=False)
137
+throughput_md_table = tabulate(throughput_results,
138
+ headers='keys',
139
+ tablefmt='pipe',
140
+ showindex=False)
141
+
142
+# document the result
143
+with open(results_folder / "benchmark_results.md", "w") as f:
144
+ if not latency_results.empty:
145
+ f.write("## Latency tests\n")
146
+ f.write(latency_md_table)
147
+ f.write("\n")
148
+ if not throughput_results.empty:
149
+ f.write("## Throughput tests\n")
150
+ f.write(throughput_md_table)
151
+ f.write("\n")
152
+ if not serving_results.empty:
153
+ f.write("## Serving tests\n")
154
+ f.write(serving_md_table)
155
+ f.write("\n")
.buildkite/nightly-benchmarks/scripts/wait-for-image.sh
new
+17
@@ -0,0 +1,17 @@
1
+#!/bin/sh
2
+TOKEN=$(curl -s -L "https://public.ecr.aws/token?service=public.ecr.aws&scope=repository:q9t5s3a7/vllm-ci-test-repo:pull" | jq -r .token)
3
+URL="https://public.ecr.aws/v2/q9t5s3a7/vllm-ci-test-repo/manifests/$BUILDKITE_COMMIT"
4
+
5
+retries=0
6
+while [ $retries -lt 1000 ]; do
7
+ if [ $(curl -s -L -H "Authorization: Bearer $TOKEN" -o /dev/null -w "%{http_code}" $URL) -eq 200 ]; then
8
+ exit 0
9
+ fi
10
+
11
+ echo "Waiting for image to be available..."
12
+
13
+ retries=$((retries + 1))
14
+ sleep 5
15
+done
16
+
17
+exit 1
\ No newline at end of file
.buildkite/nightly-benchmarks/serving-tests.json
new
+59
@@ -0,0 +1,59 @@
1
+[
2
+ {
3
+ "test_name": "serving_llama8B_tp1_sharegpt",
4
+ "qps_list": [1, 4, 16, "inf"],
5
+ "server_parameters": {
6
+ "model": "meta-llama/Meta-Llama-3-8B",
7
+ "tensor_parallel_size": 1,
8
+ "swap_space": 16,
9
+ "disable_log_stats": "",
10
+ "disable_log_requests": "",
11
+ "load_format": "dummy"
12
+ },
13
+ "client_parameters": {
14
+ "model": "meta-llama/Meta-Llama-3-8B",
15
+ "backend": "vllm",
16
+ "dataset_name": "sharegpt",
17
+ "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
18
+ "num_prompts": 200
19
+ }
20
+ },
21
+ {
22
+ "test_name": "serving_llama70B_tp4_sharegpt",
23
+ "qps_list": [1, 4, 16, "inf"],
24
+ "server_parameters": {
25
+ "model": "meta-llama/Meta-Llama-3-70B-Instruct",
26
+ "tensor_parallel_size": 4,
27
+ "swap_space": 16,
28
+ "disable_log_stats": "",
29
+ "disable_log_requests": "",
30
+ "load_format": "dummy"
31
+ },
32
+ "client_parameters": {
33
+ "model": "meta-llama/Meta-Llama-3-70B-Instruct",
34
+ "backend": "vllm",
35
+ "dataset_name": "sharegpt",
36
+ "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
37
+ "num_prompts": 200
38
+ }
39
+ },
40
+ {
41
+ "test_name": "serving_mixtral8x7B_tp2_sharegpt",
42
+ "qps_list": [1, 4, 16, "inf"],
43
+ "server_parameters": {
44
+ "model": "mistralai/Mixtral-8x7B-Instruct-v0.1",
45
+ "tensor_parallel_size": 2,
46
+ "swap_space": 16,
47
+ "disable_log_stats": "",
48
+ "disable_log_requests": "",
49
+ "load_format": "dummy"
50
+ },
51
+ "client_parameters": {
52
+ "model": "mistralai/Mixtral-8x7B-Instruct-v0.1",
53
+ "backend": "vllm",
54
+ "dataset_name": "sharegpt",
55
+ "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
56
+ "num_prompts": 200
57
+ }
58
+ }
59
+]
.buildkite/nightly-benchmarks/throughput-tests.json
new
+35
@@ -0,0 +1,35 @@
1
+[
2
+ {
3
+ "test_name": "throughput_llama8B_tp1",
4
+ "parameters": {
5
+ "model": "meta-llama/Meta-Llama-3-8B",
6
+ "tensor_parallel_size": 1,
7
+ "load_format": "dummy",
8
+ "dataset": "./ShareGPT_V3_unfiltered_cleaned_split.json",
9
+ "num_prompts": 200,
10
+ "backend": "vllm"
11
+ }
12
+ },
13
+ {
14
+ "test_name": "throughput_llama70B_tp4",
15
+ "parameters": {
16
+ "model": "meta-llama/Meta-Llama-3-70B-Instruct",
17
+ "tensor_parallel_size": 4,
18
+ "load_format": "dummy",
19
+ "dataset": "./ShareGPT_V3_unfiltered_cleaned_split.json",
20
+ "num_prompts": 200,
21
+ "backend": "vllm"
22
+ }
23
+ },
24
+ {
25
+ "test_name": "throughput_mixtral8x7B_tp2",
26
+ "parameters": {
27
+ "model": "mistralai/Mixtral-8x7B-Instruct-v0.1",
28
+ "tensor_parallel_size": 2,
29
+ "load_format": "dummy",
30
+ "dataset": "./ShareGPT_V3_unfiltered_cleaned_split.json",
31
+ "num_prompts": 200,
32
+ "backend": "vllm"
33
+ }
34
+ }
35
+]
benchmarks/benchmark_latency.py
+25
@@ -10,6 +10,7 @@ import torch
10
from tqdm import tqdm
11
12
from vllm import LLM, SamplingParams
13
+from vllm.engine.arg_utils import EngineArgs
14
from vllm.inputs import PromptStrictInputs
15
from vllm.model_executor.layers.quantization import QUANTIZATION_METHODS
16
@@ -37,6 +38,7 @@ def main(args: argparse.Namespace):
38
download_dir=args.download_dir,
39
block_size=args.block_size,
40
gpu_memory_utilization=args.gpu_memory_utilization,
41
+ load_format=args.load_format,
42
distributed_executor_backend=args.distributed_executor_backend)
43
44
sampling_params = SamplingParams(
@@ -222,6 +224,29 @@ if __name__ == '__main__':
224
help='the fraction of GPU memory to be used for '
225
'the model executor, which can range from 0 to 1.'
226
'If unspecified, will use the default value of 0.9.')
227
+ parser.add_argument(
228
+ '--load-format',
229
+ type=str,
230
+ default=EngineArgs.load_format,
231
+ choices=[
232
+ 'auto', 'pt', 'safetensors', 'npcache', 'dummy', 'tensorizer',
233
+ 'bitsandbytes'
234
+ ],
235
+ help='The format of the model weights to load.\n\n'
236
+ '* "auto" will try to load the weights in the safetensors format '
237
+ 'and fall back to the pytorch bin format if safetensors format '
238
+ 'is not available.\n'
239
+ '* "pt" will load the weights in the pytorch bin format.\n'
240
+ '* "safetensors" will load the weights in the safetensors format.\n'
241
+ '* "npcache" will load the weights in pytorch format and store '
242
+ 'a numpy cache to speed up the loading.\n'
243
+ '* "dummy" will initialize the weights with random values, '
244
+ 'which is mainly for profiling.\n'
245
+ '* "tensorizer" will load the weights using tensorizer from '
246
+ 'CoreWeave. See the Tensorize vLLM Model script in the Examples'
247
+ 'section for more information.\n'
248
+ '* "bitsandbytes" will load the weights using bitsandbytes '
249
+ 'quantization.\n')
250
parser.add_argument(
251
'--distributed-executor-backend',
252
choices=['ray', 'mp'],
benchmarks/benchmark_serving.py
+11
@@ -499,6 +499,8 @@ def main(args: argparse.Namespace):
499
# Save to file
500
base_model_id = model_id.split("/")[-1]
501
file_name = f"{backend}-{args.request_rate}qps-{base_model_id}-{current_dt}.json" #noqa
502
+ if args.result_filename:
503
+ file_name = args.result_filename
504
if args.result_dir:
505
file_name = os.path.join(args.result_dir, file_name)
506
with open(file_name, "w") as outfile:
@@ -639,6 +641,15 @@ if __name__ == "__main__":
641
help="Specify directory to save benchmark json results."
642
"If not specified, results are saved in the current directory.",
643
)
644
+ parser.add_argument(
645
+ "--result-filename",
646
+ type=str,
647
+ default=None,
648
+ help="Specify the filename to save benchmark json results."
649
+ "If not specified, results will be saved in "
650
+ "{backend}-{args.request_rate}qps-{base_model_id}-{current_dt}.json"
651
+ " format.",
652
+ )
653
654
args = parser.parse_args()
655
main(args)
benchmarks/benchmark_throughput.py
+27
-1
@@ -10,6 +10,7 @@ from tqdm import tqdm
10
from transformers import (AutoModelForCausalLM, AutoTokenizer,
11
PreTrainedTokenizerBase)
12
13
+from vllm.engine.arg_utils import EngineArgs
14
from vllm.model_executor.layers.quantization import QUANTIZATION_METHODS
15
16
@@ -81,6 +82,7 @@ def run_vllm(
82
distributed_executor_backend: Optional[str],
83
gpu_memory_utilization: float = 0.9,
84
download_dir: Optional[str] = None,
85
+ load_format: str = EngineArgs.load_format,
86
) -> float:
87
from vllm import LLM, SamplingParams
88
llm = LLM(
@@ -102,6 +104,7 @@ def run_vllm(
104
enable_chunked_prefill=enable_chunked_prefill,
105
max_num_batched_tokens=max_num_batched_tokens,
106
distributed_executor_backend=distributed_executor_backend,
107
+ load_format=load_format,
108
)
109
110
# Add the requests to the engine.
@@ -228,7 +231,7 @@ def main(args: argparse.Namespace):
231
args.quantization_param_path, args.device,
232
args.enable_prefix_caching, args.enable_chunked_prefill,
233
args.max_num_batched_tokens, args.distributed_executor_backend,
231
- args.gpu_memory_utilization, args.download_dir)
234
+ args.gpu_memory_utilization, args.download_dir, args.load_format)
235
elif args.backend == "hf":
236
assert args.tensor_parallel_size == 1
237
elapsed_time = run_hf(requests, args.model, tokenizer, args.n,
@@ -377,6 +380,29 @@ if __name__ == "__main__":
380
help='Backend to use for distributed serving. When more than 1 GPU '
381
'is used, will be automatically set to "ray" if installed '
382
'or "mp" (multiprocessing) otherwise.')
383
+ parser.add_argument(
384
+ '--load-format',
385
+ type=str,
386
+ default=EngineArgs.load_format,
387
+ choices=[
388
+ 'auto', 'pt', 'safetensors', 'npcache', 'dummy', 'tensorizer',
389
+ 'bitsandbytes'
390
+ ],
391
+ help='The format of the model weights to load.\n\n'
392
+ '* "auto" will try to load the weights in the safetensors format '
393
+ 'and fall back to the pytorch bin format if safetensors format '
394
+ 'is not available.\n'
395
+ '* "pt" will load the weights in the pytorch bin format.\n'
396
+ '* "safetensors" will load the weights in the safetensors format.\n'
397
+ '* "npcache" will load the weights in pytorch format and store '
398
+ 'a numpy cache to speed up the loading.\n'
399
+ '* "dummy" will initialize the weights with random values, '
400
+ 'which is mainly for profiling.\n'
401
+ '* "tensorizer" will load the weights using tensorizer from '
402
+ 'CoreWeave. See the Tensorize vLLM Model script in the Examples'
403
+ 'section for more information.\n'
404
+ '* "bitsandbytes" will load the weights using bitsandbytes '
405
+ 'quantization.\n')
406
args = parser.parse_args()
407
if args.tokenizer is None:
408
args.tokenizer = args.model