@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 319ad7f1d3
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 23:15 UTC

[CI/Build][Misc] Add CI that benchmarks vllm performance on those PRs with `perf-benchmarks` label (#5073)

Co-authored-by: simon-mo <simon.mo@hey.com>

Kuntai Du committed Jun 13, 2024 at 22:36 UTC 319ad7f1d386699e94f629341c9988a926821f24
13 files changed +880 -41
.buildkite/nightly-benchmarks/README.md new
+98
@@ -0,0 +1,98 @@
1 +# vLLM benchmark suite
2 +
3 +## Introduction
4 +
5 +This directory contains the performance benchmarking CI for vllm.
6 +The goal is to help developers know the impact of their PRs on the performance of vllm.
7 +
8 +This benchmark will be *triggered* upon:
9 +- A PR being merged into vllm.
10 +- Every commit for those PRs with `perf-benchmarks` label.
11 +
12 +**Benchmarking Coverage**: latency, throughput and fix-qps serving on A100 (the support for more GPUs is comming later), with different models.
13 +
14 +**Benchmarking Duration**: about 1hr.
15 +
16 +## Configuring the workload for the quick benchmark
17 +
18 +The workload of the quick benchmark contains two parts: latency tests in `latency-tests.json`, throughput tests in `throughput-tests.json` and serving tests in `serving-tests.json`.
19 +
20 +### Latency test
21 +
22 +Here is an example of one test inside `latency-tests.json`:
23 +
24 +```json
25 +[
26 + ...
27 + {
28 + "test_name": "latency_llama8B_tp1",
29 + "parameters": {
30 + "model": "meta-llama/Meta-Llama-3-8B",
31 + "tensor_parallel_size": 1,
32 + "load_format": "dummy",
33 + "num_iters_warmup": 5,
34 + "num_iters": 15
35 + }
36 + },
37 + ...
38 +]
39 +```
40 +
41 +In this example:
42 +- The `test_name` attributes is a unique identifier for the test. In `latency-tests.json`, it must start with `latency_`.
43 +- The `parameters` attribute control the command line arguments to be used for `benchmark_latency.py`. Note that please use underline `_` instead of the dash `-` when specifying the command line arguments, and `run-benchmarks-suite.sh` will convert the underline to dash when feeding the arguments to `benchmark_latency.py`. For example, the corresponding command line arguments for `benchmark_latency.py` will be `--model meta-llama/Meta-Llama-3-8B --tensor-parallel-size 1 --load-format dummy --num-iters-warmup 5 --num-iters 15`
44 +
45 +Note that the performance numbers are highly sensitive to the value of the parameters. Please make sure the parameters are set correctly.
46 +
47 +WARNING: The benchmarking script will save json results by itself, so please do not configure `--output-json` parameter in the json file.
48 +
49 +
50 +### Throughput test
51 +The tests are specified in `throughput-tests.json`. The syntax is similar to `latency-tests.json`, except for that the parameters will be fed forward to `benchmark_throughput.py`.
52 +
53 +The number of this test is also stable -- a slight change on the value of this number might vary the performance numbers by a lot.
54 +
55 +### Serving test
56 +We test the throughput by using `benchmark_serving.py` with request rate = inf to cover the online serving overhead. The corresponding parameters are in `serving-tests.json`, and here is an example:
57 +
58 +```
59 +[
60 + ...
61 + {
62 + "test_name": "serving_llama8B_tp1_sharegpt",
63 + "qps_list": [1, 4, 16, "inf"],
64 + "server_parameters": {
65 + "model": "meta-llama/Meta-Llama-3-8B",
66 + "tensor_parallel_size": 1,
67 + "swap_space": 16,
68 + "disable_log_stats": "",
69 + "disable_log_requests": "",
70 + "load_format": "dummy"
71 + },
72 + "client_parameters": {
73 + "model": "meta-llama/Meta-Llama-3-8B",
74 + "backend": "vllm",
75 + "dataset_name": "sharegpt",
76 + "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
77 + "num_prompts": 200
78 + }
79 + },
80 + ...
81 +]
82 +```
83 +
84 +Inside this example:
85 +- The `test_name` attribute is also a unique identifier for the test. It must start with `serving_`.
86 +- The `server-parameters` includes the command line arguments for vLLM server.
87 +- The `client-parameters` includes the command line arguments for `benchmark_serving.py`.
88 +- The `qps_list` controls the list of qps for test. It will be used to configure the `--request-rate` parameter in `benchmark_serving.py`
89 +
90 +The number of this test is less stable compared to the delay and latency benchmarks (due to randomized sharegpt dataset sampling inside `benchmark_serving.py`), but a large change on this number (e.g. 5% change) still vary the output greatly.
91 +
92 +WARNING: The benchmarking script will save json results by itself, so please do not configure `--save-results` or other results-saving-related parameters in `serving-tests.json`.
93 +
94 +## Visualizing the results
95 +The `convert-results-json-to-markdown.py` helps you put the benchmarking results inside a markdown table.
96 +You can find the result presented as a table inside the `buildkite/performance-benchmark` job page.
97 +If you do not see the table, please wait till the benchmark finish running.
98 +The JSON file is also attached within each buildkite job for further analysis.
\ No newline at end of file
.buildkite/nightly-benchmarks/benchmark-pipeline.yaml new
+61
@@ -0,0 +1,61 @@
1 +steps:
2 + - label: "Wait for container to be ready"
3 + agents:
4 + queue: A100
5 + plugins:
6 + - kubernetes:
7 + podSpec:
8 + containers:
9 + - image: badouralix/curl-jq
10 + command:
11 + - sh
12 + - .buildkite/nightly-benchmarks/scripts/wait-for-image.sh
13 + - wait
14 + - label: "A100 Benchmark"
15 + agents:
16 + queue: A100
17 + plugins:
18 + - kubernetes:
19 + podSpec:
20 + containers:
21 + - image: public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT
22 + command:
23 + - bash .buildkite/nightly-benchmarks/run-benchmarks-suite.sh
24 + resources:
25 + limits:
26 + nvidia.com/gpu: 8
27 + volumeMounts:
28 + - name: devshm
29 + mountPath: /dev/shm
30 + env:
31 + - name: VLLM_USAGE_SOURCE
32 + value: ci-test
33 + - name: HF_TOKEN
34 + valueFrom:
35 + secretKeyRef:
36 + name: hf-token-secret
37 + key: token
38 + nodeSelector:
39 + nvidia.com/gpu.product: NVIDIA-A100-SXM4-80GB
40 + volumes:
41 + - name: devshm
42 + emptyDir:
43 + medium: Memory
44 + # - label: "H100: NVIDIA SMI"
45 + # agents:
46 + # queue: H100
47 + # plugins:
48 + # - docker#v5.11.0:
49 + # image: public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT
50 + # command:
51 + # - bash
52 + # - .buildkite/nightly-benchmarks/run-benchmarks-suite.sh
53 + # mount-buildkite-agent: true
54 + # propagate-environment: true
55 + # propagate-uid-gid: false
56 + # ipc: host
57 + # gpus: all
58 + # environment:
59 + # - VLLM_USAGE_SOURCE
60 + # - HF_TOKEN
61 +
.buildkite/nightly-benchmarks/kickoff-pipeline.sh
+2 -1
@@ -1,5 +1,6 @@
1 #!/usr/bin/env bash
2
3 +# NOTE(simon): this script runs inside a buildkite agent with CPU only access.
4 set -euo pipefail
5
6 # Install system packages
@@ -23,4 +24,4 @@ if [ "$BUILDKITE_PULL_REQUEST" != "false" ]; then
24 fi
25
26 # Upload sample.yaml
26 -buildkite-agent pipeline upload .buildkite/nightly-benchmarks/sample.yaml
27 +buildkite-agent pipeline upload .buildkite/nightly-benchmarks/benchmark-pipeline.yaml
.buildkite/nightly-benchmarks/latency-tests.json new
+32
@@ -0,0 +1,32 @@
1 +[
2 + {
3 + "test_name": "latency_llama8B_tp1",
4 + "parameters": {
5 + "model": "meta-llama/Meta-Llama-3-8B",
6 + "tensor_parallel_size": 1,
7 + "load_format": "dummy",
8 + "num_iters_warmup": 5,
9 + "num_iters": 15
10 + }
11 + },
12 + {
13 + "test_name": "latency_llama70B_tp4",
14 + "parameters": {
15 + "model": "meta-llama/Meta-Llama-3-70B-Instruct",
16 + "tensor_parallel_size": 4,
17 + "load_format": "dummy",
18 + "num-iters-warmup": 5,
19 + "num-iters": 15
20 + }
21 + },
22 + {
23 + "test_name": "latency_mixtral8x7B_tp2",
24 + "parameters": {
25 + "model": "mistralai/Mixtral-8x7B-Instruct-v0.1",
26 + "tensor_parallel_size": 2,
27 + "load_format": "dummy",
28 + "num-iters-warmup": 5,
29 + "num-iters": 15
30 + }
31 + }
32 +]
.buildkite/nightly-benchmarks/run-benchmarks-suite.sh new
+358
@@ -0,0 +1,358 @@
1 +#!/bin/bash
2 +
3 +# This script should be run inside the CI process
4 +# This script assumes that we are already inside the vllm/ directory
5 +# Benchmarking results will be available inside vllm/benchmarks/results/
6 +
7 +# Do not set -e, as the mixtral 8x22B model tends to crash occasionally
8 +# and we still want to see other benchmarking results even when mixtral crashes.
9 +set -o pipefail
10 +
11 +check_gpus() {
12 + # check the number of GPUs and GPU type.
13 + declare -g gpu_count=$(nvidia-smi --list-gpus | wc -l)
14 + if [[ $gpu_count -gt 0 ]]; then
15 + echo "GPU found."
16 + else
17 + echo "Need at least 1 GPU to run benchmarking."
18 + exit 1
19 + fi
20 + declare -g gpu_type=$(echo $(nvidia-smi --query-gpu=name --format=csv,noheader) | awk '{print $2}')
21 + echo "GPU type is $gpu_type"
22 +}
23 +
24 +check_hf_token() {
25 + # check if HF_TOKEN is available and valid
26 + if [[ -z "$HF_TOKEN" ]]; then
27 + echo "Error: HF_TOKEN is not set."
28 + exit 1
29 + elif [[ ! "$HF_TOKEN" =~ ^hf_ ]]; then
30 + echo "Error: HF_TOKEN does not start with 'hf_'."
31 + exit 1
32 + else
33 + echo "HF_TOKEN is set and valid."
34 + fi
35 +}
36 +
37 +json2args() {
38 + # transforms the JSON string to command line args, and '_' is replaced to '-'
39 + # example:
40 + # input: { "model": "meta-llama/Llama-2-7b-chat-hf", "tensor_parallel_size": 1 }
41 + # output: --model meta-llama/Llama-2-7b-chat-hf --tensor-parallel-size 1
42 + local json_string=$1
43 + local args=$(
44 + echo "$json_string" | jq -r '
45 + to_entries |
46 + map("--" + (.key | gsub("_"; "-")) + " " + (.value | tostring)) |
47 + join(" ")
48 + '
49 + )
50 + echo "$args"
51 +}
52 +
53 +wait_for_server() {
54 + # wait for vllm server to start
55 + # return 1 if vllm server crashes
56 + timeout 1200 bash -c '
57 + until curl localhost:8000/v1/completions; do
58 + sleep 1
59 + done' && return 0 || return 1
60 +}
61 +
62 +kill_gpu_processes() {
63 + # kill all processes on GPU.
64 + pids=$(nvidia-smi --query-compute-apps=pid --format=csv,noheader)
65 + if [ -z "$pids" ]; then
66 + echo "No GPU processes found."
67 + else
68 + for pid in $pids; do
69 + kill -9 "$pid"
70 + echo "Killed process with PID: $pid"
71 + done
72 +
73 + echo "All GPU processes have been killed."
74 + fi
75 +
76 + # waiting for GPU processes to be fully killed
77 + sleep 10
78 +
79 + # remove vllm config file
80 + rm -rf ~/.config/vllm
81 +
82 + # Print the GPU memory usage
83 + # so that we know if all GPU processes are killed.
84 + gpu_memory_usage=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits -i 0)
85 + # The memory usage should be 0 MB.
86 + echo "GPU 0 Memory Usage: $gpu_memory_usage MB"
87 +}
88 +
89 +upload_to_buildkite() {
90 + # upload the benchmarking results to buildkite
91 +
92 + # if the agent binary is not found, skip uploading the results, exit 0
93 + if [ ! -f /workspace/buildkite-agent ]; then
94 + echo "buildkite-agent binary not found. Skip uploading the results."
95 + return 0
96 + fi
97 + /workspace/buildkite-agent annotate --style "info" --context "benchmark-results" < $RESULTS_FOLDER/benchmark_results.md
98 + /workspace/buildkite-agent artifact upload "$RESULTS_FOLDER/*"
99 +}
100 +
101 +run_latency_tests() {
102 + # run latency tests using `benchmark_latency.py`
103 + # $1: a json file specifying latency test cases
104 +
105 + local latency_test_file
106 + latency_test_file=$1
107 +
108 + # Iterate over latency tests
109 + jq -c '.[]' "$latency_test_file" | while read -r params; do
110 + # get the test name, and append the GPU type back to it.
111 + test_name=$(echo "$params" | jq -r '.test_name')
112 + if [[ ! "$test_name" =~ ^latency_ ]]; then
113 + echo "In latency-test.json, test_name must start with \"latency_\"."
114 + exit 1
115 + fi
116 +
117 + # if TEST_SELECTOR is set, only run the test cases that match the selector
118 + if [[ -n "$TEST_SELECTOR" ]] && [[ ! "$test_name" =~ $TEST_SELECTOR ]]; then
119 + echo "Skip test case $test_name."
120 + continue
121 + fi
122 +
123 + # get arguments
124 + latency_params=$(echo "$params" | jq -r '.parameters')
125 + latency_args=$(json2args "$latency_params")
126 +
127 + # check if there is enough GPU to run the test
128 + tp=$(echo "$latency_params" | jq -r '.tensor_parallel_size')
129 + if [[ $gpu_count -lt $tp ]]; then
130 + echo "Required tensor-parallel-size $tp but only $gpu_count GPU found. Skip testcase $testname."
131 + continue
132 + fi
133 +
134 + latency_command="python3 benchmark_latency.py \
135 + --output-json $RESULTS_FOLDER/${test_name}.json \
136 + $latency_args"
137 +
138 + echo "Running test case $test_name"
139 + echo "Latency command: $latency_command"
140 +
141 + # recoding benchmarking command ang GPU command
142 + jq_output=$(jq -n \
143 + --arg latency "$latency_command" \
144 + --arg gpu "$gpu_type" \
145 + '{
146 + latency_command: $latency,
147 + gpu_type: $gpu
148 + }')
149 + echo "$jq_output" > "$RESULTS_FOLDER/$test_name.commands"
150 +
151 + # run the benchmark
152 + eval "$latency_command"
153 +
154 + kill_gpu_processes
155 +
156 + done
157 +}
158 +
159 +
160 +run_throughput_tests() {
161 + # run throughput tests using `benchmark_throughput.py`
162 + # $1: a json file specifying throughput test cases
163 +
164 + local throughput_test_file
165 + throughput_test_file=$1
166 +
167 + # Iterate over throughput tests
168 + jq -c '.[]' "$throughput_test_file" | while read -r params; do
169 + # get the test name, and append the GPU type back to it.
170 + test_name=$(echo "$params" | jq -r '.test_name')
171 + if [[ ! "$test_name" =~ ^throughput_ ]]; then
172 + echo "In throughput-test.json, test_name must start with \"throughput_\"."
173 + exit 1
174 + fi
175 +
176 + # if TEST_SELECTOR is set, only run the test cases that match the selector
177 + if [[ -n "$TEST_SELECTOR" ]] && [[ ! "$test_name" =~ $TEST_SELECTOR ]]; then
178 + echo "Skip test case $test_name."
179 + continue
180 + fi
181 +
182 + # get arguments
183 + throughput_params=$(echo "$params" | jq -r '.parameters')
184 + throughput_args=$(json2args "$throughput_params")
185 +
186 + # check if there is enough GPU to run the test
187 + tp=$(echo $throughput_params | jq -r '.tensor_parallel_size')
188 + if [[ $gpu_count -lt $tp ]]; then
189 + echo "Required tensor-parallel-size $tp but only $gpu_count GPU found. Skip testcase $testname."
190 + continue
191 + fi
192 +
193 + throughput_command="python3 benchmark_throughput.py \
194 + --output-json $RESULTS_FOLDER/${test_name}.json \
195 + $throughput_args"
196 +
197 + echo "Running test case $test_name"
198 + echo "Throughput command: $throughput_command"
199 + # recoding benchmarking command ang GPU command
200 + jq_output=$(jq -n \
201 + --arg command "$throughput_command" \
202 + --arg gpu "$gpu_type" \
203 + '{
204 + throughput_command: $command,
205 + gpu_type: $gpu
206 + }')
207 + echo "$jq_output" > "$RESULTS_FOLDER/$test_name.commands"
208 +
209 + # run the benchmark
210 + eval "$throughput_command"
211 +
212 + kill_gpu_processes
213 +
214 + done
215 +}
216 +
217 +run_serving_tests() {
218 + # run serving tests using `benchmark_serving.py`
219 + # $1: a json file specifying serving test cases
220 +
221 + local serving_test_file
222 + serving_test_file=$1
223 +
224 + # Iterate over serving tests
225 + jq -c '.[]' "$serving_test_file" | while read -r params; do
226 + # get the test name, and append the GPU type back to it.
227 + test_name=$(echo "$params" | jq -r '.test_name')
228 + if [[ ! "$test_name" =~ ^serving_ ]]; then
229 + echo "In serving-test.json, test_name must start with \"serving_\"."
230 + exit 1
231 + fi
232 +
233 + # if TEST_SELECTOR is set, only run the test cases that match the selector
234 + if [[ -n "$TEST_SELECTOR" ]] && [[ ! "$test_name" =~ $TEST_SELECTOR ]]; then
235 + echo "Skip test case $test_name."
236 + continue
237 + fi
238 +
239 +
240 + # get client and server arguments
241 + server_params=$(echo "$params" | jq -r '.server_parameters')
242 + client_params=$(echo "$params" | jq -r '.client_parameters')
243 + server_args=$(json2args "$server_params")
244 + client_args=$(json2args "$client_params")
245 + qps_list=$(echo "$params" | jq -r '.qps_list')
246 + qps_list=$(echo "$qps_list" | jq -r '.[] | @sh')
247 + echo "Running over qps list $qps_list"
248 +
249 + # check if there is enough GPU to run the test
250 + tp=$(echo "$server_params" | jq -r '.tensor_parallel_size')
251 + if [[ $gpu_count -lt $tp ]]; then
252 + echo "Required tensor-parallel-size $tp but only $gpu_count GPU found. Skip testcase $testname."
253 + continue
254 + fi
255 +
256 + # check if server model and client model is aligned
257 + server_model=$(echo "$server_params" | jq -r '.model')
258 + client_model=$(echo "$client_params" | jq -r '.model')
259 + if [[ $server_model != "$client_model" ]]; then
260 + echo "Server model and client model must be the same. Skip testcase $testname."
261 + continue
262 + fi
263 +
264 + server_command="python3 \
265 + -m vllm.entrypoints.openai.api_server \
266 + $server_args"
267 +
268 + # run the server
269 + echo "Running test case $test_name"
270 + echo "Server command: $server_command"
271 + eval "$server_command" &
272 +
273 + # wait until the server is alive
274 + wait_for_server
275 + if [ $? -eq 0 ]; then
276 + echo ""
277 + echo "vllm server is up and running."
278 + else
279 + echo ""
280 + echo "vllm failed to start within the timeout period."
281 + fi
282 +
283 + # iterate over different QPS
284 + for qps in $qps_list; do
285 + # remove the surrounding single quote from qps
286 + if [[ "$qps" == *"inf"* ]]; then
287 + echo "qps was $qps"
288 + qps="inf"
289 + echo "now qps is $qps"
290 + fi
291 +
292 + new_test_name=$test_name"_qps_"$qps
293 +
294 + client_command="python3 benchmark_serving.py \
295 + --save-result \
296 + --result-dir $RESULTS_FOLDER \
297 + --result-filename ${new_test_name}.json \
298 + --request-rate $qps \
299 + $client_args"
300 +
301 + echo "Running test case $test_name with qps $qps"
302 + echo "Client command: $client_command"
303 +
304 + eval "$client_command"
305 +
306 + # record the benchmarking commands
307 + jq_output=$(jq -n \
308 + --arg server "$server_command" \
309 + --arg client "$client_command" \
310 + --arg gpu "$gpu_type" \
311 + '{
312 + server_command: $server,
313 + client_command: $client,
314 + gpu_type: $gpu
315 + }')
316 + echo "$jq_output" > "$RESULTS_FOLDER/${new_test_name}.commands"
317 +
318 + done
319 +
320 + # clean up
321 + kill_gpu_processes
322 + done
323 +}
324 +
325 +main() {
326 + check_gpus
327 + check_hf_token
328 +
329 + # dependencies
330 + (which wget && which curl) || (apt-get update && apt-get install -y wget curl)
331 + (which jq) || (apt-get update && apt-get -y install jq)
332 +
333 + # get the current IP address, required by benchmark_serving.py
334 + export VLLM_HOST_IP=$(hostname -I | awk '{print $1}')
335 + # turn of the reporting of the status of each request, to clean up the terminal output
336 + export VLLM_LOG_LEVEL="WARNING"
337 +
338 + # prepare for benchmarking
339 + cd benchmarks || exit 1
340 + wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
341 + declare -g RESULTS_FOLDER=results/
342 + mkdir -p $RESULTS_FOLDER
343 + QUICK_BENCHMARK_ROOT=../.buildkite/nightly-benchmarks/
344 +
345 + # benchmarking
346 + run_serving_tests $QUICK_BENCHMARK_ROOT/serving-tests.json
347 + run_latency_tests $QUICK_BENCHMARK_ROOT/latency-tests.json
348 + run_throughput_tests $QUICK_BENCHMARK_ROOT/throughput-tests.json
349 +
350 +
351 + # postprocess benchmarking results
352 + pip install tabulate pandas
353 + python3 $QUICK_BENCHMARK_ROOT/scripts/convert-results-json-to-markdown.py
354 +
355 + upload_to_buildkite
356 +}
357 +
358 +main "$@"
.buildkite/nightly-benchmarks/sample.yaml deleted
-39
@@ -1,39 +0,0 @@
1 -steps:
2 - # NOTE(simon): You can create separate blocks for different jobs
3 - - label: "A100: NVIDIA SMI"
4 - agents:
5 - queue: A100
6 - plugins:
7 - - kubernetes:
8 - podSpec:
9 - containers:
10 - # - image: us-central1-docker.pkg.dev/vllm-405802/vllm-ci-test-repo/vllm-test:$BUILDKITE_COMMIT
11 - # TODO(simon): check latest main branch or use the PR image.
12 - - image: us-central1-docker.pkg.dev/vllm-405802/vllm-ci-test-repo/vllm-test:45c35f0d58f4508bf43bd6af1d3d0d0ec0c915e6
13 - command:
14 - - bash -c 'nvidia-smi && nvidia-smi topo -m && pwd && ls'
15 - resources:
16 - limits:
17 - nvidia.com/gpu: 8
18 - volumeMounts:
19 - - name: devshm
20 - mountPath: /dev/shm
21 - nodeSelector:
22 - nvidia.com/gpu.product: NVIDIA-A100-SXM4-80GB
23 - volumes:
24 - - name: devshm
25 - emptyDir:
26 - medium: Memory
27 - # TODO(simon): bring H100 online
28 - # - label: "H100: NVIDIA SMI"
29 - # agents:
30 - # queue: H100
31 - # plugins:
32 - # - docker#v5.11.0:
33 - # image: us-central1-docker.pkg.dev/vllm-405802/vllm-ci-test-repo/vllm-test:45c35f0d58f4508bf43bd6af1d3d0d0ec0c915e6
34 - # command:
35 - # - bash -c 'nvidia-smi && nvidia-smi topo -m'
36 - # propagate-environment: true
37 - # ipc: host
38 - # gpus: all
39 -
.buildkite/nightly-benchmarks/scripts/convert-results-json-to-markdown.py new
+155
@@ -0,0 +1,155 @@
1 +import json
2 +from pathlib import Path
3 +
4 +import pandas as pd
5 +from tabulate import tabulate
6 +
7 +results_folder = Path("results/")
8 +
9 +# latency results and the keys that will be printed into markdown
10 +latency_results = []
11 +latency_column_mapping = {
12 + "test_name": "Test name",
13 + "gpu_type": "GPU",
14 + "avg_latency": "Average latency (s)",
15 + "P10": "P10 (s)",
16 + "P25": "P25 (s)",
17 + "P50": "P50 (s)",
18 + "P75": "P75 (s)",
19 + "P90": "P90 (s)",
20 +}
21 +
22 +# thoughput tests and the keys that will be printed into markdown
23 +throughput_results = []
24 +throughput_results_column_mapping = {
25 + "test_name": "Test name",
26 + "gpu_type": "GPU",
27 + "num_requests": "# of req.",
28 + "total_num_tokens": "Total # of tokens",
29 + "elapsed_time": "Elapsed time (s)",
30 + "requests_per_second": "Tput (req/s)",
31 + "tokens_per_second": "Tput (tok/s)",
32 +}
33 +
34 +# serving results and the keys that will be printed into markdown
35 +serving_results = []
36 +serving_column_mapping = {
37 + "test_name": "Test name",
38 + "gpu_type": "GPU",
39 + "completed": "# of req.",
40 + "request_throughput": "Tput (req/s)",
41 + "input_throughput": "Input Tput (tok/s)",
42 + "output_throughput": "Output Tput (tok/s)",
43 + "mean_ttft_ms": "Mean TTFT (ms)",
44 + # do not say TTFT again to avoid the table getting too wide
45 + "median_ttft_ms": "Median",
46 + "p99_ttft_ms": "P99",
47 + "mean_tpot_ms": "Mean TPOT (ms)",
48 + "median_tpot_ms": "Median",
49 + "p99_tpot_ms": "P99",
50 + "mean_itl_ms": "Mean ITL (ms)",
51 + "median_itl_ms": "Median",
52 + "p99_itl_ms": "P99",
53 +}
54 +
55 +for test_file in results_folder.glob("*.json"):
56 +
57 + with open(test_file, "r") as f:
58 + raw_result = json.loads(f.read())
59 +
60 + if "serving" in str(test_file):
61 + # this result is generated via `benchmark_serving.py`
62 +
63 + # attach the benchmarking command to raw_result
64 + with open(test_file.with_suffix(".commands"), "r") as f:
65 + command = json.loads(f.read())
66 + raw_result.update(command)
67 +
68 + # update the test name of this result
69 + raw_result.update({"test_name": test_file.stem})
70 +
71 + # add the result to raw_result
72 + serving_results.append(raw_result)
73 + continue
74 +
75 + elif "latency" in f.name:
76 + # this result is generated via `benchmark_latency.py`
77 +
78 + # attach the benchmarking command to raw_result
79 + with open(test_file.with_suffix(".commands"), "r") as f:
80 + command = json.loads(f.read())
81 + raw_result.update(command)
82 +
83 + # update the test name of this result
84 + raw_result.update({"test_name": test_file.stem})
85 +
86 + # get different percentiles
87 + for perc in [10, 25, 50, 75, 90]:
88 + raw_result.update(
89 + {f"P{perc}": raw_result["percentiles"][str(perc)]})
90 +
91 + # add the result to raw_result
92 + latency_results.append(raw_result)
93 + continue
94 +
95 + elif "throughput" in f.name:
96 + # this result is generated via `benchmark_throughput.py`
97 +
98 + # attach the benchmarking command to raw_result
99 + with open(test_file.with_suffix(".commands"), "r") as f:
100 + command = json.loads(f.read())
101 + raw_result.update(command)
102 +
103 + # update the test name of this result
104 + raw_result.update({"test_name": test_file.stem})
105 +
106 + # add the result to raw_result
107 + throughput_results.append(raw_result)
108 + continue
109 +
110 + print(f"Skipping {test_file}")
111 +
112 +latency_results = pd.DataFrame.from_dict(latency_results)
113 +serving_results = pd.DataFrame.from_dict(serving_results)
114 +throughput_results = pd.DataFrame.from_dict(throughput_results)
115 +
116 +# remapping the key, for visualization purpose
117 +if not latency_results.empty:
118 + latency_results = latency_results[list(
119 + latency_column_mapping.keys())].rename(columns=latency_column_mapping)
120 +if not serving_results.empty:
121 + serving_results = serving_results[list(
122 + serving_column_mapping.keys())].rename(columns=serving_column_mapping)
123 +if not throughput_results.empty:
124 + throughput_results = throughput_results[list(
125 + throughput_results_column_mapping.keys())].rename(
126 + columns=throughput_results_column_mapping)
127 +
128 +# get markdown tables
129 +latency_md_table = tabulate(latency_results,
130 + headers='keys',
131 + tablefmt='pipe',
132 + showindex=False)
133 +serving_md_table = tabulate(serving_results,
134 + headers='keys',
135 + tablefmt='pipe',
136 + showindex=False)
137 +throughput_md_table = tabulate(throughput_results,
138 + headers='keys',
139 + tablefmt='pipe',
140 + showindex=False)
141 +
142 +# document the result
143 +with open(results_folder / "benchmark_results.md", "w") as f:
144 + if not latency_results.empty:
145 + f.write("## Latency tests\n")
146 + f.write(latency_md_table)
147 + f.write("\n")
148 + if not throughput_results.empty:
149 + f.write("## Throughput tests\n")
150 + f.write(throughput_md_table)
151 + f.write("\n")
152 + if not serving_results.empty:
153 + f.write("## Serving tests\n")
154 + f.write(serving_md_table)
155 + f.write("\n")
.buildkite/nightly-benchmarks/scripts/wait-for-image.sh new
+17
@@ -0,0 +1,17 @@
1 +#!/bin/sh
2 +TOKEN=$(curl -s -L "https://public.ecr.aws/token?service=public.ecr.aws&scope=repository:q9t5s3a7/vllm-ci-test-repo:pull" | jq -r .token)
3 +URL="https://public.ecr.aws/v2/q9t5s3a7/vllm-ci-test-repo/manifests/$BUILDKITE_COMMIT"
4 +
5 +retries=0
6 +while [ $retries -lt 1000 ]; do
7 + if [ $(curl -s -L -H "Authorization: Bearer $TOKEN" -o /dev/null -w "%{http_code}" $URL) -eq 200 ]; then
8 + exit 0
9 + fi
10 +
11 + echo "Waiting for image to be available..."
12 +
13 + retries=$((retries + 1))
14 + sleep 5
15 +done
16 +
17 +exit 1
\ No newline at end of file
.buildkite/nightly-benchmarks/serving-tests.json new
+59
@@ -0,0 +1,59 @@
1 +[
2 + {
3 + "test_name": "serving_llama8B_tp1_sharegpt",
4 + "qps_list": [1, 4, 16, "inf"],
5 + "server_parameters": {
6 + "model": "meta-llama/Meta-Llama-3-8B",
7 + "tensor_parallel_size": 1,
8 + "swap_space": 16,
9 + "disable_log_stats": "",
10 + "disable_log_requests": "",
11 + "load_format": "dummy"
12 + },
13 + "client_parameters": {
14 + "model": "meta-llama/Meta-Llama-3-8B",
15 + "backend": "vllm",
16 + "dataset_name": "sharegpt",
17 + "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
18 + "num_prompts": 200
19 + }
20 + },
21 + {
22 + "test_name": "serving_llama70B_tp4_sharegpt",
23 + "qps_list": [1, 4, 16, "inf"],
24 + "server_parameters": {
25 + "model": "meta-llama/Meta-Llama-3-70B-Instruct",
26 + "tensor_parallel_size": 4,
27 + "swap_space": 16,
28 + "disable_log_stats": "",
29 + "disable_log_requests": "",
30 + "load_format": "dummy"
31 + },
32 + "client_parameters": {
33 + "model": "meta-llama/Meta-Llama-3-70B-Instruct",
34 + "backend": "vllm",
35 + "dataset_name": "sharegpt",
36 + "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
37 + "num_prompts": 200
38 + }
39 + },
40 + {
41 + "test_name": "serving_mixtral8x7B_tp2_sharegpt",
42 + "qps_list": [1, 4, 16, "inf"],
43 + "server_parameters": {
44 + "model": "mistralai/Mixtral-8x7B-Instruct-v0.1",
45 + "tensor_parallel_size": 2,
46 + "swap_space": 16,
47 + "disable_log_stats": "",
48 + "disable_log_requests": "",
49 + "load_format": "dummy"
50 + },
51 + "client_parameters": {
52 + "model": "mistralai/Mixtral-8x7B-Instruct-v0.1",
53 + "backend": "vllm",
54 + "dataset_name": "sharegpt",
55 + "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
56 + "num_prompts": 200
57 + }
58 + }
59 +]
.buildkite/nightly-benchmarks/throughput-tests.json new
+35
@@ -0,0 +1,35 @@
1 +[
2 + {
3 + "test_name": "throughput_llama8B_tp1",
4 + "parameters": {
5 + "model": "meta-llama/Meta-Llama-3-8B",
6 + "tensor_parallel_size": 1,
7 + "load_format": "dummy",
8 + "dataset": "./ShareGPT_V3_unfiltered_cleaned_split.json",
9 + "num_prompts": 200,
10 + "backend": "vllm"
11 + }
12 + },
13 + {
14 + "test_name": "throughput_llama70B_tp4",
15 + "parameters": {
16 + "model": "meta-llama/Meta-Llama-3-70B-Instruct",
17 + "tensor_parallel_size": 4,
18 + "load_format": "dummy",
19 + "dataset": "./ShareGPT_V3_unfiltered_cleaned_split.json",
20 + "num_prompts": 200,
21 + "backend": "vllm"
22 + }
23 + },
24 + {
25 + "test_name": "throughput_mixtral8x7B_tp2",
26 + "parameters": {
27 + "model": "mistralai/Mixtral-8x7B-Instruct-v0.1",
28 + "tensor_parallel_size": 2,
29 + "load_format": "dummy",
30 + "dataset": "./ShareGPT_V3_unfiltered_cleaned_split.json",
31 + "num_prompts": 200,
32 + "backend": "vllm"
33 + }
34 + }
35 +]
benchmarks/benchmark_latency.py
+25
@@ -10,6 +10,7 @@ import torch
10 from tqdm import tqdm
11
12 from vllm import LLM, SamplingParams
13 +from vllm.engine.arg_utils import EngineArgs
14 from vllm.inputs import PromptStrictInputs
15 from vllm.model_executor.layers.quantization import QUANTIZATION_METHODS
16
@@ -37,6 +38,7 @@ def main(args: argparse.Namespace):
38 download_dir=args.download_dir,
39 block_size=args.block_size,
40 gpu_memory_utilization=args.gpu_memory_utilization,
41 + load_format=args.load_format,
42 distributed_executor_backend=args.distributed_executor_backend)
43
44 sampling_params = SamplingParams(
@@ -222,6 +224,29 @@ if __name__ == '__main__':
224 help='the fraction of GPU memory to be used for '
225 'the model executor, which can range from 0 to 1.'
226 'If unspecified, will use the default value of 0.9.')
227 + parser.add_argument(
228 + '--load-format',
229 + type=str,
230 + default=EngineArgs.load_format,
231 + choices=[
232 + 'auto', 'pt', 'safetensors', 'npcache', 'dummy', 'tensorizer',
233 + 'bitsandbytes'
234 + ],
235 + help='The format of the model weights to load.\n\n'
236 + '* "auto" will try to load the weights in the safetensors format '
237 + 'and fall back to the pytorch bin format if safetensors format '
238 + 'is not available.\n'
239 + '* "pt" will load the weights in the pytorch bin format.\n'
240 + '* "safetensors" will load the weights in the safetensors format.\n'
241 + '* "npcache" will load the weights in pytorch format and store '
242 + 'a numpy cache to speed up the loading.\n'
243 + '* "dummy" will initialize the weights with random values, '
244 + 'which is mainly for profiling.\n'
245 + '* "tensorizer" will load the weights using tensorizer from '
246 + 'CoreWeave. See the Tensorize vLLM Model script in the Examples'
247 + 'section for more information.\n'
248 + '* "bitsandbytes" will load the weights using bitsandbytes '
249 + 'quantization.\n')
250 parser.add_argument(
251 '--distributed-executor-backend',
252 choices=['ray', 'mp'],
benchmarks/benchmark_serving.py
+11
@@ -499,6 +499,8 @@ def main(args: argparse.Namespace):
499 # Save to file
500 base_model_id = model_id.split("/")[-1]
501 file_name = f"{backend}-{args.request_rate}qps-{base_model_id}-{current_dt}.json" #noqa
502 + if args.result_filename:
503 + file_name = args.result_filename
504 if args.result_dir:
505 file_name = os.path.join(args.result_dir, file_name)
506 with open(file_name, "w") as outfile:
@@ -639,6 +641,15 @@ if __name__ == "__main__":
641 help="Specify directory to save benchmark json results."
642 "If not specified, results are saved in the current directory.",
643 )
644 + parser.add_argument(
645 + "--result-filename",
646 + type=str,
647 + default=None,
648 + help="Specify the filename to save benchmark json results."
649 + "If not specified, results will be saved in "
650 + "{backend}-{args.request_rate}qps-{base_model_id}-{current_dt}.json"
651 + " format.",
652 + )
653
654 args = parser.parse_args()
655 main(args)
benchmarks/benchmark_throughput.py
+27 -1
@@ -10,6 +10,7 @@ from tqdm import tqdm
10 from transformers import (AutoModelForCausalLM, AutoTokenizer,
11 PreTrainedTokenizerBase)
12
13 +from vllm.engine.arg_utils import EngineArgs
14 from vllm.model_executor.layers.quantization import QUANTIZATION_METHODS
15
16
@@ -81,6 +82,7 @@ def run_vllm(
82 distributed_executor_backend: Optional[str],
83 gpu_memory_utilization: float = 0.9,
84 download_dir: Optional[str] = None,
85 + load_format: str = EngineArgs.load_format,
86 ) -> float:
87 from vllm import LLM, SamplingParams
88 llm = LLM(
@@ -102,6 +104,7 @@ def run_vllm(
104 enable_chunked_prefill=enable_chunked_prefill,
105 max_num_batched_tokens=max_num_batched_tokens,
106 distributed_executor_backend=distributed_executor_backend,
107 + load_format=load_format,
108 )
109
110 # Add the requests to the engine.
@@ -228,7 +231,7 @@ def main(args: argparse.Namespace):
231 args.quantization_param_path, args.device,
232 args.enable_prefix_caching, args.enable_chunked_prefill,
233 args.max_num_batched_tokens, args.distributed_executor_backend,
231 - args.gpu_memory_utilization, args.download_dir)
234 + args.gpu_memory_utilization, args.download_dir, args.load_format)
235 elif args.backend == "hf":
236 assert args.tensor_parallel_size == 1
237 elapsed_time = run_hf(requests, args.model, tokenizer, args.n,
@@ -377,6 +380,29 @@ if __name__ == "__main__":
380 help='Backend to use for distributed serving. When more than 1 GPU '
381 'is used, will be automatically set to "ray" if installed '
382 'or "mp" (multiprocessing) otherwise.')
383 + parser.add_argument(
384 + '--load-format',
385 + type=str,
386 + default=EngineArgs.load_format,
387 + choices=[
388 + 'auto', 'pt', 'safetensors', 'npcache', 'dummy', 'tensorizer',
389 + 'bitsandbytes'
390 + ],
391 + help='The format of the model weights to load.\n\n'
392 + '* "auto" will try to load the weights in the safetensors format '
393 + 'and fall back to the pytorch bin format if safetensors format '
394 + 'is not available.\n'
395 + '* "pt" will load the weights in the pytorch bin format.\n'
396 + '* "safetensors" will load the weights in the safetensors format.\n'
397 + '* "npcache" will load the weights in pytorch format and store '
398 + 'a numpy cache to speed up the loading.\n'
399 + '* "dummy" will initialize the weights with random values, '
400 + 'which is mainly for profiling.\n'
401 + '* "tensorizer" will load the weights using tensorizer from '
402 + 'CoreWeave. See the Tensorize vLLM Model script in the Examples'
403 + 'section for more information.\n'
404 + '* "bitsandbytes" will load the weights using bitsandbytes '
405 + 'quantization.\n')
406 args = parser.parse_args()
407 if args.tokenizer is None:
408 args.tokenizer = args.model