A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 22:00 UTC
[CI/Build] Add nightly benchmarking for tgi, tensorrt-llm and lmdeploy (#5362)
Kuntai Du committed
Jul 11, 2024 at 13:28 UTC
a4feba929b61f287ae0b7407c3d615c6dec193d6
18 files changed
+1623
-27
.buildkite/nightly-benchmarks/README.md
+1
@@ -1,5 +1,6 @@
1
# vLLM benchmark suite
2
3
+
4
## Introduction
5
6
This directory contains the performance benchmarking CI for vllm.
.buildkite/nightly-benchmarks/kickoff-pipeline.sh
deleted
-27
@@ -1,27 +0,0 @@
1
-#!/usr/bin/env bash
2
-
3
-# NOTE(simon): this script runs inside a buildkite agent with CPU only access.
4
-set -euo pipefail
5
-
6
-# Install system packages
7
-apt update
8
-apt install -y curl jq
9
-
10
-# Install minijinja for templating
11
-curl -sSfL https://github.com/mitsuhiko/minijinja/releases/latest/download/minijinja-cli-installer.sh | sh
12
-source $HOME/.cargo/env
13
-
14
-# If BUILDKITE_PULL_REQUEST != "false", then we check the PR labels using curl and jq
15
-if [ "$BUILDKITE_PULL_REQUEST" != "false" ]; then
16
- PR_LABELS=$(curl -s "https://api.github.com/repos/vllm-project/vllm/pulls/$BUILDKITE_PULL_REQUEST" | jq -r '.labels[].name')
17
-
18
- if [[ $PR_LABELS == *"perf-benchmarks"* ]]; then
19
- echo "This PR has the 'perf-benchmarks' label. Proceeding with the nightly benchmarks."
20
- else
21
- echo "This PR does not have the 'perf-benchmarks' label. Skipping the nightly benchmarks."
22
- exit 0
23
- fi
24
-fi
25
-
26
-# Upload sample.yaml
27
-buildkite-agent pipeline upload .buildkite/nightly-benchmarks/benchmark-pipeline.yaml
.buildkite/nightly-benchmarks/nightly-descriptions.md
new
+45
@@ -0,0 +1,45 @@
1
+
2
+# Nightly benchmark
3
+
4
+The main goal of this benchmarking is two-fold:
5
+- Performance clarity: Provide clarity on which one (vllm, tensorrt-llm, lmdeploy and tgi) leads in performance in what workload.
6
+- Reproducible: one can run the exact same set of benchmarking commands inside the exact same docker by following reproducing instructions in [reproduce.md]().
7
+
8
+
9
+## Docker images
10
+
11
+We benchmark vllm, tensorrt-llm, lmdeploy and tgi using the following docker images:
12
+- vllm/vllm-openai:v0.5.0.post1
13
+- nvcr.io/nvidia/tritonserver:24.04-trtllm-python-py3
14
+- openmmlab/lmdeploy:v0.5.0
15
+- ghcr.io/huggingface/text-generation-inference:2.1
16
+
17
+<!-- Please check <a href="artifact://workspace/build/buildkite/vllm/performance-benchmark/.buildkite/nightly-benchmarks/nightly-pipeline.yaml">nightly-pipeline.yaml</a> artifact for more details on how we deploy the docker images. -->
18
+
19
+
20
+## Hardware
21
+
22
+One AWS node with 8x NVIDIA A100 GPUs.
23
+
24
+
25
+## Workload description
26
+
27
+We benchmark vllm, tensorrt-llm, lmdeploy and tgi using the following workload:
28
+
29
+- Input length: randomly sample 500 prompts from ShareGPT dataset (with fixed random seed).
30
+- Output length: the corresponding output length of these 500 prompts.
31
+- Models: llama-3 8B, llama-3 70B, mixtral 8x7B.
32
+- Average QPS (query per second): 4 for the small model (llama-3 8B) and 2 for other two models. For each QPS, the arrival time of each query is determined using a random Poisson process (with fixed random seed).
33
+- Evaluation metrics: Throughput (higher the better), TTFT (time to the first token, lower the better), ITL (inter-token latency, lower the better).
34
+
35
+<!-- Check <a href="artifact://workspace/build/buildkite/vllm/performance-benchmark/.buildkite/nightly-benchmarks/tests/nightly-tests.json">nightly-tests.json</a> artifact for more details. -->
36
+
37
+## Plots
38
+
39
+In the following plots, the dot shows the mean and the error bar shows the standard error of the mean. Value 0 means that the corresponding benchmark crashed.
40
+
41
+<img src="artifact://nightly_results.png" alt="Benchmarking results" height=250 >
42
+
43
+## Results
44
+
45
+{nightly_results_benchmarking_table}
.buildkite/nightly-benchmarks/nightly-pipeline.yaml
new
+120
@@ -0,0 +1,120 @@
1
+common_pod_spec: &common_pod_spec
2
+ priorityClassName: perf-benchmark
3
+ nodeSelector:
4
+ nvidia.com/gpu.product: NVIDIA-A100-SXM4-80GB
5
+ volumes:
6
+ - name: devshm
7
+ emptyDir:
8
+ medium: Memory
9
+ - name: hf-cache
10
+ hostPath:
11
+ path: /root/.cache/huggingface
12
+ type: Directory
13
+
14
+common_container_settings: &common_container_settings
15
+ command:
16
+ - bash .buildkite/nightly-benchmarks/run-nightly-suite.sh
17
+ resources:
18
+ limits:
19
+ nvidia.com/gpu: 8
20
+ volumeMounts:
21
+ - name: devshm
22
+ mountPath: /dev/shm
23
+ - name: hf-cache
24
+ mountPath: /root/.cache/huggingface
25
+ env:
26
+ - name: VLLM_USAGE_SOURCE
27
+ value: ci-test
28
+ - name: HF_HOME
29
+ value: /root/.cache/huggingface
30
+ - name: VLLM_SOURCE_CODE_LOC
31
+ value: /workspace/build/buildkite/vllm/performance-benchmark
32
+ - name: HF_TOKEN
33
+ valueFrom:
34
+ secretKeyRef:
35
+ name: hf-token-secret
36
+ key: token
37
+
38
+steps:
39
+ - block: ":rocket: Ready for comparing vllm against alternatives? This will take 4 hours."
40
+ - label: "A100 trt benchmark"
41
+ priority: 100
42
+ agents:
43
+ queue: A100
44
+ plugins:
45
+ - kubernetes:
46
+ podSpec:
47
+ <<: *common_pod_spec
48
+ containers:
49
+ - image: nvcr.io/nvidia/tritonserver:24.04-trtllm-python-py3
50
+ <<: *common_container_settings
51
+
52
+ - label: "A100 lmdeploy benchmark"
53
+ priority: 100
54
+ agents:
55
+ queue: A100
56
+ plugins:
57
+ - kubernetes:
58
+ podSpec:
59
+ <<: *common_pod_spec
60
+ containers:
61
+ - image: openmmlab/lmdeploy:v0.5.0
62
+ <<: *common_container_settings
63
+
64
+
65
+ - label: "A100 vllm benchmark"
66
+ priority: 100
67
+ agents:
68
+ queue: A100
69
+ plugins:
70
+ - kubernetes:
71
+ podSpec:
72
+ <<: *common_pod_spec
73
+ containers:
74
+ - image: vllm/vllm-openai:latest
75
+ <<: *common_container_settings
76
+
77
+ - label: "A100 tgi benchmark"
78
+ priority: 100
79
+ agents:
80
+ queue: A100
81
+ plugins:
82
+ - kubernetes:
83
+ podSpec:
84
+ <<: *common_pod_spec
85
+ containers:
86
+ - image: ghcr.io/huggingface/text-generation-inference:2.1
87
+ <<: *common_container_settings
88
+
89
+ - wait
90
+
91
+ - label: "Plot"
92
+ priority: 100
93
+ agents:
94
+ queue: A100
95
+ plugins:
96
+ - kubernetes:
97
+ podSpec:
98
+ <<: *common_pod_spec
99
+ containers:
100
+ - image: vllm/vllm-openai:v0.5.0.post1
101
+ command:
102
+ - bash .buildkite/nightly-benchmarks/scripts/nightly-annotate.sh
103
+ resources:
104
+ limits:
105
+ nvidia.com/gpu: 8
106
+ volumeMounts:
107
+ - name: devshm
108
+ mountPath: /dev/shm
109
+ env:
110
+ - name: VLLM_USAGE_SOURCE
111
+ value: ci-test
112
+ - name: VLLM_SOURCE_CODE_LOC
113
+ value: /workspace/build/buildkite/vllm/performance-benchmark
114
+ - name: HF_TOKEN
115
+ valueFrom:
116
+ secretKeyRef:
117
+ name: hf-token-secret
118
+ key: token
119
+
120
+ - wait
\ No newline at end of file
.buildkite/nightly-benchmarks/run-nightly-suite.sh
new
+76
@@ -0,0 +1,76 @@
1
+#!/bin/bash
2
+
3
+set -o pipefail
4
+set -x
5
+
6
+check_gpus() {
7
+ # check the number of GPUs and GPU type.
8
+ declare -g gpu_count=$(nvidia-smi --list-gpus | wc -l)
9
+ if [[ $gpu_count -gt 0 ]]; then
10
+ echo "GPU found."
11
+ else
12
+ echo "Need at least 1 GPU to run benchmarking."
13
+ exit 1
14
+ fi
15
+ declare -g gpu_type=$(echo $(nvidia-smi --query-gpu=name --format=csv,noheader) | awk '{print $2}')
16
+ echo "GPU type is $gpu_type"
17
+}
18
+
19
+check_hf_token() {
20
+ # check if HF_TOKEN is available and valid
21
+ if [[ -z "$HF_TOKEN" ]]; then
22
+ echo "Error: HF_TOKEN is not set."
23
+ exit 1
24
+ elif [[ ! "$HF_TOKEN" =~ ^hf_ ]]; then
25
+ echo "Error: HF_TOKEN does not start with 'hf_'."
26
+ exit 1
27
+ else
28
+ echo "HF_TOKEN is set and valid."
29
+ fi
30
+}
31
+
32
+main() {
33
+
34
+ check_gpus
35
+ check_hf_token
36
+
37
+ df -h
38
+
39
+ (which wget && which curl) || (apt-get update && apt-get install -y wget curl)
40
+ (which jq) || (apt-get update && apt-get -y install jq)
41
+
42
+ cd $VLLM_SOURCE_CODE_LOC/benchmarks
43
+ wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
44
+
45
+
46
+ # run lmdeploy
47
+ if which lmdeploy >/dev/null; then
48
+ echo "lmdeploy is available, redirect to run-lmdeploy-nightly.sh"
49
+ bash ../.buildkite/nightly-benchmarks/scripts/run-lmdeploy-nightly.sh
50
+ exit 0
51
+ fi
52
+
53
+ # run tgi
54
+ if [ -e /tgi-entrypoint.sh ]; then
55
+ echo "tgi is available, redirect to run-tgi-nightly.sh"
56
+ bash ../.buildkite/nightly-benchmarks/scripts/run-tgi-nightly.sh
57
+ exit 0
58
+ fi
59
+
60
+ # run trt
61
+ if which trtllm-build >/dev/null; then
62
+ echo "trtllm is available, redirect to run-trt-nightly.sh"
63
+ bash ../.buildkite/nightly-benchmarks/scripts/run-trt-nightly.sh
64
+ exit 0
65
+ fi
66
+
67
+ # run vllm
68
+ if [ -e /vllm-workspace ]; then
69
+ echo "vllm is available, redirect to run-vllm-nightly.sh"
70
+ bash ../.buildkite/nightly-benchmarks/scripts/run-vllm-nightly.sh
71
+ exit 0
72
+ fi
73
+
74
+}
75
+
76
+main "$@"
\ No newline at end of file
.buildkite/nightly-benchmarks/scripts/download-tokenizer.py
new
+26
@@ -0,0 +1,26 @@
1
+import argparse
2
+
3
+from transformers import AutoTokenizer
4
+
5
+
6
+def main(model, cachedir):
7
+ # Load the tokenizer and save it to the specified directory
8
+ tokenizer = AutoTokenizer.from_pretrained(model)
9
+ tokenizer.save_pretrained(cachedir)
10
+ print(f"Tokenizer saved to {cachedir}")
11
+
12
+
13
+if __name__ == "__main__":
14
+ parser = argparse.ArgumentParser(
15
+ description="Download and save Hugging Face tokenizer")
16
+ parser.add_argument("--model",
17
+ type=str,
18
+ required=True,
19
+ help="Name of the model")
20
+ parser.add_argument("--cachedir",
21
+ type=str,
22
+ required=True,
23
+ help="Directory to save the tokenizer")
24
+
25
+ args = parser.parse_args()
26
+ main(args.model, args.cachedir)
.buildkite/nightly-benchmarks/scripts/get-lmdeploy-modelname.py
new
+6
@@ -0,0 +1,6 @@
1
+from lmdeploy.serve.openai.api_client import APIClient
2
+
3
+api_client = APIClient("http://localhost:8000")
4
+model_name = api_client.available_models[0]
5
+
6
+print(model_name)
.buildkite/nightly-benchmarks/scripts/launch-trt-server.sh
new
+102
@@ -0,0 +1,102 @@
1
+#!/bin/bash
2
+
3
+
4
+server_params=$1
5
+common_params=$2
6
+
7
+
8
+
9
+model_path=$(echo "$common_params" | jq -r '.model')
10
+model_name="${model_path#*/}"
11
+model_type=$(echo "$server_params" | jq -r '.model_type')
12
+model_dtype=$(echo "$server_params" | jq -r '.model_dtype')
13
+model_tp_size=$(echo "$common_params" | jq -r '.tp')
14
+max_batch_size=$(echo "$server_params" | jq -r '.max_batch_size')
15
+max_input_len=$(echo "$server_params" | jq -r '.max_input_len')
16
+max_output_len=$(echo "$server_params" | jq -r '.max_output_len')
17
+trt_llm_version=$(echo "$server_params" | jq -r '.trt_llm_version')
18
+
19
+cd ~
20
+rm -rf models
21
+mkdir -p models
22
+cd models
23
+models_dir=$(pwd)
24
+trt_model_path=${models_dir}/${model_name}-trt-ckpt
25
+trt_engine_path=${models_dir}/${model_name}-trt-engine
26
+
27
+cd ~
28
+rm -rf tensorrt-demo
29
+git clone https://github.com/neuralmagic/tensorrt-demo.git
30
+cd tensorrt-demo
31
+tensorrt_demo_dir=$(pwd)
32
+
33
+# make sure the parameter inside tensorrt_demo is consistent to envvar
34
+sed -i.bak "/key: \"tokenizer_dir\"/,/string_value:/s|string_value: \".*\"|string_value: \"$model_path\"|" ./triton_model_repo/postprocessing/config.pbtxt
35
+sed -i.bak "/key: \"tokenizer_dir\"/,/string_value:/s|string_value: \".*\"|string_value: \"$model_path\"|" ./triton_model_repo/preprocessing/config.pbtxt
36
+sed -i.bak "s|\(max_batch_size:\s*\)[0-9]*|\1$max_batch_size|g" ./triton_model_repo/ensemble/config.pbtxt
37
+sed -i.bak "s|\(max_batch_size:\s*\)[0-9]*|\1$max_batch_size|g" ./triton_model_repo/preprocessing/config.pbtxt
38
+sed -i.bak "s|\(max_batch_size:\s*\)[0-9]*|\1$max_batch_size|g" ./triton_model_repo/postprocessing/config.pbtxt
39
+sed -i.bak "s|\(max_batch_size:\s*\)[0-9]*|\1$max_batch_size|g" ./triton_model_repo/tensorrt_llm_bls/config.pbtxt
40
+
41
+
42
+cd /
43
+rm -rf tensorrtllm_backend
44
+git clone https://github.com/triton-inference-server/tensorrtllm_backend.git
45
+git lfs install
46
+cd tensorrtllm_backend
47
+git checkout $trt_llm_version
48
+tensorrtllm_backend_dir=$(pwd)
49
+git submodule update --init --recursive
50
+cp -r ${tensorrt_demo_dir}/triton_model_repo ${tensorrtllm_backend_dir}/
51
+
52
+cd /tensorrtllm_backend
53
+cd ./tensorrt_llm/examples/${model_type}
54
+
55
+
56
+if echo "$common_params" | jq -e 'has("fp8")' > /dev/null; then
57
+
58
+ echo "Key 'fp8' exists in common params. Use quantize.py instead of convert_checkpoint.py"
59
+ echo "Reference: https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/llama/README.md"
60
+ python ../quantization/quantize.py \
61
+ --model_dir ${model_path} \
62
+ --dtype ${model_dtype} \
63
+ --tp_size ${model_tp_size} \
64
+ --output_dir ${trt_model_path} \
65
+ --qformat fp8 \
66
+ --kv_cache_dtype fp8 \
67
+ --calib_size 2
68
+
69
+else
70
+
71
+ echo "Key 'fp8' does not exist in common params. Use convert_checkpoint.py"
72
+ python3 convert_checkpoint.py \
73
+ --model_dir ${model_path} \
74
+ --dtype ${model_dtype} \
75
+ --tp_size ${model_tp_size} \
76
+ --output_dir ${trt_model_path}
77
+
78
+fi
79
+
80
+
81
+
82
+trtllm-build \
83
+--checkpoint_dir=${trt_model_path} \
84
+--gpt_attention_plugin=${model_dtype} \
85
+--gemm_plugin=${model_dtype} \
86
+--remove_input_padding=enable \
87
+--paged_kv_cache=enable \
88
+--tp_size=${model_tp_size} \
89
+--max_batch_size=${max_batch_size} \
90
+--max_input_len=${max_input_len} \
91
+--max_output_len=${max_output_len} \
92
+--max_num_tokens=${max_output_len} \
93
+--opt_num_tokens=${max_output_len} \
94
+--output_dir=${trt_engine_path}
95
+
96
+cd /tensorrtllm_backend/triton_model_repo
97
+rm -rf ./tensorrt_llm/1/*
98
+cp -r ${trt_engine_path}/* ./tensorrt_llm/1
99
+cd /tensorrtllm_backend
100
+python3 scripts/launch_triton_server.py \
101
+--world_size=${model_tp_size} \
102
+--model_repo=/tensorrtllm_backend/triton_model_repo &
\ No newline at end of file
.buildkite/nightly-benchmarks/scripts/nightly-annotate.sh
new
+40
@@ -0,0 +1,40 @@
1
+#!/bin/bash
2
+
3
+set -ex
4
+set -o pipefail
5
+
6
+
7
+main() {
8
+
9
+ (which wget && which curl) || (apt-get update && apt-get install -y wget curl)
10
+ (which jq) || (apt-get update && apt-get -y install jq)
11
+
12
+ if [ ! -f /workspace/buildkite-agent ]; then
13
+ echo "buildkite-agent binary not found. Skip plotting the results."
14
+ exit 0
15
+ fi
16
+
17
+ # initial annotation
18
+ description="$VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/nightly-descriptions.md"
19
+
20
+ # download results
21
+ cd $VLLM_SOURCE_CODE_LOC/benchmarks
22
+ mkdir -p results/
23
+ /workspace/buildkite-agent artifact download 'results/*nightly_results.json' results/
24
+ ls
25
+ ls results/
26
+
27
+ # generate figures
28
+ python3 -m pip install tabulate pandas matplotlib
29
+ python3 $VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/scripts/plot-nightly-results.py \
30
+ --description $description \
31
+ --results-folder results/
32
+
33
+ # upload results and figures
34
+ /workspace/buildkite-agent artifact upload "nightly_results.png"
35
+ /workspace/buildkite-agent artifact upload $VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/nightly-pipeline.yaml
36
+ /workspace/buildkite-agent artifact upload $VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/tests/nightly-tests.json
37
+ /workspace/buildkite-agent annotate --style "success" --context "nightly-benchmarks-results" --append < nightly_results.md
38
+}
39
+
40
+main "$@"
\ No newline at end of file
.buildkite/nightly-benchmarks/scripts/plot-nightly-results.py
new
+135
@@ -0,0 +1,135 @@
1
+import argparse
2
+import json
3
+import math
4
+from pathlib import Path
5
+
6
+import matplotlib.pyplot as plt
7
+import pandas as pd
8
+from tabulate import tabulate
9
+
10
+
11
+def parse_arguments():
12
+ parser = argparse.ArgumentParser(
13
+ description=
14
+ 'Parse command line arguments for summary-nightly-results script.')
15
+ parser.add_argument('--results-folder',
16
+ type=str,
17
+ required=True,
18
+ help='The folder where the results are stored.')
19
+ parser.add_argument('--description',
20
+ type=str,
21
+ required=True,
22
+ help='Description of the results.')
23
+
24
+ args = parser.parse_args()
25
+ return args
26
+
27
+
28
+def main(args):
29
+ bar_colors = ['#56B4E9', '#009E73', '#D55E00', '#E69F00']
30
+ results_folder = Path(args.results_folder)
31
+
32
+ results = []
33
+
34
+ # collect results
35
+ for test_file in results_folder.glob("*_nightly_results.json"):
36
+ with open(test_file, "r") as f:
37
+ results = results + json.loads(f.read())
38
+
39
+ # generate markdown table
40
+ df = pd.DataFrame.from_dict(results)
41
+
42
+ md_table = tabulate(df, headers='keys', tablefmt='pipe', showindex=False)
43
+
44
+ with open(args.description, "r") as f:
45
+ description = f.read()
46
+
47
+ description = description.format(
48
+ nightly_results_benchmarking_table=md_table)
49
+
50
+ with open("nightly_results.md", "w") as f:
51
+ f.write(description)
52
+
53
+ plt.rcParams.update({'font.size': 20})
54
+
55
+ # plot results
56
+ fig, axes = plt.subplots(3, 3, figsize=(16, 14))
57
+ fig.subplots_adjust(hspace=1)
58
+ methods = ["vllm", "trt", "lmdeploy", "tgi"]
59
+ for i, model in enumerate(["llama8B", "llama70B", "mixtral8x7B"]):
60
+ for j, metric in enumerate(["TTFT", "ITL"]):
61
+ means, stds = [], []
62
+ for method in methods:
63
+ target = df['Test name'].str.contains(model)
64
+ target = target & df['Engine'].str.contains(method)
65
+ filtered_df = df[target]
66
+
67
+ if filtered_df.empty:
68
+ means.append(0.)
69
+ stds.append(0.)
70
+ else:
71
+ means.append(filtered_df[f"Mean {metric} (ms)"].values[0])
72
+ std = filtered_df[f"Std {metric} (ms)"].values[0]
73
+ success = filtered_df["Successful req."].values[0]
74
+ stds.append(std / math.sqrt(success))
75
+
76
+ print(model, metric)
77
+ print(means, stds)
78
+
79
+ ax = axes[i, j + 1]
80
+
81
+ bars = ax.bar(
82
+ ["vllm", "trt", "lmdeploy", "tgi"],
83
+ means,
84
+ yerr=stds,
85
+ capsize=10,
86
+ )
87
+ for idx, bar in enumerate(bars):
88
+ bar.set_color(bar_colors[idx])
89
+ ax.set_ylim(bottom=0)
90
+
91
+ ax.set_ylabel(f"{metric} (ms)")
92
+ ax.set_title(f"{model} {metric}")
93
+ ax.grid(axis='y')
94
+
95
+ metric = "Tput"
96
+ j = 0
97
+ if True:
98
+ tputs = []
99
+ for method in methods:
100
+ target = df['Test name'].str.contains(model)
101
+ target = target & df['Engine'].str.contains(method)
102
+ filtered_df = df[target]
103
+
104
+ if filtered_df.empty:
105
+ tputs.append(0.)
106
+ else:
107
+ input_tput = filtered_df["Input Tput (tok/s)"].values[0]
108
+ output_tput = filtered_df["Output Tput (tok/s)"].values[0]
109
+ tputs.append(input_tput + output_tput)
110
+
111
+ print(model, metric)
112
+ print(tputs)
113
+
114
+ ax = axes[i, j]
115
+
116
+ bars = ax.bar(
117
+ ["vllm", "trt", "lmdeploy", "tgi"],
118
+ tputs,
119
+ )
120
+ for idx, bar in enumerate(bars):
121
+ bar.set_color(bar_colors[idx])
122
+
123
+ ax.set_ylim(bottom=0)
124
+
125
+ ax.set_ylabel("Tput (token/s)")
126
+ ax.set_title(f"{model} {metric}")
127
+ ax.grid(axis='y')
128
+
129
+ fig.tight_layout()
130
+ fig.savefig("nightly_results.png", bbox_inches='tight', dpi=400)
131
+
132
+
133
+if __name__ == '__main__':
134
+ args = parse_arguments()
135
+ main(args)
.buildkite/nightly-benchmarks/scripts/run-lmdeploy-nightly.sh
new
+218
@@ -0,0 +1,218 @@
1
+#!/bin/bash
2
+
3
+set -o pipefail
4
+
5
+check_gpus() {
6
+ # check the number of GPUs and GPU type.
7
+ declare -g gpu_count=$(nvidia-smi --list-gpus | wc -l)
8
+ if [[ $gpu_count -gt 0 ]]; then
9
+ echo "GPU found."
10
+ else
11
+ echo "Need at least 1 GPU to run benchmarking."
12
+ exit 1
13
+ fi
14
+ declare -g gpu_type=$(echo $(nvidia-smi --query-gpu=name --format=csv,noheader) | awk '{print $2}')
15
+ echo "GPU type is $gpu_type"
16
+}
17
+
18
+kill_gpu_processes() {
19
+ pkill lmdeploy || true
20
+ # waiting for GPU processes to be fully killed
21
+ sleep 10
22
+ # Print the GPU memory usage
23
+ # so that we know if all GPU processes are killed.
24
+ gpu_memory_usage=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits -i 0)
25
+ # The memory usage should be 0 MB.
26
+ echo "GPU 0 Memory Usage: $gpu_memory_usage MB"
27
+}
28
+
29
+json2args() {
30
+ # transforms the JSON string to command line args, and '_' is replaced to '-'
31
+ # example:
32
+ # input: { "model": "meta-llama/Llama-2-7b-chat-hf", "tensor_parallel_size": 1 }
33
+ # output: --model meta-llama/Llama-2-7b-chat-hf --tensor-parallel-size 1
34
+ local json_string=$1
35
+ local args=$(
36
+ echo "$json_string" | jq -r '
37
+ to_entries |
38
+ map("--" + (.key | gsub("_"; "-")) + " " + (.value | tostring)) |
39
+ join(" ")
40
+ '
41
+ )
42
+ echo "$args"
43
+}
44
+
45
+wait_for_server() {
46
+ # wait for vllm server to start
47
+ # return 1 if vllm server crashes
48
+ timeout 1200 bash -c '
49
+ until curl -s localhost:8000/v1/completions > /dev/null; do
50
+ sleep 1
51
+ done' && return 0 || return 1
52
+}
53
+
54
+run_serving_tests() {
55
+ # run serving tests using `benchmark_serving.py`
56
+ # $1: a json file specifying serving test cases
57
+
58
+ local serving_test_file
59
+ serving_test_file=$1
60
+
61
+ # Iterate over serving tests
62
+ jq -c '.[]' "$serving_test_file" | while read -r params; do
63
+ # get the test name, and append the GPU type back to it.
64
+ test_name=$(echo "$params" | jq -r '.test_name')
65
+
66
+ # if TEST_SELECTOR is set, only run the test cases that match the selector
67
+ if [[ -n "$TEST_SELECTOR" ]] && [[ ! "$test_name" =~ $TEST_SELECTOR ]]; then
68
+ echo "Skip test case $test_name."
69
+ continue
70
+ fi
71
+
72
+ # append lmdeploy to the test name
73
+ test_name=lmdeploy_$test_name
74
+
75
+ # get common parameters
76
+ common_params=$(echo "$params" | jq -r '.common_parameters')
77
+ model=$(echo "$common_params" | jq -r '.model')
78
+ tp=$(echo "$common_params" | jq -r '.tp')
79
+ dataset_name=$(echo "$common_params" | jq -r '.dataset_name')
80
+ dataset_path=$(echo "$common_params" | jq -r '.dataset_path')
81
+ port=$(echo "$common_params" | jq -r '.port')
82
+ num_prompts=$(echo "$common_params" | jq -r '.num_prompts')
83
+
84
+
85
+
86
+ # get client and server arguments
87
+ server_params=$(echo "$params" | jq -r '.lmdeploy_server_parameters')
88
+ client_params=$(echo "$params" | jq -r '.lmdeploy_client_parameters')
89
+ server_args=$(json2args "$server_params")
90
+ client_args=$(json2args "$client_params")
91
+ qps_list=$(echo "$params" | jq -r '.qps_list')
92
+ qps_list=$(echo "$qps_list" | jq -r '.[] | @sh')
93
+ echo "Running over qps list $qps_list"
94
+
95
+ # check if there is enough GPU to run the test
96
+ if [[ $gpu_count -lt $tp ]]; then
97
+ echo "Required tensor-parallel-size $tp but only $gpu_count GPU found. Skip testcase $test_name."
98
+ continue
99
+ fi
100
+
101
+ # prepare tokenizer
102
+ rm -rf /tokenizer_cache
103
+ mkdir /tokenizer_cache
104
+ python ../.buildkite/nightly-benchmarks/scripts/download-tokenizer.py \
105
+ --model "$model" \
106
+ --cachedir /tokenizer_cache
107
+
108
+ server_command="lmdeploy serve api_server $model \
109
+ --tp $tp \
110
+ --server-port $port \
111
+ $server_args"
112
+
113
+ # run the server
114
+ echo "Running test case $test_name"
115
+ echo "Server command: $server_command"
116
+ bash -c "$server_command" &
117
+
118
+ # wait until the server is alive
119
+ wait_for_server
120
+ if [ $? -eq 0 ]; then
121
+ echo ""
122
+ echo "lmdeploy server is up and running."
123
+ else
124
+ echo ""
125
+ echo "lmdeploy failed to start within the timeout period."
126
+ break
127
+ fi
128
+
129
+ # get model name
130
+ model_name=$(python ../.buildkite/nightly-benchmarks/scripts/get-lmdeploy-modelname.py)
131
+
132
+ # iterate over different QPS
133
+ for qps in $qps_list; do
134
+ # remove the surrounding single quote from qps
135
+ if [[ "$qps" == *"inf"* ]]; then
136
+ echo "qps was $qps"
137
+ qps="inf"
138
+ echo "now qps is $qps"
139
+ fi
140
+
141
+ new_test_name=$test_name"_qps_"$qps
142
+
143
+ client_command="python3 benchmark_serving.py \
144
+ --backend lmdeploy \
145
+ --tokenizer /tokenizer_cache \
146
+ --dataset-name $dataset_name \
147
+ --dataset-path $dataset_path \
148
+ --num-prompts $num_prompts \
149
+ --port $port \
150
+ --save-result \
151
+ --result-dir $RESULTS_FOLDER \
152
+ --result-filename ${new_test_name}.json \
153
+ --request-rate $qps \
154
+ --model \"$model_name\" \
155
+ $client_args"
156
+
157
+ echo "Running test case $test_name with qps $qps"
158
+ echo "Client command: $client_command"
159
+
160
+ eval "$client_command"
161
+
162
+ # record the benchmarking commands
163
+ jq_output=$(jq -n \
164
+ --arg server "$server_command" \
165
+ --arg client "$client_command" \
166
+ --arg gpu "$gpu_type" \
167
+ --arg engine "lmdeploy" \
168
+ '{
169
+ server_command: $server,
170
+ client_command: $client,
171
+ gpu_type: $gpu,
172
+ engine: $engine
173
+ }')
174
+ echo "$jq_output" >"$RESULTS_FOLDER/${new_test_name}.commands"
175
+
176
+ done
177
+
178
+ # clean up
179
+ kill_gpu_processes
180
+ rm -rf /root/.cache/huggingface/*
181
+ done
182
+}
183
+
184
+
185
+upload_to_buildkite() {
186
+ # upload the benchmarking results to buildkite
187
+
188
+ # if the agent binary is not found, skip uploading the results, exit 0
189
+ if [ ! -f /workspace/buildkite-agent ]; then
190
+ echo "buildkite-agent binary not found. Skip uploading the results."
191
+ return 0
192
+ fi
193
+ # /workspace/buildkite-agent annotate --style "success" --context "benchmark-results" --append < $RESULTS_FOLDER/${CURRENT_LLM_SERVING_ENGINE}_nightly_results.md
194
+ /workspace/buildkite-agent artifact upload "$RESULTS_FOLDER/*"
195
+}
196
+
197
+
198
+main() {
199
+
200
+ check_gpus
201
+ # enter vllm directory
202
+ cd $VLLM_SOURCE_CODE_LOC/benchmarks
203
+
204
+ declare -g RESULTS_FOLDER=results/
205
+ mkdir -p $RESULTS_FOLDER
206
+ BENCHMARK_ROOT=../.buildkite/nightly-benchmarks/
207
+
208
+ python -m pip install transformers==4.41.2
209
+
210
+ export CURRENT_LLM_SERVING_ENGINE=lmdeploy
211
+ run_serving_tests $BENCHMARK_ROOT/tests/nightly-tests.json
212
+ python -m pip install tabulate pandas
213
+ python $BENCHMARK_ROOT/scripts/summary-nightly-results.py
214
+ upload_to_buildkite
215
+
216
+}
217
+
218
+main "$@"
.buildkite/nightly-benchmarks/scripts/run-tgi-nightly.sh
new
+216
@@ -0,0 +1,216 @@
1
+#!/bin/bash
2
+
3
+set -o pipefail
4
+
5
+check_gpus() {
6
+ # check the number of GPUs and GPU type.
7
+ declare -g gpu_count=$(nvidia-smi --list-gpus | wc -l)
8
+ if [[ $gpu_count -gt 0 ]]; then
9
+ echo "GPU found."
10
+ else
11
+ echo "Need at least 1 GPU to run benchmarking."
12
+ exit 1
13
+ fi
14
+ declare -g gpu_type=$(echo $(nvidia-smi --query-gpu=name --format=csv,noheader) | awk '{print $2}')
15
+ echo "GPU type is $gpu_type"
16
+}
17
+
18
+kill_gpu_processes() {
19
+ pkill text-generation || true
20
+ # waiting for GPU processes to be fully killed
21
+ sleep 10
22
+ # Print the GPU memory usage
23
+ # so that we know if all GPU processes are killed.
24
+ gpu_memory_usage=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits -i 0)
25
+ # The memory usage should be 0 MB.
26
+ echo "GPU 0 Memory Usage: $gpu_memory_usage MB"
27
+}
28
+
29
+json2args() {
30
+ # transforms the JSON string to command line args, and '_' is replaced to '-'
31
+ # example:
32
+ # input: { "model": "meta-llama/Llama-2-7b-chat-hf", "tensor_parallel_size": 1 }
33
+ # output: --model meta-llama/Llama-2-7b-chat-hf --tensor-parallel-size 1
34
+ local json_string=$1
35
+ local args=$(
36
+ echo "$json_string" | jq -r '
37
+ to_entries |
38
+ map("--" + (.key | gsub("_"; "-")) + " " + (.value | tostring)) |
39
+ join(" ")
40
+ '
41
+ )
42
+ echo "$args"
43
+}
44
+
45
+wait_for_server() {
46
+ timeout 1200 bash -c '
47
+ until curl -s localhost:8000/generate_stream > /dev/null; do
48
+ sleep 1
49
+ done' && return 0 || return 1
50
+}
51
+
52
+run_serving_tests() {
53
+ # run serving tests using `benchmark_serving.py`
54
+ # $1: a json file specifying serving test cases
55
+
56
+ local serving_test_file
57
+ serving_test_file=$1
58
+
59
+ # Iterate over serving tests
60
+ jq -c '.[]' "$serving_test_file" | while read -r params; do
61
+ # get the test name, and append the GPU type back to it.
62
+ test_name=$(echo "$params" | jq -r '.test_name')
63
+
64
+
65
+ # if TEST_SELECTOR is set, only run the test cases that match the selector
66
+ if [[ -n "$TEST_SELECTOR" ]] && [[ ! "$test_name" =~ $TEST_SELECTOR ]]; then
67
+ echo "Skip test case $test_name."
68
+ continue
69
+ fi
70
+
71
+ # append tgi to the test name
72
+ test_name=tgi_$test_name
73
+
74
+ # get common parameters
75
+ common_params=$(echo "$params" | jq -r '.common_parameters')
76
+ model=$(echo "$common_params" | jq -r '.model')
77
+ tp=$(echo "$common_params" | jq -r '.tp')
78
+ dataset_name=$(echo "$common_params" | jq -r '.dataset_name')
79
+ dataset_path=$(echo "$common_params" | jq -r '.dataset_path')
80
+ port=$(echo "$common_params" | jq -r '.port')
81
+ num_prompts=$(echo "$common_params" | jq -r '.num_prompts')
82
+
83
+ # get client and server arguments
84
+ server_params=$(echo "$params" | jq -r '.tgi_server_parameters')
85
+ client_params=$(echo "$params" | jq -r '.tgi_client_parameters')
86
+ server_args=$(json2args "$server_params")
87
+ client_args=$(json2args "$client_params")
88
+ qps_list=$(echo "$params" | jq -r '.qps_list')
89
+ qps_list=$(echo "$qps_list" | jq -r '.[] | @sh')
90
+ echo "Running over qps list $qps_list"
91
+
92
+ # check if there is enough GPU to run the test
93
+ if [[ $gpu_count -lt $tp ]]; then
94
+ echo "Required num-shard $tp but only $gpu_count GPU found. Skip testcase $test_name."
95
+ continue
96
+ fi
97
+
98
+ if echo "$common_params" | jq -e 'has("fp8")' > /dev/null; then
99
+ echo "Key 'fp8' exists in common params."
100
+ server_command="/tgi-entrypoint.sh \
101
+ --model-id $model \
102
+ --num-shard $tp \
103
+ --port $port \
104
+ --quantize fp8 \
105
+ $server_args"
106
+ else
107
+ echo "Key 'fp8' does not exist in common params."
108
+ server_command="/tgi-entrypoint.sh \
109
+ --model-id $model \
110
+ --num-shard $tp \
111
+ --port $port \
112
+ $server_args"
113
+ fi
114
+
115
+
116
+
117
+
118
+ # run the server
119
+ echo "Running test case $test_name"
120
+ echo "Server command: $server_command"
121
+ eval "$server_command" &
122
+
123
+ # wait until the server is alive
124
+ wait_for_server
125
+ if [ $? -eq 0 ]; then
126
+ echo ""
127
+ echo "tgi server is up and running."
128
+ else
129
+ echo ""
130
+ echo "tgi failed to start within the timeout period."
131
+ break
132
+ fi
133
+
134
+ # iterate over different QPS
135
+ for qps in $qps_list; do
136
+ # remove the surrounding single quote from qps
137
+ if [[ "$qps" == *"inf"* ]]; then
138
+ echo "qps was $qps"
139
+ qps="inf"
140
+ echo "now qps is $qps"
141
+ fi
142
+
143
+ new_test_name=$test_name"_qps_"$qps
144
+
145
+ client_command="python3 benchmark_serving.py \
146
+ --backend tgi \
147
+ --model $model \
148
+ --dataset-name $dataset_name \
149
+ --dataset-path $dataset_path \
150
+ --num-prompts $num_prompts \
151
+ --port $port \
152
+ --save-result \
153
+ --result-dir $RESULTS_FOLDER \
154
+ --result-filename ${new_test_name}.json \
155
+ --request-rate $qps \
156
+ $client_args"
157
+
158
+ echo "Running test case $test_name with qps $qps"
159
+ echo "Client command: $client_command"
160
+
161
+ eval "$client_command"
162
+
163
+ # record the benchmarking commands
164
+ jq_output=$(jq -n \
165
+ --arg server "$server_command" \
166
+ --arg client "$client_command" \
167
+ --arg gpu "$gpu_type" \
168
+ --arg engine "tgi" \
169
+ '{
170
+ server_command: $server,
171
+ client_command: $client,
172
+ gpu_type: $gpu,
173
+ engine: $engine
174
+ }')
175
+ echo "$jq_output" >"$RESULTS_FOLDER/${new_test_name}.commands"
176
+
177
+ done
178
+
179
+ # clean up
180
+ kill_gpu_processes
181
+ rm -rf /root/.cache/huggingface/*
182
+ done
183
+}
184
+
185
+
186
+
187
+upload_to_buildkite() {
188
+ # upload the benchmarking results to buildkite
189
+
190
+ # if the agent binary is not found, skip uploading the results, exit 0
191
+ if [ ! -f /workspace/buildkite-agent ]; then
192
+ echo "buildkite-agent binary not found. Skip uploading the results."
193
+ return 0
194
+ fi
195
+ # /workspace/buildkite-agent annotate --style "success" --context "benchmark-results" --append < $RESULTS_FOLDER/${CURRENT_LLM_SERVING_ENGINE}_nightly_results.md
196
+ /workspace/buildkite-agent artifact upload "$RESULTS_FOLDER/*"
197
+}
198
+
199
+main() {
200
+
201
+ check_gpus
202
+ # enter vllm directory
203
+ cd $VLLM_SOURCE_CODE_LOC/benchmarks
204
+ declare -g RESULTS_FOLDER=results/
205
+ mkdir -p $RESULTS_FOLDER
206
+ BENCHMARK_ROOT=../.buildkite/nightly-benchmarks/
207
+
208
+ export CURRENT_LLM_SERVING_ENGINE=tgi
209
+ run_serving_tests $BENCHMARK_ROOT/tests/nightly-tests.json
210
+ python -m pip install tabulate pandas
211
+ python $BENCHMARK_ROOT/scripts/summary-nightly-results.py
212
+ upload_to_buildkite
213
+
214
+}
215
+
216
+main "$@"
.buildkite/nightly-benchmarks/scripts/run-trt-nightly.sh
new
+214
@@ -0,0 +1,214 @@
1
+#!/bin/bash
2
+
3
+set -o pipefail
4
+
5
+check_gpus() {
6
+ # check the number of GPUs and GPU type.
7
+ declare -g gpu_count=$(nvidia-smi --list-gpus | wc -l)
8
+ if [[ $gpu_count -gt 0 ]]; then
9
+ echo "GPU found."
10
+ else
11
+ echo "Need at least 1 GPU to run benchmarking."
12
+ exit 1
13
+ fi
14
+ declare -g gpu_type=$(echo $(nvidia-smi --query-gpu=name --format=csv,noheader) | awk '{print $2}')
15
+ echo "GPU type is $gpu_type"
16
+}
17
+
18
+kill_gpu_processes() {
19
+ pkill tritonserver || true
20
+ # waiting for GPU processes to be fully killed
21
+ sleep 20
22
+ # Print the GPU memory usage
23
+ # so that we know if all GPU processes are killed.
24
+ gpu_memory_usage=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits -i 0)
25
+ # The memory usage should be 0 MB.
26
+ echo "GPU 0 Memory Usage: $gpu_memory_usage MB"
27
+}
28
+
29
+json2args() {
30
+ # transforms the JSON string to command line args, and '_' is replaced to '-'
31
+ # example:
32
+ # input: { "model": "meta-llama/Llama-2-7b-chat-hf", "tensor_parallel_size": 1 }
33
+ # output: --model meta-llama/Llama-2-7b-chat-hf --tensor-parallel-size 1
34
+ local json_string=$1
35
+ local args=$(
36
+ echo "$json_string" | jq -r '
37
+ to_entries |
38
+ map("--" + (.key | gsub("_"; "-")) + " " + (.value | tostring)) |
39
+ join(" ")
40
+ '
41
+ )
42
+ echo "$args"
43
+}
44
+
45
+wait_for_server() {
46
+ timeout 1200 bash -c '
47
+ until curl -s localhost:8000/generate_stream > /dev/null; do
48
+ sleep 1
49
+ done' && return 0 || return 1
50
+}
51
+
52
+run_serving_tests() {
53
+ # run serving tests using `benchmark_serving.py`
54
+ # $1: a json file specifying serving test cases
55
+
56
+ local serving_test_file
57
+ serving_test_file=$1
58
+
59
+ # Iterate over serving tests
60
+ jq -c '.[]' "$serving_test_file" | while read -r params; do
61
+ # get the test name, and append the GPU type back to it.
62
+ test_name=$(echo "$params" | jq -r '.test_name')
63
+
64
+ # if TEST_SELECTOR is set, only run the test cases that match the selector
65
+ if [[ -n "$TEST_SELECTOR" ]] && [[ ! "$test_name" =~ $TEST_SELECTOR ]]; then
66
+ echo "Skip test case $test_name."
67
+ continue
68
+ fi
69
+
70
+ # append trt to the test name
71
+ test_name=trt_$test_name
72
+
73
+ # get common parameters
74
+ common_params=$(echo "$params" | jq -r '.common_parameters')
75
+ model=$(echo "$common_params" | jq -r '.model')
76
+ tp=$(echo "$common_params" | jq -r '.tp')
77
+ dataset_name=$(echo "$common_params" | jq -r '.dataset_name')
78
+ dataset_path=$(echo "$common_params" | jq -r '.dataset_path')
79
+ port=$(echo "$common_params" | jq -r '.port')
80
+ num_prompts=$(echo "$common_params" | jq -r '.num_prompts')
81
+
82
+ # get client and server arguments
83
+ server_params=$(echo "$params" | jq -r '.trt_server_parameters')
84
+ client_params=$(echo "$params" | jq -r '.trt_client_parameters')
85
+ client_args=$(json2args "$client_params")
86
+ qps_list=$(echo "$params" | jq -r '.qps_list')
87
+ qps_list=$(echo "$qps_list" | jq -r '.[] | @sh')
88
+ echo "Running over qps list $qps_list"
89
+
90
+ # check if there is enough GPU to run the test
91
+ if [[ $gpu_count -lt $tp ]]; then
92
+ echo "Required model_tp_size $tp but only $gpu_count GPU found. Skip testcase $test_name."
93
+ continue
94
+ fi
95
+
96
+
97
+
98
+ cd $VLLM_SOURCE_CODE_LOC/benchmarks
99
+
100
+
101
+ echo "Running test case $test_name"
102
+ bash ../.buildkite/nightly-benchmarks/scripts/launch-trt-server.sh "$server_params" "$common_params"
103
+
104
+ # wait until the server is alive
105
+ wait_for_server
106
+ if [ $? -eq 0 ]; then
107
+ echo ""
108
+ echo "trt server is up and running."
109
+ else
110
+ echo ""
111
+ echo "trt failed to start within the timeout period."
112
+ break
113
+ fi
114
+
115
+ # prepare tokenizer
116
+ cd $VLLM_SOURCE_CODE_LOC/benchmarks
117
+ rm -rf /tokenizer_cache
118
+ mkdir /tokenizer_cache
119
+ python ../.buildkite/nightly-benchmarks/scripts/download-tokenizer.py \
120
+ --model "$model" \
121
+ --cachedir /tokenizer_cache
122
+ cd $VLLM_SOURCE_CODE_LOC/benchmarks
123
+
124
+
125
+ # iterate over different QPS
126
+ for qps in $qps_list; do
127
+ # remove the surrounding single quote from qps
128
+ if [[ "$qps" == *"inf"* ]]; then
129
+ echo "qps was $qps"
130
+ qps="inf"
131
+ echo "now qps is $qps"
132
+ fi
133
+
134
+ new_test_name=$test_name"_qps_"$qps
135
+
136
+ client_command="python3 benchmark_serving.py \
137
+ --backend tensorrt-llm \
138
+ --tokenizer /tokenizer_cache \
139
+ --model $model \
140
+ --dataset-name $dataset_name \
141
+ --dataset-path $dataset_path \
142
+ --num-prompts $num_prompts \
143
+ --port $port \
144
+ --save-result \
145
+ --result-dir $RESULTS_FOLDER \
146
+ --result-filename ${new_test_name}.json \
147
+ --request-rate $qps \
148
+ $client_args"
149
+
150
+ echo "Running test case $test_name with qps $qps"
151
+ echo "Client command: $client_command"
152
+
153
+ eval "$client_command"
154
+
155
+ server_command=""
156
+ # record the benchmarking commands
157
+ jq_output=$(jq -n \
158
+ --arg server "$server_command" \
159
+ --arg client "$client_command" \
160
+ --arg gpu "$gpu_type" \
161
+ --arg engine "trt" \
162
+ '{
163
+ server_command: $server,
164
+ client_command: $client,
165
+ gpu_type: $gpu,
166
+ engine: $engine
167
+ }')
168
+ echo "$jq_output" >"$RESULTS_FOLDER/${new_test_name}.commands"
169
+
170
+ done
171
+
172
+ # clean up
173
+ kill_gpu_processes
174
+ rm -rf /root/.cache/huggingface/*
175
+ done
176
+}
177
+
178
+upload_to_buildkite() {
179
+ # upload the benchmarking results to buildkite
180
+
181
+ # if the agent binary is not found, skip uploading the results, exit 0
182
+ if [ ! -f /workspace/buildkite-agent ]; then
183
+ echo "buildkite-agent binary not found. Skip uploading the results."
184
+ return 0
185
+ fi
186
+ # /workspace/buildkite-agent annotate --style "success" --context "benchmark-results" --append < $RESULTS_FOLDER/${CURRENT_LLM_SERVING_ENGINE}_nightly_results.md
187
+ /workspace/buildkite-agent artifact upload "$RESULTS_FOLDER/*"
188
+}
189
+
190
+
191
+main() {
192
+
193
+ check_gpus
194
+
195
+
196
+ # enter vllm directory
197
+ cd $VLLM_SOURCE_CODE_LOC/benchmarks
198
+
199
+ declare -g RESULTS_FOLDER=results/
200
+ mkdir -p $RESULTS_FOLDER
201
+ BENCHMARK_ROOT=../.buildkite/nightly-benchmarks/
202
+
203
+ # update transformers package, to make sure mixtral tokenizer is available
204
+ python -m pip install transformers -U
205
+
206
+ export CURRENT_LLM_SERVING_ENGINE=trt
207
+ run_serving_tests $BENCHMARK_ROOT/tests/nightly-tests.json
208
+ python -m pip install tabulate pandas
209
+ python $BENCHMARK_ROOT/scripts/summary-nightly-results.py
210
+ upload_to_buildkite
211
+
212
+}
213
+
214
+main "$@"
.buildkite/nightly-benchmarks/scripts/run-vllm-nightly.sh
new
+221
@@ -0,0 +1,221 @@
1
+#!/bin/bash
2
+
3
+set -o pipefail
4
+
5
+check_gpus() {
6
+ # check the number of GPUs and GPU type.
7
+ declare -g gpu_count=$(nvidia-smi --list-gpus | wc -l)
8
+ if [[ $gpu_count -gt 0 ]]; then
9
+ echo "GPU found."
10
+ else
11
+ echo "Need at least 1 GPU to run benchmarking."
12
+ exit 1
13
+ fi
14
+ declare -g gpu_type=$(echo $(nvidia-smi --query-gpu=name --format=csv,noheader) | awk '{print $2}')
15
+ echo "GPU type is $gpu_type"
16
+}
17
+
18
+kill_gpu_processes() {
19
+ # kill all processes on GPU.
20
+ pkill pt_main_thread
21
+ sleep 10
22
+
23
+ # remove vllm config file
24
+ rm -rf ~/.config/vllm
25
+
26
+ # Print the GPU memory usage
27
+ # so that we know if all GPU processes are killed.
28
+ gpu_memory_usage=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits -i 0)
29
+ # The memory usage should be 0 MB.
30
+ echo "GPU 0 Memory Usage: $gpu_memory_usage MB"
31
+}
32
+
33
+json2args() {
34
+ # transforms the JSON string to command line args, and '_' is replaced to '-'
35
+ # example:
36
+ # input: { "model": "meta-llama/Llama-2-7b-chat-hf", "tensor_parallel_size": 1 }
37
+ # output: --model meta-llama/Llama-2-7b-chat-hf --tensor-parallel-size 1
38
+ local json_string=$1
39
+ local args=$(
40
+ echo "$json_string" | jq -r '
41
+ to_entries |
42
+ map("--" + (.key | gsub("_"; "-")) + " " + (.value | tostring)) |
43
+ join(" ")
44
+ '
45
+ )
46
+ echo "$args"
47
+}
48
+
49
+wait_for_server() {
50
+ # wait for vllm server to start
51
+ # return 1 if vllm server crashes
52
+ timeout 1200 bash -c '
53
+ until curl -s localhost:8000/v1/completions > /dev/null; do
54
+ sleep 1
55
+ done' && return 0 || return 1
56
+}
57
+
58
+run_serving_tests() {
59
+ # run serving tests using `benchmark_serving.py`
60
+ # $1: a json file specifying serving test cases
61
+
62
+ local serving_test_file
63
+ serving_test_file=$1
64
+
65
+ # Iterate over serving tests
66
+ jq -c '.[]' "$serving_test_file" | while read -r params; do
67
+ # get the test name, and append the GPU type back to it.
68
+ test_name=$(echo "$params" | jq -r '.test_name')
69
+
70
+ # if TEST_SELECTOR is set, only run the test cases that match the selector
71
+ if [[ -n "$TEST_SELECTOR" ]] && [[ ! "$test_name" =~ $TEST_SELECTOR ]]; then
72
+ echo "Skip test case $test_name."
73
+ continue
74
+ fi
75
+
76
+ # append vllm to the test name
77
+ test_name=vllm_$test_name
78
+
79
+
80
+ # get common parameters
81
+ common_params=$(echo "$params" | jq -r '.common_parameters')
82
+ model=$(echo "$common_params" | jq -r '.model')
83
+ tp=$(echo "$common_params" | jq -r '.tp')
84
+ dataset_name=$(echo "$common_params" | jq -r '.dataset_name')
85
+ dataset_path=$(echo "$common_params" | jq -r '.dataset_path')
86
+ port=$(echo "$common_params" | jq -r '.port')
87
+ num_prompts=$(echo "$common_params" | jq -r '.num_prompts')
88
+
89
+ # get client and server arguments
90
+ server_params=$(echo "$params" | jq -r '.vllm_server_parameters')
91
+ client_params=$(echo "$params" | jq -r '.vllm_client_parameters')
92
+ server_args=$(json2args "$server_params")
93
+ client_args=$(json2args "$client_params")
94
+ qps_list=$(echo "$params" | jq -r '.qps_list')
95
+ qps_list=$(echo "$qps_list" | jq -r '.[] | @sh')
96
+ echo "Running over qps list $qps_list"
97
+
98
+ # check if there is enough GPU to run the test
99
+ if [[ $gpu_count -lt $tp ]]; then
100
+ echo "Required tensor-parallel-size $tp but only $gpu_count GPU found. Skip testcase $test_name."
101
+ continue
102
+ fi
103
+
104
+ if echo "$common_params" | jq -e 'has("fp8")' > /dev/null; then
105
+ echo "Key 'fp8' exists in common params. Use neuralmagic fp8 model for convenience."
106
+ model=$(echo "$common_params" | jq -r '.neuralmagic_quantized_model')
107
+ server_command="python3 \
108
+ -m vllm.entrypoints.openai.api_server \
109
+ -tp $tp \
110
+ --model $model \
111
+ --port $port \
112
+ $server_args"
113
+ else
114
+ echo "Key 'fp8' does not exist in common params."
115
+ server_command="python3 \
116
+ -m vllm.entrypoints.openai.api_server \
117
+ -tp $tp \
118
+ --model $model \
119
+ --port $port \
120
+ $server_args"
121
+ fi
122
+
123
+ # run the server
124
+ echo "Running test case $test_name"
125
+ echo "Server command: $server_command"
126
+ eval "$server_command" &
127
+
128
+ # wait until the server is alive
129
+ wait_for_server
130
+ if [ $? -eq 0 ]; then
131
+ echo ""
132
+ echo "vllm server is up and running."
133
+ else
134
+ echo ""
135
+ echo "vllm failed to start within the timeout period."
136
+ break
137
+ fi
138
+
139
+ # iterate over different QPS
140
+ for qps in $qps_list; do
141
+ # remove the surrounding single quote from qps
142
+ if [[ "$qps" == *"inf"* ]]; then
143
+ echo "qps was $qps"
144
+ qps="inf"
145
+ echo "now qps is $qps"
146
+ fi
147
+
148
+ new_test_name=$test_name"_qps_"$qps
149
+
150
+ client_command="python3 benchmark_serving.py \
151
+ --backend vllm \
152
+ --model $model \
153
+ --dataset-name $dataset_name \
154
+ --dataset-path $dataset_path \
155
+ --num-prompts $num_prompts \
156
+ --port $port \
157
+ --save-result \
158
+ --result-dir $RESULTS_FOLDER \
159
+ --result-filename ${new_test_name}.json \
160
+ --request-rate $qps \
161
+ $client_args"
162
+
163
+ echo "Running test case $test_name with qps $qps"
164
+ echo "Client command: $client_command"
165
+
166
+ eval "$client_command"
167
+
168
+ # record the benchmarking commands
169
+ jq_output=$(jq -n \
170
+ --arg server "$server_command" \
171
+ --arg client "$client_command" \
172
+ --arg gpu "$gpu_type" \
173
+ --arg engine "vllm" \
174
+ '{
175
+ server_command: $server,
176
+ client_command: $client,
177
+ gpu_type: $gpu,
178
+ engine: $engine
179
+ }')
180
+ echo "$jq_output" >"$RESULTS_FOLDER/${new_test_name}.commands"
181
+
182
+ done
183
+
184
+ # clean up
185
+ kill_gpu_processes
186
+ rm -rf /root/.cache/huggingface/*
187
+ done
188
+}
189
+
190
+
191
+upload_to_buildkite() {
192
+ # upload the benchmarking results to buildkite
193
+
194
+ # if the agent binary is not found, skip uploading the results, exit 0
195
+ if [ ! -f /workspace/buildkite-agent ]; then
196
+ echo "buildkite-agent binary not found. Skip uploading the results."
197
+ return 0
198
+ fi
199
+ # /workspace/buildkite-agent annotate --style "success" --context "benchmark-results" --append < $RESULTS_FOLDER/${CURRENT_LLM_SERVING_ENGINE}_nightly_results.md
200
+ /workspace/buildkite-agent artifact upload "$RESULTS_FOLDER/*"
201
+}
202
+
203
+main() {
204
+
205
+ check_gpus
206
+ # enter vllm directory
207
+ cd $VLLM_SOURCE_CODE_LOC/benchmarks
208
+ declare -g RESULTS_FOLDER=results/
209
+ mkdir -p $RESULTS_FOLDER
210
+ BENCHMARK_ROOT=../.buildkite/nightly-benchmarks/
211
+
212
+ export CURRENT_LLM_SERVING_ENGINE=vllm
213
+ run_serving_tests $BENCHMARK_ROOT/tests/nightly-tests.json
214
+
215
+ python3 -m pip install tabulate pandas
216
+ python3 $BENCHMARK_ROOT/scripts/summary-nightly-results.py
217
+ upload_to_buildkite
218
+
219
+}
220
+
221
+main "$@"
.buildkite/nightly-benchmarks/scripts/summary-nightly-results.py
new
+76
@@ -0,0 +1,76 @@
1
+import datetime
2
+import json
3
+import os
4
+from pathlib import Path
5
+
6
+import pandas as pd
7
+from tabulate import tabulate
8
+
9
+results_folder = Path("results/")
10
+
11
+# serving results and the keys that will be printed into markdown
12
+serving_results = []
13
+serving_column_mapping = {
14
+ "test_name": "Test name",
15
+ "gpu_type": "GPU",
16
+ "completed": "Successful req.",
17
+ "request_throughput": "Tput (req/s)",
18
+ "mean_ttft_ms": "Mean TTFT (ms)",
19
+ "std_ttft_ms": "Std TTFT (ms)",
20
+ "mean_itl_ms": "Mean ITL (ms)",
21
+ "std_itl_ms": "Std ITL (ms)",
22
+ "input_throughput": "Input Tput (tok/s)",
23
+ "output_throughput": "Output Tput (tok/s)",
24
+ "engine": "Engine",
25
+}
26
+
27
+if __name__ == "__main__":
28
+
29
+ # collect results
30
+ for test_file in results_folder.glob("*.json"):
31
+
32
+ with open(test_file, "r") as f:
33
+ raw_result = json.loads(f.read())
34
+
35
+ # attach the benchmarking command to raw_result
36
+ with open(test_file.with_suffix(".commands"), "r") as f:
37
+ command = json.loads(f.read())
38
+ raw_result.update(command)
39
+
40
+ # update the test name of this result
41
+ raw_result.update({"test_name": test_file.stem})
42
+
43
+ # add the result to raw_result
44
+ serving_results.append(raw_result)
45
+ continue
46
+
47
+ serving_results = pd.DataFrame.from_dict(serving_results)
48
+
49
+ if not serving_results.empty:
50
+ serving_results = serving_results[list(
51
+ serving_column_mapping.keys())].rename(
52
+ columns=serving_column_mapping)
53
+
54
+ serving_md_table_with_headers = tabulate(serving_results,
55
+ headers='keys',
56
+ tablefmt='pipe',
57
+ showindex=False)
58
+ # remove the first line of header
59
+ serving_md_table_lines = serving_md_table_with_headers.split('\n')
60
+ serving_md_table_without_header = '\n'.join(serving_md_table_lines[2:])
61
+
62
+ prefix = datetime.datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
63
+ prefix = prefix + "_" + os.environ.get("CURRENT_LLM_SERVING_ENGINE")
64
+
65
+ # document benchmarking results in markdown
66
+ with open(results_folder / f"{prefix}_nightly_results.md", "w") as f:
67
+ # document results with header.
68
+ # for those who wants to reproduce our benchmark.
69
+ f.write(serving_md_table_with_headers)
70
+ f.write('\n')
71
+
72
+ # document benchmarking results in json
73
+ with open(results_folder / f"{prefix}_nightly_results.json", "w") as f:
74
+
75
+ results = serving_results.to_dict(orient='records')
76
+ f.write(json.dumps(results))
.buildkite/nightly-benchmarks/tests/nightly-tests.json
new
+116
@@ -0,0 +1,116 @@
1
+[
2
+ {
3
+ "test_name": "llama8B_tp1",
4
+ "qps_list": [4],
5
+ "common_parameters": {
6
+ "model": "meta-llama/Meta-Llama-3-8B",
7
+ "tp": 1,
8
+ "dataset_name": "sharegpt",
9
+ "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
10
+ "num_prompts": 500,
11
+ "port": 8000
12
+ },
13
+ "lmdeploy_server_parameters": {
14
+ },
15
+ "lmdeploy_client_parameters": {
16
+ },
17
+ "tgi_server_parameters": {
18
+ },
19
+ "tgi_client_parameters": {
20
+ "endpoint": "/generate_stream"
21
+ },
22
+ "trt_server_parameters": {
23
+ "model_type": "llama",
24
+ "model_dtype": "float16",
25
+ "max_batch_size": 256,
26
+ "max_input_len": 4096,
27
+ "max_output_len": 4096,
28
+ "trt_llm_version": "r24.04"
29
+ },
30
+ "trt_client_parameters": {
31
+ "endpoint": "/v2/models/ensemble/generate_stream"
32
+ },
33
+ "vllm_server_parameters": {
34
+ "disable_log_stats": "",
35
+ "disable_log_requests": ""
36
+ },
37
+ "vllm_client_parameters": {
38
+ }
39
+ },
40
+ {
41
+ "test_name": "llama70B_tp4",
42
+ "qps_list": [2],
43
+ "common_parameters": {
44
+ "model": "meta-llama/Meta-Llama-3-70B-Instruct",
45
+ "tp": 4,
46
+ "dataset_name": "sharegpt",
47
+ "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
48
+ "num_prompts": 500,
49
+ "port": 8000
50
+ },
51
+ "lmdeploy_server_parameters": {
52
+ },
53
+ "lmdeploy_client_parameters": {
54
+ },
55
+ "tgi_server_parameters": {
56
+ },
57
+ "tgi_client_parameters": {
58
+ "endpoint": "/generate_stream"
59
+ },
60
+ "trt_server_parameters": {
61
+ "model_type": "llama",
62
+ "model_dtype": "float16",
63
+ "max_batch_size": 256,
64
+ "max_input_len": 4096,
65
+ "max_output_len": 4096,
66
+ "trt_llm_version": "r24.04"
67
+ },
68
+ "trt_client_parameters": {
69
+ "endpoint": "/v2/models/ensemble/generate_stream"
70
+ },
71
+ "vllm_server_parameters": {
72
+ "disable_log_stats": "",
73
+ "disable_log_requests": ""
74
+ },
75
+ "vllm_client_parameters": {
76
+ }
77
+ },
78
+ {
79
+ "test_name": "mixtral8x7B_tp2",
80
+ "qps_list": [2],
81
+ "common_parameters": {
82
+ "model": "mistralai/Mixtral-8x7B-Instruct-v0.1",
83
+ "tp": 2,
84
+ "dataset_name": "sharegpt",
85
+ "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
86
+ "num_prompts": 500,
87
+ "port": 8000
88
+ },
89
+ "lmdeploy_server_parameters": {
90
+ },
91
+ "lmdeploy_client_parameters": {
92
+ },
93
+ "tgi_server_parameters": {
94
+ },
95
+ "tgi_client_parameters": {
96
+ "endpoint": "/generate_stream"
97
+ },
98
+ "trt_server_parameters": {
99
+ "model_type": "llama",
100
+ "model_dtype": "float16",
101
+ "max_batch_size": 256,
102
+ "max_input_len": 4096,
103
+ "max_output_len": 4096,
104
+ "trt_llm_version": "r24.04"
105
+ },
106
+ "trt_client_parameters": {
107
+ "endpoint": "/v2/models/ensemble/generate_stream"
108
+ },
109
+ "vllm_server_parameters": {
110
+ "disable_log_stats": "",
111
+ "disable_log_requests": ""
112
+ },
113
+ "vllm_client_parameters": {
114
+ }
115
+ }
116
+]
\ No newline at end of file
README.md
+2
@@ -52,6 +52,8 @@ vLLM is fast with:
52
- Quantization: [GPTQ](https://arxiv.org/abs/2210.17323), [AWQ](https://arxiv.org/abs/2306.00978), [SqueezeLLM](https://arxiv.org/abs/2306.07629), FP8 KV Cache
53
- Optimized CUDA kernels
54
55
+**Performance benchmark**: We include a [performance benchmark](https://buildkite.com/vllm/performance-benchmark/builds/3924) that compares the performance of vllm against other LLM serving engines ([TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM), [text-generation-inference](https://github.com/huggingface/text-generation-inference) and [lmdeploy](https://github.com/InternLM/lmdeploy)).
56
+
57
vLLM is flexible and easy to use with:
58
59
- Seamless integration with popular Hugging Face models
benchmarks/benchmark_serving.py
+9
@@ -60,12 +60,15 @@ class BenchmarkMetrics:
60
output_throughput: float
61
mean_ttft_ms: float
62
median_ttft_ms: float
63
+ std_ttft_ms: float
64
p99_ttft_ms: float
65
mean_tpot_ms: float
66
median_tpot_ms: float
67
+ std_tpot_ms: float
68
p99_tpot_ms: float
69
mean_itl_ms: float
70
median_itl_ms: float
71
+ std_itl_ms: float
72
p99_itl_ms: float
73
74
@@ -274,12 +277,15 @@ def calculate_metrics(
277
mean_ttft_ms=np.mean(ttfts or 0) *
278
1000, # ttfts is empty if streaming is not supported by backend
279
median_ttft_ms=np.median(ttfts or 0) * 1000,
280
+ std_ttft_ms=np.std(ttfts or 0) * 1000,
281
p99_ttft_ms=np.percentile(ttfts or 0, 99) * 1000,
282
mean_tpot_ms=np.mean(tpots or 0) * 1000,
283
median_tpot_ms=np.median(tpots or 0) * 1000,
284
+ std_tpot_ms=np.std(tpots or 0) * 1000,
285
p99_tpot_ms=np.percentile(tpots or 0, 99) * 1000,
286
mean_itl_ms=np.mean(itls or 0) * 1000,
287
median_itl_ms=np.median(itls or 0) * 1000,
288
+ std_itl_ms=np.std(itls or 0) * 1000,
289
p99_itl_ms=np.percentile(itls or 0, 99) * 1000,
290
)
291
@@ -396,12 +402,15 @@ async def benchmark(
402
"output_throughput": metrics.output_throughput,
403
"mean_ttft_ms": metrics.mean_ttft_ms,
404
"median_ttft_ms": metrics.median_ttft_ms,
405
+ "std_ttft_ms": metrics.std_ttft_ms,
406
"p99_ttft_ms": metrics.p99_ttft_ms,
407
"mean_tpot_ms": metrics.mean_tpot_ms,
408
"median_tpot_ms": metrics.median_tpot_ms,
409
+ "std_tpot_ms": metrics.std_tpot_ms,
410
"p99_tpot_ms": metrics.p99_tpot_ms,
411
"mean_itl_ms": metrics.mean_itl_ms,
412
"median_itl_ms": metrics.median_itl_ms,
413
+ "std_itl_ms": metrics.std_itl_ms,
414
"p99_itl_ms": metrics.p99_itl_ms,
415
"input_lens": [output.prompt_len for output in outputs],
416
"output_lens": actual_output_lens,