@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / a4feba929b
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 22:00 UTC

[CI/Build] Add nightly benchmarking for tgi, tensorrt-llm and lmdeploy (#5362)

Kuntai Du committed Jul 11, 2024 at 13:28 UTC a4feba929b61f287ae0b7407c3d615c6dec193d6
18 files changed +1623 -27
.buildkite/nightly-benchmarks/README.md
+1
@@ -1,5 +1,6 @@
1 # vLLM benchmark suite
2
3 +
4 ## Introduction
5
6 This directory contains the performance benchmarking CI for vllm.
.buildkite/nightly-benchmarks/kickoff-pipeline.sh deleted
-27
@@ -1,27 +0,0 @@
1 -#!/usr/bin/env bash
2 -
3 -# NOTE(simon): this script runs inside a buildkite agent with CPU only access.
4 -set -euo pipefail
5 -
6 -# Install system packages
7 -apt update
8 -apt install -y curl jq
9 -
10 -# Install minijinja for templating
11 -curl -sSfL https://github.com/mitsuhiko/minijinja/releases/latest/download/minijinja-cli-installer.sh | sh
12 -source $HOME/.cargo/env
13 -
14 -# If BUILDKITE_PULL_REQUEST != "false", then we check the PR labels using curl and jq
15 -if [ "$BUILDKITE_PULL_REQUEST" != "false" ]; then
16 - PR_LABELS=$(curl -s "https://api.github.com/repos/vllm-project/vllm/pulls/$BUILDKITE_PULL_REQUEST" | jq -r '.labels[].name')
17 -
18 - if [[ $PR_LABELS == *"perf-benchmarks"* ]]; then
19 - echo "This PR has the 'perf-benchmarks' label. Proceeding with the nightly benchmarks."
20 - else
21 - echo "This PR does not have the 'perf-benchmarks' label. Skipping the nightly benchmarks."
22 - exit 0
23 - fi
24 -fi
25 -
26 -# Upload sample.yaml
27 -buildkite-agent pipeline upload .buildkite/nightly-benchmarks/benchmark-pipeline.yaml
.buildkite/nightly-benchmarks/nightly-descriptions.md new
+45
@@ -0,0 +1,45 @@
1 +
2 +# Nightly benchmark
3 +
4 +The main goal of this benchmarking is two-fold:
5 +- Performance clarity: Provide clarity on which one (vllm, tensorrt-llm, lmdeploy and tgi) leads in performance in what workload.
6 +- Reproducible: one can run the exact same set of benchmarking commands inside the exact same docker by following reproducing instructions in [reproduce.md]().
7 +
8 +
9 +## Docker images
10 +
11 +We benchmark vllm, tensorrt-llm, lmdeploy and tgi using the following docker images:
12 +- vllm/vllm-openai:v0.5.0.post1
13 +- nvcr.io/nvidia/tritonserver:24.04-trtllm-python-py3
14 +- openmmlab/lmdeploy:v0.5.0
15 +- ghcr.io/huggingface/text-generation-inference:2.1
16 +
17 +<!-- Please check <a href="artifact://workspace/build/buildkite/vllm/performance-benchmark/.buildkite/nightly-benchmarks/nightly-pipeline.yaml">nightly-pipeline.yaml</a> artifact for more details on how we deploy the docker images. -->
18 +
19 +
20 +## Hardware
21 +
22 +One AWS node with 8x NVIDIA A100 GPUs.
23 +
24 +
25 +## Workload description
26 +
27 +We benchmark vllm, tensorrt-llm, lmdeploy and tgi using the following workload:
28 +
29 +- Input length: randomly sample 500 prompts from ShareGPT dataset (with fixed random seed).
30 +- Output length: the corresponding output length of these 500 prompts.
31 +- Models: llama-3 8B, llama-3 70B, mixtral 8x7B.
32 +- Average QPS (query per second): 4 for the small model (llama-3 8B) and 2 for other two models. For each QPS, the arrival time of each query is determined using a random Poisson process (with fixed random seed).
33 +- Evaluation metrics: Throughput (higher the better), TTFT (time to the first token, lower the better), ITL (inter-token latency, lower the better).
34 +
35 +<!-- Check <a href="artifact://workspace/build/buildkite/vllm/performance-benchmark/.buildkite/nightly-benchmarks/tests/nightly-tests.json">nightly-tests.json</a> artifact for more details. -->
36 +
37 +## Plots
38 +
39 +In the following plots, the dot shows the mean and the error bar shows the standard error of the mean. Value 0 means that the corresponding benchmark crashed.
40 +
41 +<img src="artifact://nightly_results.png" alt="Benchmarking results" height=250 >
42 +
43 +## Results
44 +
45 +{nightly_results_benchmarking_table}
.buildkite/nightly-benchmarks/nightly-pipeline.yaml new
+120
@@ -0,0 +1,120 @@
1 +common_pod_spec: &common_pod_spec
2 + priorityClassName: perf-benchmark
3 + nodeSelector:
4 + nvidia.com/gpu.product: NVIDIA-A100-SXM4-80GB
5 + volumes:
6 + - name: devshm
7 + emptyDir:
8 + medium: Memory
9 + - name: hf-cache
10 + hostPath:
11 + path: /root/.cache/huggingface
12 + type: Directory
13 +
14 +common_container_settings: &common_container_settings
15 + command:
16 + - bash .buildkite/nightly-benchmarks/run-nightly-suite.sh
17 + resources:
18 + limits:
19 + nvidia.com/gpu: 8
20 + volumeMounts:
21 + - name: devshm
22 + mountPath: /dev/shm
23 + - name: hf-cache
24 + mountPath: /root/.cache/huggingface
25 + env:
26 + - name: VLLM_USAGE_SOURCE
27 + value: ci-test
28 + - name: HF_HOME
29 + value: /root/.cache/huggingface
30 + - name: VLLM_SOURCE_CODE_LOC
31 + value: /workspace/build/buildkite/vllm/performance-benchmark
32 + - name: HF_TOKEN
33 + valueFrom:
34 + secretKeyRef:
35 + name: hf-token-secret
36 + key: token
37 +
38 +steps:
39 + - block: ":rocket: Ready for comparing vllm against alternatives? This will take 4 hours."
40 + - label: "A100 trt benchmark"
41 + priority: 100
42 + agents:
43 + queue: A100
44 + plugins:
45 + - kubernetes:
46 + podSpec:
47 + <<: *common_pod_spec
48 + containers:
49 + - image: nvcr.io/nvidia/tritonserver:24.04-trtllm-python-py3
50 + <<: *common_container_settings
51 +
52 + - label: "A100 lmdeploy benchmark"
53 + priority: 100
54 + agents:
55 + queue: A100
56 + plugins:
57 + - kubernetes:
58 + podSpec:
59 + <<: *common_pod_spec
60 + containers:
61 + - image: openmmlab/lmdeploy:v0.5.0
62 + <<: *common_container_settings
63 +
64 +
65 + - label: "A100 vllm benchmark"
66 + priority: 100
67 + agents:
68 + queue: A100
69 + plugins:
70 + - kubernetes:
71 + podSpec:
72 + <<: *common_pod_spec
73 + containers:
74 + - image: vllm/vllm-openai:latest
75 + <<: *common_container_settings
76 +
77 + - label: "A100 tgi benchmark"
78 + priority: 100
79 + agents:
80 + queue: A100
81 + plugins:
82 + - kubernetes:
83 + podSpec:
84 + <<: *common_pod_spec
85 + containers:
86 + - image: ghcr.io/huggingface/text-generation-inference:2.1
87 + <<: *common_container_settings
88 +
89 + - wait
90 +
91 + - label: "Plot"
92 + priority: 100
93 + agents:
94 + queue: A100
95 + plugins:
96 + - kubernetes:
97 + podSpec:
98 + <<: *common_pod_spec
99 + containers:
100 + - image: vllm/vllm-openai:v0.5.0.post1
101 + command:
102 + - bash .buildkite/nightly-benchmarks/scripts/nightly-annotate.sh
103 + resources:
104 + limits:
105 + nvidia.com/gpu: 8
106 + volumeMounts:
107 + - name: devshm
108 + mountPath: /dev/shm
109 + env:
110 + - name: VLLM_USAGE_SOURCE
111 + value: ci-test
112 + - name: VLLM_SOURCE_CODE_LOC
113 + value: /workspace/build/buildkite/vllm/performance-benchmark
114 + - name: HF_TOKEN
115 + valueFrom:
116 + secretKeyRef:
117 + name: hf-token-secret
118 + key: token
119 +
120 + - wait
\ No newline at end of file
.buildkite/nightly-benchmarks/run-nightly-suite.sh new
+76
@@ -0,0 +1,76 @@
1 +#!/bin/bash
2 +
3 +set -o pipefail
4 +set -x
5 +
6 +check_gpus() {
7 + # check the number of GPUs and GPU type.
8 + declare -g gpu_count=$(nvidia-smi --list-gpus | wc -l)
9 + if [[ $gpu_count -gt 0 ]]; then
10 + echo "GPU found."
11 + else
12 + echo "Need at least 1 GPU to run benchmarking."
13 + exit 1
14 + fi
15 + declare -g gpu_type=$(echo $(nvidia-smi --query-gpu=name --format=csv,noheader) | awk '{print $2}')
16 + echo "GPU type is $gpu_type"
17 +}
18 +
19 +check_hf_token() {
20 + # check if HF_TOKEN is available and valid
21 + if [[ -z "$HF_TOKEN" ]]; then
22 + echo "Error: HF_TOKEN is not set."
23 + exit 1
24 + elif [[ ! "$HF_TOKEN" =~ ^hf_ ]]; then
25 + echo "Error: HF_TOKEN does not start with 'hf_'."
26 + exit 1
27 + else
28 + echo "HF_TOKEN is set and valid."
29 + fi
30 +}
31 +
32 +main() {
33 +
34 + check_gpus
35 + check_hf_token
36 +
37 + df -h
38 +
39 + (which wget && which curl) || (apt-get update && apt-get install -y wget curl)
40 + (which jq) || (apt-get update && apt-get -y install jq)
41 +
42 + cd $VLLM_SOURCE_CODE_LOC/benchmarks
43 + wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
44 +
45 +
46 + # run lmdeploy
47 + if which lmdeploy >/dev/null; then
48 + echo "lmdeploy is available, redirect to run-lmdeploy-nightly.sh"
49 + bash ../.buildkite/nightly-benchmarks/scripts/run-lmdeploy-nightly.sh
50 + exit 0
51 + fi
52 +
53 + # run tgi
54 + if [ -e /tgi-entrypoint.sh ]; then
55 + echo "tgi is available, redirect to run-tgi-nightly.sh"
56 + bash ../.buildkite/nightly-benchmarks/scripts/run-tgi-nightly.sh
57 + exit 0
58 + fi
59 +
60 + # run trt
61 + if which trtllm-build >/dev/null; then
62 + echo "trtllm is available, redirect to run-trt-nightly.sh"
63 + bash ../.buildkite/nightly-benchmarks/scripts/run-trt-nightly.sh
64 + exit 0
65 + fi
66 +
67 + # run vllm
68 + if [ -e /vllm-workspace ]; then
69 + echo "vllm is available, redirect to run-vllm-nightly.sh"
70 + bash ../.buildkite/nightly-benchmarks/scripts/run-vllm-nightly.sh
71 + exit 0
72 + fi
73 +
74 +}
75 +
76 +main "$@"
\ No newline at end of file
.buildkite/nightly-benchmarks/scripts/download-tokenizer.py new
+26
@@ -0,0 +1,26 @@
1 +import argparse
2 +
3 +from transformers import AutoTokenizer
4 +
5 +
6 +def main(model, cachedir):
7 + # Load the tokenizer and save it to the specified directory
8 + tokenizer = AutoTokenizer.from_pretrained(model)
9 + tokenizer.save_pretrained(cachedir)
10 + print(f"Tokenizer saved to {cachedir}")
11 +
12 +
13 +if __name__ == "__main__":
14 + parser = argparse.ArgumentParser(
15 + description="Download and save Hugging Face tokenizer")
16 + parser.add_argument("--model",
17 + type=str,
18 + required=True,
19 + help="Name of the model")
20 + parser.add_argument("--cachedir",
21 + type=str,
22 + required=True,
23 + help="Directory to save the tokenizer")
24 +
25 + args = parser.parse_args()
26 + main(args.model, args.cachedir)
.buildkite/nightly-benchmarks/scripts/get-lmdeploy-modelname.py new
+6
@@ -0,0 +1,6 @@
1 +from lmdeploy.serve.openai.api_client import APIClient
2 +
3 +api_client = APIClient("http://localhost:8000")
4 +model_name = api_client.available_models[0]
5 +
6 +print(model_name)
.buildkite/nightly-benchmarks/scripts/launch-trt-server.sh new
+102
@@ -0,0 +1,102 @@
1 +#!/bin/bash
2 +
3 +
4 +server_params=$1
5 +common_params=$2
6 +
7 +
8 +
9 +model_path=$(echo "$common_params" | jq -r '.model')
10 +model_name="${model_path#*/}"
11 +model_type=$(echo "$server_params" | jq -r '.model_type')
12 +model_dtype=$(echo "$server_params" | jq -r '.model_dtype')
13 +model_tp_size=$(echo "$common_params" | jq -r '.tp')
14 +max_batch_size=$(echo "$server_params" | jq -r '.max_batch_size')
15 +max_input_len=$(echo "$server_params" | jq -r '.max_input_len')
16 +max_output_len=$(echo "$server_params" | jq -r '.max_output_len')
17 +trt_llm_version=$(echo "$server_params" | jq -r '.trt_llm_version')
18 +
19 +cd ~
20 +rm -rf models
21 +mkdir -p models
22 +cd models
23 +models_dir=$(pwd)
24 +trt_model_path=${models_dir}/${model_name}-trt-ckpt
25 +trt_engine_path=${models_dir}/${model_name}-trt-engine
26 +
27 +cd ~
28 +rm -rf tensorrt-demo
29 +git clone https://github.com/neuralmagic/tensorrt-demo.git
30 +cd tensorrt-demo
31 +tensorrt_demo_dir=$(pwd)
32 +
33 +# make sure the parameter inside tensorrt_demo is consistent to envvar
34 +sed -i.bak "/key: \"tokenizer_dir\"/,/string_value:/s|string_value: \".*\"|string_value: \"$model_path\"|" ./triton_model_repo/postprocessing/config.pbtxt
35 +sed -i.bak "/key: \"tokenizer_dir\"/,/string_value:/s|string_value: \".*\"|string_value: \"$model_path\"|" ./triton_model_repo/preprocessing/config.pbtxt
36 +sed -i.bak "s|\(max_batch_size:\s*\)[0-9]*|\1$max_batch_size|g" ./triton_model_repo/ensemble/config.pbtxt
37 +sed -i.bak "s|\(max_batch_size:\s*\)[0-9]*|\1$max_batch_size|g" ./triton_model_repo/preprocessing/config.pbtxt
38 +sed -i.bak "s|\(max_batch_size:\s*\)[0-9]*|\1$max_batch_size|g" ./triton_model_repo/postprocessing/config.pbtxt
39 +sed -i.bak "s|\(max_batch_size:\s*\)[0-9]*|\1$max_batch_size|g" ./triton_model_repo/tensorrt_llm_bls/config.pbtxt
40 +
41 +
42 +cd /
43 +rm -rf tensorrtllm_backend
44 +git clone https://github.com/triton-inference-server/tensorrtllm_backend.git
45 +git lfs install
46 +cd tensorrtllm_backend
47 +git checkout $trt_llm_version
48 +tensorrtllm_backend_dir=$(pwd)
49 +git submodule update --init --recursive
50 +cp -r ${tensorrt_demo_dir}/triton_model_repo ${tensorrtllm_backend_dir}/
51 +
52 +cd /tensorrtllm_backend
53 +cd ./tensorrt_llm/examples/${model_type}
54 +
55 +
56 +if echo "$common_params" | jq -e 'has("fp8")' > /dev/null; then
57 +
58 + echo "Key 'fp8' exists in common params. Use quantize.py instead of convert_checkpoint.py"
59 + echo "Reference: https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/llama/README.md"
60 + python ../quantization/quantize.py \
61 + --model_dir ${model_path} \
62 + --dtype ${model_dtype} \
63 + --tp_size ${model_tp_size} \
64 + --output_dir ${trt_model_path} \
65 + --qformat fp8 \
66 + --kv_cache_dtype fp8 \
67 + --calib_size 2
68 +
69 +else
70 +
71 + echo "Key 'fp8' does not exist in common params. Use convert_checkpoint.py"
72 + python3 convert_checkpoint.py \
73 + --model_dir ${model_path} \
74 + --dtype ${model_dtype} \
75 + --tp_size ${model_tp_size} \
76 + --output_dir ${trt_model_path}
77 +
78 +fi
79 +
80 +
81 +
82 +trtllm-build \
83 +--checkpoint_dir=${trt_model_path} \
84 +--gpt_attention_plugin=${model_dtype} \
85 +--gemm_plugin=${model_dtype} \
86 +--remove_input_padding=enable \
87 +--paged_kv_cache=enable \
88 +--tp_size=${model_tp_size} \
89 +--max_batch_size=${max_batch_size} \
90 +--max_input_len=${max_input_len} \
91 +--max_output_len=${max_output_len} \
92 +--max_num_tokens=${max_output_len} \
93 +--opt_num_tokens=${max_output_len} \
94 +--output_dir=${trt_engine_path}
95 +
96 +cd /tensorrtllm_backend/triton_model_repo
97 +rm -rf ./tensorrt_llm/1/*
98 +cp -r ${trt_engine_path}/* ./tensorrt_llm/1
99 +cd /tensorrtllm_backend
100 +python3 scripts/launch_triton_server.py \
101 +--world_size=${model_tp_size} \
102 +--model_repo=/tensorrtllm_backend/triton_model_repo &
\ No newline at end of file
.buildkite/nightly-benchmarks/scripts/nightly-annotate.sh new
+40
@@ -0,0 +1,40 @@
1 +#!/bin/bash
2 +
3 +set -ex
4 +set -o pipefail
5 +
6 +
7 +main() {
8 +
9 + (which wget && which curl) || (apt-get update && apt-get install -y wget curl)
10 + (which jq) || (apt-get update && apt-get -y install jq)
11 +
12 + if [ ! -f /workspace/buildkite-agent ]; then
13 + echo "buildkite-agent binary not found. Skip plotting the results."
14 + exit 0
15 + fi
16 +
17 + # initial annotation
18 + description="$VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/nightly-descriptions.md"
19 +
20 + # download results
21 + cd $VLLM_SOURCE_CODE_LOC/benchmarks
22 + mkdir -p results/
23 + /workspace/buildkite-agent artifact download 'results/*nightly_results.json' results/
24 + ls
25 + ls results/
26 +
27 + # generate figures
28 + python3 -m pip install tabulate pandas matplotlib
29 + python3 $VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/scripts/plot-nightly-results.py \
30 + --description $description \
31 + --results-folder results/
32 +
33 + # upload results and figures
34 + /workspace/buildkite-agent artifact upload "nightly_results.png"
35 + /workspace/buildkite-agent artifact upload $VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/nightly-pipeline.yaml
36 + /workspace/buildkite-agent artifact upload $VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/tests/nightly-tests.json
37 + /workspace/buildkite-agent annotate --style "success" --context "nightly-benchmarks-results" --append < nightly_results.md
38 +}
39 +
40 +main "$@"
\ No newline at end of file
.buildkite/nightly-benchmarks/scripts/plot-nightly-results.py new
+135
@@ -0,0 +1,135 @@
1 +import argparse
2 +import json
3 +import math
4 +from pathlib import Path
5 +
6 +import matplotlib.pyplot as plt
7 +import pandas as pd
8 +from tabulate import tabulate
9 +
10 +
11 +def parse_arguments():
12 + parser = argparse.ArgumentParser(
13 + description=
14 + 'Parse command line arguments for summary-nightly-results script.')
15 + parser.add_argument('--results-folder',
16 + type=str,
17 + required=True,
18 + help='The folder where the results are stored.')
19 + parser.add_argument('--description',
20 + type=str,
21 + required=True,
22 + help='Description of the results.')
23 +
24 + args = parser.parse_args()
25 + return args
26 +
27 +
28 +def main(args):
29 + bar_colors = ['#56B4E9', '#009E73', '#D55E00', '#E69F00']
30 + results_folder = Path(args.results_folder)
31 +
32 + results = []
33 +
34 + # collect results
35 + for test_file in results_folder.glob("*_nightly_results.json"):
36 + with open(test_file, "r") as f:
37 + results = results + json.loads(f.read())
38 +
39 + # generate markdown table
40 + df = pd.DataFrame.from_dict(results)
41 +
42 + md_table = tabulate(df, headers='keys', tablefmt='pipe', showindex=False)
43 +
44 + with open(args.description, "r") as f:
45 + description = f.read()
46 +
47 + description = description.format(
48 + nightly_results_benchmarking_table=md_table)
49 +
50 + with open("nightly_results.md", "w") as f:
51 + f.write(description)
52 +
53 + plt.rcParams.update({'font.size': 20})
54 +
55 + # plot results
56 + fig, axes = plt.subplots(3, 3, figsize=(16, 14))
57 + fig.subplots_adjust(hspace=1)
58 + methods = ["vllm", "trt", "lmdeploy", "tgi"]
59 + for i, model in enumerate(["llama8B", "llama70B", "mixtral8x7B"]):
60 + for j, metric in enumerate(["TTFT", "ITL"]):
61 + means, stds = [], []
62 + for method in methods:
63 + target = df['Test name'].str.contains(model)
64 + target = target & df['Engine'].str.contains(method)
65 + filtered_df = df[target]
66 +
67 + if filtered_df.empty:
68 + means.append(0.)
69 + stds.append(0.)
70 + else:
71 + means.append(filtered_df[f"Mean {metric} (ms)"].values[0])
72 + std = filtered_df[f"Std {metric} (ms)"].values[0]
73 + success = filtered_df["Successful req."].values[0]
74 + stds.append(std / math.sqrt(success))
75 +
76 + print(model, metric)
77 + print(means, stds)
78 +
79 + ax = axes[i, j + 1]
80 +
81 + bars = ax.bar(
82 + ["vllm", "trt", "lmdeploy", "tgi"],
83 + means,
84 + yerr=stds,
85 + capsize=10,
86 + )
87 + for idx, bar in enumerate(bars):
88 + bar.set_color(bar_colors[idx])
89 + ax.set_ylim(bottom=0)
90 +
91 + ax.set_ylabel(f"{metric} (ms)")
92 + ax.set_title(f"{model} {metric}")
93 + ax.grid(axis='y')
94 +
95 + metric = "Tput"
96 + j = 0
97 + if True:
98 + tputs = []
99 + for method in methods:
100 + target = df['Test name'].str.contains(model)
101 + target = target & df['Engine'].str.contains(method)
102 + filtered_df = df[target]
103 +
104 + if filtered_df.empty:
105 + tputs.append(0.)
106 + else:
107 + input_tput = filtered_df["Input Tput (tok/s)"].values[0]
108 + output_tput = filtered_df["Output Tput (tok/s)"].values[0]
109 + tputs.append(input_tput + output_tput)
110 +
111 + print(model, metric)
112 + print(tputs)
113 +
114 + ax = axes[i, j]
115 +
116 + bars = ax.bar(
117 + ["vllm", "trt", "lmdeploy", "tgi"],
118 + tputs,
119 + )
120 + for idx, bar in enumerate(bars):
121 + bar.set_color(bar_colors[idx])
122 +
123 + ax.set_ylim(bottom=0)
124 +
125 + ax.set_ylabel("Tput (token/s)")
126 + ax.set_title(f"{model} {metric}")
127 + ax.grid(axis='y')
128 +
129 + fig.tight_layout()
130 + fig.savefig("nightly_results.png", bbox_inches='tight', dpi=400)
131 +
132 +
133 +if __name__ == '__main__':
134 + args = parse_arguments()
135 + main(args)
.buildkite/nightly-benchmarks/scripts/run-lmdeploy-nightly.sh new
+218
@@ -0,0 +1,218 @@
1 +#!/bin/bash
2 +
3 +set -o pipefail
4 +
5 +check_gpus() {
6 + # check the number of GPUs and GPU type.
7 + declare -g gpu_count=$(nvidia-smi --list-gpus | wc -l)
8 + if [[ $gpu_count -gt 0 ]]; then
9 + echo "GPU found."
10 + else
11 + echo "Need at least 1 GPU to run benchmarking."
12 + exit 1
13 + fi
14 + declare -g gpu_type=$(echo $(nvidia-smi --query-gpu=name --format=csv,noheader) | awk '{print $2}')
15 + echo "GPU type is $gpu_type"
16 +}
17 +
18 +kill_gpu_processes() {
19 + pkill lmdeploy || true
20 + # waiting for GPU processes to be fully killed
21 + sleep 10
22 + # Print the GPU memory usage
23 + # so that we know if all GPU processes are killed.
24 + gpu_memory_usage=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits -i 0)
25 + # The memory usage should be 0 MB.
26 + echo "GPU 0 Memory Usage: $gpu_memory_usage MB"
27 +}
28 +
29 +json2args() {
30 + # transforms the JSON string to command line args, and '_' is replaced to '-'
31 + # example:
32 + # input: { "model": "meta-llama/Llama-2-7b-chat-hf", "tensor_parallel_size": 1 }
33 + # output: --model meta-llama/Llama-2-7b-chat-hf --tensor-parallel-size 1
34 + local json_string=$1
35 + local args=$(
36 + echo "$json_string" | jq -r '
37 + to_entries |
38 + map("--" + (.key | gsub("_"; "-")) + " " + (.value | tostring)) |
39 + join(" ")
40 + '
41 + )
42 + echo "$args"
43 +}
44 +
45 +wait_for_server() {
46 + # wait for vllm server to start
47 + # return 1 if vllm server crashes
48 + timeout 1200 bash -c '
49 + until curl -s localhost:8000/v1/completions > /dev/null; do
50 + sleep 1
51 + done' && return 0 || return 1
52 +}
53 +
54 +run_serving_tests() {
55 + # run serving tests using `benchmark_serving.py`
56 + # $1: a json file specifying serving test cases
57 +
58 + local serving_test_file
59 + serving_test_file=$1
60 +
61 + # Iterate over serving tests
62 + jq -c '.[]' "$serving_test_file" | while read -r params; do
63 + # get the test name, and append the GPU type back to it.
64 + test_name=$(echo "$params" | jq -r '.test_name')
65 +
66 + # if TEST_SELECTOR is set, only run the test cases that match the selector
67 + if [[ -n "$TEST_SELECTOR" ]] && [[ ! "$test_name" =~ $TEST_SELECTOR ]]; then
68 + echo "Skip test case $test_name."
69 + continue
70 + fi
71 +
72 + # append lmdeploy to the test name
73 + test_name=lmdeploy_$test_name
74 +
75 + # get common parameters
76 + common_params=$(echo "$params" | jq -r '.common_parameters')
77 + model=$(echo "$common_params" | jq -r '.model')
78 + tp=$(echo "$common_params" | jq -r '.tp')
79 + dataset_name=$(echo "$common_params" | jq -r '.dataset_name')
80 + dataset_path=$(echo "$common_params" | jq -r '.dataset_path')
81 + port=$(echo "$common_params" | jq -r '.port')
82 + num_prompts=$(echo "$common_params" | jq -r '.num_prompts')
83 +
84 +
85 +
86 + # get client and server arguments
87 + server_params=$(echo "$params" | jq -r '.lmdeploy_server_parameters')
88 + client_params=$(echo "$params" | jq -r '.lmdeploy_client_parameters')
89 + server_args=$(json2args "$server_params")
90 + client_args=$(json2args "$client_params")
91 + qps_list=$(echo "$params" | jq -r '.qps_list')
92 + qps_list=$(echo "$qps_list" | jq -r '.[] | @sh')
93 + echo "Running over qps list $qps_list"
94 +
95 + # check if there is enough GPU to run the test
96 + if [[ $gpu_count -lt $tp ]]; then
97 + echo "Required tensor-parallel-size $tp but only $gpu_count GPU found. Skip testcase $test_name."
98 + continue
99 + fi
100 +
101 + # prepare tokenizer
102 + rm -rf /tokenizer_cache
103 + mkdir /tokenizer_cache
104 + python ../.buildkite/nightly-benchmarks/scripts/download-tokenizer.py \
105 + --model "$model" \
106 + --cachedir /tokenizer_cache
107 +
108 + server_command="lmdeploy serve api_server $model \
109 + --tp $tp \
110 + --server-port $port \
111 + $server_args"
112 +
113 + # run the server
114 + echo "Running test case $test_name"
115 + echo "Server command: $server_command"
116 + bash -c "$server_command" &
117 +
118 + # wait until the server is alive
119 + wait_for_server
120 + if [ $? -eq 0 ]; then
121 + echo ""
122 + echo "lmdeploy server is up and running."
123 + else
124 + echo ""
125 + echo "lmdeploy failed to start within the timeout period."
126 + break
127 + fi
128 +
129 + # get model name
130 + model_name=$(python ../.buildkite/nightly-benchmarks/scripts/get-lmdeploy-modelname.py)
131 +
132 + # iterate over different QPS
133 + for qps in $qps_list; do
134 + # remove the surrounding single quote from qps
135 + if [[ "$qps" == *"inf"* ]]; then
136 + echo "qps was $qps"
137 + qps="inf"
138 + echo "now qps is $qps"
139 + fi
140 +
141 + new_test_name=$test_name"_qps_"$qps
142 +
143 + client_command="python3 benchmark_serving.py \
144 + --backend lmdeploy \
145 + --tokenizer /tokenizer_cache \
146 + --dataset-name $dataset_name \
147 + --dataset-path $dataset_path \
148 + --num-prompts $num_prompts \
149 + --port $port \
150 + --save-result \
151 + --result-dir $RESULTS_FOLDER \
152 + --result-filename ${new_test_name}.json \
153 + --request-rate $qps \
154 + --model \"$model_name\" \
155 + $client_args"
156 +
157 + echo "Running test case $test_name with qps $qps"
158 + echo "Client command: $client_command"
159 +
160 + eval "$client_command"
161 +
162 + # record the benchmarking commands
163 + jq_output=$(jq -n \
164 + --arg server "$server_command" \
165 + --arg client "$client_command" \
166 + --arg gpu "$gpu_type" \
167 + --arg engine "lmdeploy" \
168 + '{
169 + server_command: $server,
170 + client_command: $client,
171 + gpu_type: $gpu,
172 + engine: $engine
173 + }')
174 + echo "$jq_output" >"$RESULTS_FOLDER/${new_test_name}.commands"
175 +
176 + done
177 +
178 + # clean up
179 + kill_gpu_processes
180 + rm -rf /root/.cache/huggingface/*
181 + done
182 +}
183 +
184 +
185 +upload_to_buildkite() {
186 + # upload the benchmarking results to buildkite
187 +
188 + # if the agent binary is not found, skip uploading the results, exit 0
189 + if [ ! -f /workspace/buildkite-agent ]; then
190 + echo "buildkite-agent binary not found. Skip uploading the results."
191 + return 0
192 + fi
193 + # /workspace/buildkite-agent annotate --style "success" --context "benchmark-results" --append < $RESULTS_FOLDER/${CURRENT_LLM_SERVING_ENGINE}_nightly_results.md
194 + /workspace/buildkite-agent artifact upload "$RESULTS_FOLDER/*"
195 +}
196 +
197 +
198 +main() {
199 +
200 + check_gpus
201 + # enter vllm directory
202 + cd $VLLM_SOURCE_CODE_LOC/benchmarks
203 +
204 + declare -g RESULTS_FOLDER=results/
205 + mkdir -p $RESULTS_FOLDER
206 + BENCHMARK_ROOT=../.buildkite/nightly-benchmarks/
207 +
208 + python -m pip install transformers==4.41.2
209 +
210 + export CURRENT_LLM_SERVING_ENGINE=lmdeploy
211 + run_serving_tests $BENCHMARK_ROOT/tests/nightly-tests.json
212 + python -m pip install tabulate pandas
213 + python $BENCHMARK_ROOT/scripts/summary-nightly-results.py
214 + upload_to_buildkite
215 +
216 +}
217 +
218 +main "$@"
.buildkite/nightly-benchmarks/scripts/run-tgi-nightly.sh new
+216
@@ -0,0 +1,216 @@
1 +#!/bin/bash
2 +
3 +set -o pipefail
4 +
5 +check_gpus() {
6 + # check the number of GPUs and GPU type.
7 + declare -g gpu_count=$(nvidia-smi --list-gpus | wc -l)
8 + if [[ $gpu_count -gt 0 ]]; then
9 + echo "GPU found."
10 + else
11 + echo "Need at least 1 GPU to run benchmarking."
12 + exit 1
13 + fi
14 + declare -g gpu_type=$(echo $(nvidia-smi --query-gpu=name --format=csv,noheader) | awk '{print $2}')
15 + echo "GPU type is $gpu_type"
16 +}
17 +
18 +kill_gpu_processes() {
19 + pkill text-generation || true
20 + # waiting for GPU processes to be fully killed
21 + sleep 10
22 + # Print the GPU memory usage
23 + # so that we know if all GPU processes are killed.
24 + gpu_memory_usage=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits -i 0)
25 + # The memory usage should be 0 MB.
26 + echo "GPU 0 Memory Usage: $gpu_memory_usage MB"
27 +}
28 +
29 +json2args() {
30 + # transforms the JSON string to command line args, and '_' is replaced to '-'
31 + # example:
32 + # input: { "model": "meta-llama/Llama-2-7b-chat-hf", "tensor_parallel_size": 1 }
33 + # output: --model meta-llama/Llama-2-7b-chat-hf --tensor-parallel-size 1
34 + local json_string=$1
35 + local args=$(
36 + echo "$json_string" | jq -r '
37 + to_entries |
38 + map("--" + (.key | gsub("_"; "-")) + " " + (.value | tostring)) |
39 + join(" ")
40 + '
41 + )
42 + echo "$args"
43 +}
44 +
45 +wait_for_server() {
46 + timeout 1200 bash -c '
47 + until curl -s localhost:8000/generate_stream > /dev/null; do
48 + sleep 1
49 + done' && return 0 || return 1
50 +}
51 +
52 +run_serving_tests() {
53 + # run serving tests using `benchmark_serving.py`
54 + # $1: a json file specifying serving test cases
55 +
56 + local serving_test_file
57 + serving_test_file=$1
58 +
59 + # Iterate over serving tests
60 + jq -c '.[]' "$serving_test_file" | while read -r params; do
61 + # get the test name, and append the GPU type back to it.
62 + test_name=$(echo "$params" | jq -r '.test_name')
63 +
64 +
65 + # if TEST_SELECTOR is set, only run the test cases that match the selector
66 + if [[ -n "$TEST_SELECTOR" ]] && [[ ! "$test_name" =~ $TEST_SELECTOR ]]; then
67 + echo "Skip test case $test_name."
68 + continue
69 + fi
70 +
71 + # append tgi to the test name
72 + test_name=tgi_$test_name
73 +
74 + # get common parameters
75 + common_params=$(echo "$params" | jq -r '.common_parameters')
76 + model=$(echo "$common_params" | jq -r '.model')
77 + tp=$(echo "$common_params" | jq -r '.tp')
78 + dataset_name=$(echo "$common_params" | jq -r '.dataset_name')
79 + dataset_path=$(echo "$common_params" | jq -r '.dataset_path')
80 + port=$(echo "$common_params" | jq -r '.port')
81 + num_prompts=$(echo "$common_params" | jq -r '.num_prompts')
82 +
83 + # get client and server arguments
84 + server_params=$(echo "$params" | jq -r '.tgi_server_parameters')
85 + client_params=$(echo "$params" | jq -r '.tgi_client_parameters')
86 + server_args=$(json2args "$server_params")
87 + client_args=$(json2args "$client_params")
88 + qps_list=$(echo "$params" | jq -r '.qps_list')
89 + qps_list=$(echo "$qps_list" | jq -r '.[] | @sh')
90 + echo "Running over qps list $qps_list"
91 +
92 + # check if there is enough GPU to run the test
93 + if [[ $gpu_count -lt $tp ]]; then
94 + echo "Required num-shard $tp but only $gpu_count GPU found. Skip testcase $test_name."
95 + continue
96 + fi
97 +
98 + if echo "$common_params" | jq -e 'has("fp8")' > /dev/null; then
99 + echo "Key 'fp8' exists in common params."
100 + server_command="/tgi-entrypoint.sh \
101 + --model-id $model \
102 + --num-shard $tp \
103 + --port $port \
104 + --quantize fp8 \
105 + $server_args"
106 + else
107 + echo "Key 'fp8' does not exist in common params."
108 + server_command="/tgi-entrypoint.sh \
109 + --model-id $model \
110 + --num-shard $tp \
111 + --port $port \
112 + $server_args"
113 + fi
114 +
115 +
116 +
117 +
118 + # run the server
119 + echo "Running test case $test_name"
120 + echo "Server command: $server_command"
121 + eval "$server_command" &
122 +
123 + # wait until the server is alive
124 + wait_for_server
125 + if [ $? -eq 0 ]; then
126 + echo ""
127 + echo "tgi server is up and running."
128 + else
129 + echo ""
130 + echo "tgi failed to start within the timeout period."
131 + break
132 + fi
133 +
134 + # iterate over different QPS
135 + for qps in $qps_list; do
136 + # remove the surrounding single quote from qps
137 + if [[ "$qps" == *"inf"* ]]; then
138 + echo "qps was $qps"
139 + qps="inf"
140 + echo "now qps is $qps"
141 + fi
142 +
143 + new_test_name=$test_name"_qps_"$qps
144 +
145 + client_command="python3 benchmark_serving.py \
146 + --backend tgi \
147 + --model $model \
148 + --dataset-name $dataset_name \
149 + --dataset-path $dataset_path \
150 + --num-prompts $num_prompts \
151 + --port $port \
152 + --save-result \
153 + --result-dir $RESULTS_FOLDER \
154 + --result-filename ${new_test_name}.json \
155 + --request-rate $qps \
156 + $client_args"
157 +
158 + echo "Running test case $test_name with qps $qps"
159 + echo "Client command: $client_command"
160 +
161 + eval "$client_command"
162 +
163 + # record the benchmarking commands
164 + jq_output=$(jq -n \
165 + --arg server "$server_command" \
166 + --arg client "$client_command" \
167 + --arg gpu "$gpu_type" \
168 + --arg engine "tgi" \
169 + '{
170 + server_command: $server,
171 + client_command: $client,
172 + gpu_type: $gpu,
173 + engine: $engine
174 + }')
175 + echo "$jq_output" >"$RESULTS_FOLDER/${new_test_name}.commands"
176 +
177 + done
178 +
179 + # clean up
180 + kill_gpu_processes
181 + rm -rf /root/.cache/huggingface/*
182 + done
183 +}
184 +
185 +
186 +
187 +upload_to_buildkite() {
188 + # upload the benchmarking results to buildkite
189 +
190 + # if the agent binary is not found, skip uploading the results, exit 0
191 + if [ ! -f /workspace/buildkite-agent ]; then
192 + echo "buildkite-agent binary not found. Skip uploading the results."
193 + return 0
194 + fi
195 + # /workspace/buildkite-agent annotate --style "success" --context "benchmark-results" --append < $RESULTS_FOLDER/${CURRENT_LLM_SERVING_ENGINE}_nightly_results.md
196 + /workspace/buildkite-agent artifact upload "$RESULTS_FOLDER/*"
197 +}
198 +
199 +main() {
200 +
201 + check_gpus
202 + # enter vllm directory
203 + cd $VLLM_SOURCE_CODE_LOC/benchmarks
204 + declare -g RESULTS_FOLDER=results/
205 + mkdir -p $RESULTS_FOLDER
206 + BENCHMARK_ROOT=../.buildkite/nightly-benchmarks/
207 +
208 + export CURRENT_LLM_SERVING_ENGINE=tgi
209 + run_serving_tests $BENCHMARK_ROOT/tests/nightly-tests.json
210 + python -m pip install tabulate pandas
211 + python $BENCHMARK_ROOT/scripts/summary-nightly-results.py
212 + upload_to_buildkite
213 +
214 +}
215 +
216 +main "$@"
.buildkite/nightly-benchmarks/scripts/run-trt-nightly.sh new
+214
@@ -0,0 +1,214 @@
1 +#!/bin/bash
2 +
3 +set -o pipefail
4 +
5 +check_gpus() {
6 + # check the number of GPUs and GPU type.
7 + declare -g gpu_count=$(nvidia-smi --list-gpus | wc -l)
8 + if [[ $gpu_count -gt 0 ]]; then
9 + echo "GPU found."
10 + else
11 + echo "Need at least 1 GPU to run benchmarking."
12 + exit 1
13 + fi
14 + declare -g gpu_type=$(echo $(nvidia-smi --query-gpu=name --format=csv,noheader) | awk '{print $2}')
15 + echo "GPU type is $gpu_type"
16 +}
17 +
18 +kill_gpu_processes() {
19 + pkill tritonserver || true
20 + # waiting for GPU processes to be fully killed
21 + sleep 20
22 + # Print the GPU memory usage
23 + # so that we know if all GPU processes are killed.
24 + gpu_memory_usage=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits -i 0)
25 + # The memory usage should be 0 MB.
26 + echo "GPU 0 Memory Usage: $gpu_memory_usage MB"
27 +}
28 +
29 +json2args() {
30 + # transforms the JSON string to command line args, and '_' is replaced to '-'
31 + # example:
32 + # input: { "model": "meta-llama/Llama-2-7b-chat-hf", "tensor_parallel_size": 1 }
33 + # output: --model meta-llama/Llama-2-7b-chat-hf --tensor-parallel-size 1
34 + local json_string=$1
35 + local args=$(
36 + echo "$json_string" | jq -r '
37 + to_entries |
38 + map("--" + (.key | gsub("_"; "-")) + " " + (.value | tostring)) |
39 + join(" ")
40 + '
41 + )
42 + echo "$args"
43 +}
44 +
45 +wait_for_server() {
46 + timeout 1200 bash -c '
47 + until curl -s localhost:8000/generate_stream > /dev/null; do
48 + sleep 1
49 + done' && return 0 || return 1
50 +}
51 +
52 +run_serving_tests() {
53 + # run serving tests using `benchmark_serving.py`
54 + # $1: a json file specifying serving test cases
55 +
56 + local serving_test_file
57 + serving_test_file=$1
58 +
59 + # Iterate over serving tests
60 + jq -c '.[]' "$serving_test_file" | while read -r params; do
61 + # get the test name, and append the GPU type back to it.
62 + test_name=$(echo "$params" | jq -r '.test_name')
63 +
64 + # if TEST_SELECTOR is set, only run the test cases that match the selector
65 + if [[ -n "$TEST_SELECTOR" ]] && [[ ! "$test_name" =~ $TEST_SELECTOR ]]; then
66 + echo "Skip test case $test_name."
67 + continue
68 + fi
69 +
70 + # append trt to the test name
71 + test_name=trt_$test_name
72 +
73 + # get common parameters
74 + common_params=$(echo "$params" | jq -r '.common_parameters')
75 + model=$(echo "$common_params" | jq -r '.model')
76 + tp=$(echo "$common_params" | jq -r '.tp')
77 + dataset_name=$(echo "$common_params" | jq -r '.dataset_name')
78 + dataset_path=$(echo "$common_params" | jq -r '.dataset_path')
79 + port=$(echo "$common_params" | jq -r '.port')
80 + num_prompts=$(echo "$common_params" | jq -r '.num_prompts')
81 +
82 + # get client and server arguments
83 + server_params=$(echo "$params" | jq -r '.trt_server_parameters')
84 + client_params=$(echo "$params" | jq -r '.trt_client_parameters')
85 + client_args=$(json2args "$client_params")
86 + qps_list=$(echo "$params" | jq -r '.qps_list')
87 + qps_list=$(echo "$qps_list" | jq -r '.[] | @sh')
88 + echo "Running over qps list $qps_list"
89 +
90 + # check if there is enough GPU to run the test
91 + if [[ $gpu_count -lt $tp ]]; then
92 + echo "Required model_tp_size $tp but only $gpu_count GPU found. Skip testcase $test_name."
93 + continue
94 + fi
95 +
96 +
97 +
98 + cd $VLLM_SOURCE_CODE_LOC/benchmarks
99 +
100 +
101 + echo "Running test case $test_name"
102 + bash ../.buildkite/nightly-benchmarks/scripts/launch-trt-server.sh "$server_params" "$common_params"
103 +
104 + # wait until the server is alive
105 + wait_for_server
106 + if [ $? -eq 0 ]; then
107 + echo ""
108 + echo "trt server is up and running."
109 + else
110 + echo ""
111 + echo "trt failed to start within the timeout period."
112 + break
113 + fi
114 +
115 + # prepare tokenizer
116 + cd $VLLM_SOURCE_CODE_LOC/benchmarks
117 + rm -rf /tokenizer_cache
118 + mkdir /tokenizer_cache
119 + python ../.buildkite/nightly-benchmarks/scripts/download-tokenizer.py \
120 + --model "$model" \
121 + --cachedir /tokenizer_cache
122 + cd $VLLM_SOURCE_CODE_LOC/benchmarks
123 +
124 +
125 + # iterate over different QPS
126 + for qps in $qps_list; do
127 + # remove the surrounding single quote from qps
128 + if [[ "$qps" == *"inf"* ]]; then
129 + echo "qps was $qps"
130 + qps="inf"
131 + echo "now qps is $qps"
132 + fi
133 +
134 + new_test_name=$test_name"_qps_"$qps
135 +
136 + client_command="python3 benchmark_serving.py \
137 + --backend tensorrt-llm \
138 + --tokenizer /tokenizer_cache \
139 + --model $model \
140 + --dataset-name $dataset_name \
141 + --dataset-path $dataset_path \
142 + --num-prompts $num_prompts \
143 + --port $port \
144 + --save-result \
145 + --result-dir $RESULTS_FOLDER \
146 + --result-filename ${new_test_name}.json \
147 + --request-rate $qps \
148 + $client_args"
149 +
150 + echo "Running test case $test_name with qps $qps"
151 + echo "Client command: $client_command"
152 +
153 + eval "$client_command"
154 +
155 + server_command=""
156 + # record the benchmarking commands
157 + jq_output=$(jq -n \
158 + --arg server "$server_command" \
159 + --arg client "$client_command" \
160 + --arg gpu "$gpu_type" \
161 + --arg engine "trt" \
162 + '{
163 + server_command: $server,
164 + client_command: $client,
165 + gpu_type: $gpu,
166 + engine: $engine
167 + }')
168 + echo "$jq_output" >"$RESULTS_FOLDER/${new_test_name}.commands"
169 +
170 + done
171 +
172 + # clean up
173 + kill_gpu_processes
174 + rm -rf /root/.cache/huggingface/*
175 + done
176 +}
177 +
178 +upload_to_buildkite() {
179 + # upload the benchmarking results to buildkite
180 +
181 + # if the agent binary is not found, skip uploading the results, exit 0
182 + if [ ! -f /workspace/buildkite-agent ]; then
183 + echo "buildkite-agent binary not found. Skip uploading the results."
184 + return 0
185 + fi
186 + # /workspace/buildkite-agent annotate --style "success" --context "benchmark-results" --append < $RESULTS_FOLDER/${CURRENT_LLM_SERVING_ENGINE}_nightly_results.md
187 + /workspace/buildkite-agent artifact upload "$RESULTS_FOLDER/*"
188 +}
189 +
190 +
191 +main() {
192 +
193 + check_gpus
194 +
195 +
196 + # enter vllm directory
197 + cd $VLLM_SOURCE_CODE_LOC/benchmarks
198 +
199 + declare -g RESULTS_FOLDER=results/
200 + mkdir -p $RESULTS_FOLDER
201 + BENCHMARK_ROOT=../.buildkite/nightly-benchmarks/
202 +
203 + # update transformers package, to make sure mixtral tokenizer is available
204 + python -m pip install transformers -U
205 +
206 + export CURRENT_LLM_SERVING_ENGINE=trt
207 + run_serving_tests $BENCHMARK_ROOT/tests/nightly-tests.json
208 + python -m pip install tabulate pandas
209 + python $BENCHMARK_ROOT/scripts/summary-nightly-results.py
210 + upload_to_buildkite
211 +
212 +}
213 +
214 +main "$@"
.buildkite/nightly-benchmarks/scripts/run-vllm-nightly.sh new
+221
@@ -0,0 +1,221 @@
1 +#!/bin/bash
2 +
3 +set -o pipefail
4 +
5 +check_gpus() {
6 + # check the number of GPUs and GPU type.
7 + declare -g gpu_count=$(nvidia-smi --list-gpus | wc -l)
8 + if [[ $gpu_count -gt 0 ]]; then
9 + echo "GPU found."
10 + else
11 + echo "Need at least 1 GPU to run benchmarking."
12 + exit 1
13 + fi
14 + declare -g gpu_type=$(echo $(nvidia-smi --query-gpu=name --format=csv,noheader) | awk '{print $2}')
15 + echo "GPU type is $gpu_type"
16 +}
17 +
18 +kill_gpu_processes() {
19 + # kill all processes on GPU.
20 + pkill pt_main_thread
21 + sleep 10
22 +
23 + # remove vllm config file
24 + rm -rf ~/.config/vllm
25 +
26 + # Print the GPU memory usage
27 + # so that we know if all GPU processes are killed.
28 + gpu_memory_usage=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits -i 0)
29 + # The memory usage should be 0 MB.
30 + echo "GPU 0 Memory Usage: $gpu_memory_usage MB"
31 +}
32 +
33 +json2args() {
34 + # transforms the JSON string to command line args, and '_' is replaced to '-'
35 + # example:
36 + # input: { "model": "meta-llama/Llama-2-7b-chat-hf", "tensor_parallel_size": 1 }
37 + # output: --model meta-llama/Llama-2-7b-chat-hf --tensor-parallel-size 1
38 + local json_string=$1
39 + local args=$(
40 + echo "$json_string" | jq -r '
41 + to_entries |
42 + map("--" + (.key | gsub("_"; "-")) + " " + (.value | tostring)) |
43 + join(" ")
44 + '
45 + )
46 + echo "$args"
47 +}
48 +
49 +wait_for_server() {
50 + # wait for vllm server to start
51 + # return 1 if vllm server crashes
52 + timeout 1200 bash -c '
53 + until curl -s localhost:8000/v1/completions > /dev/null; do
54 + sleep 1
55 + done' && return 0 || return 1
56 +}
57 +
58 +run_serving_tests() {
59 + # run serving tests using `benchmark_serving.py`
60 + # $1: a json file specifying serving test cases
61 +
62 + local serving_test_file
63 + serving_test_file=$1
64 +
65 + # Iterate over serving tests
66 + jq -c '.[]' "$serving_test_file" | while read -r params; do
67 + # get the test name, and append the GPU type back to it.
68 + test_name=$(echo "$params" | jq -r '.test_name')
69 +
70 + # if TEST_SELECTOR is set, only run the test cases that match the selector
71 + if [[ -n "$TEST_SELECTOR" ]] && [[ ! "$test_name" =~ $TEST_SELECTOR ]]; then
72 + echo "Skip test case $test_name."
73 + continue
74 + fi
75 +
76 + # append vllm to the test name
77 + test_name=vllm_$test_name
78 +
79 +
80 + # get common parameters
81 + common_params=$(echo "$params" | jq -r '.common_parameters')
82 + model=$(echo "$common_params" | jq -r '.model')
83 + tp=$(echo "$common_params" | jq -r '.tp')
84 + dataset_name=$(echo "$common_params" | jq -r '.dataset_name')
85 + dataset_path=$(echo "$common_params" | jq -r '.dataset_path')
86 + port=$(echo "$common_params" | jq -r '.port')
87 + num_prompts=$(echo "$common_params" | jq -r '.num_prompts')
88 +
89 + # get client and server arguments
90 + server_params=$(echo "$params" | jq -r '.vllm_server_parameters')
91 + client_params=$(echo "$params" | jq -r '.vllm_client_parameters')
92 + server_args=$(json2args "$server_params")
93 + client_args=$(json2args "$client_params")
94 + qps_list=$(echo "$params" | jq -r '.qps_list')
95 + qps_list=$(echo "$qps_list" | jq -r '.[] | @sh')
96 + echo "Running over qps list $qps_list"
97 +
98 + # check if there is enough GPU to run the test
99 + if [[ $gpu_count -lt $tp ]]; then
100 + echo "Required tensor-parallel-size $tp but only $gpu_count GPU found. Skip testcase $test_name."
101 + continue
102 + fi
103 +
104 + if echo "$common_params" | jq -e 'has("fp8")' > /dev/null; then
105 + echo "Key 'fp8' exists in common params. Use neuralmagic fp8 model for convenience."
106 + model=$(echo "$common_params" | jq -r '.neuralmagic_quantized_model')
107 + server_command="python3 \
108 + -m vllm.entrypoints.openai.api_server \
109 + -tp $tp \
110 + --model $model \
111 + --port $port \
112 + $server_args"
113 + else
114 + echo "Key 'fp8' does not exist in common params."
115 + server_command="python3 \
116 + -m vllm.entrypoints.openai.api_server \
117 + -tp $tp \
118 + --model $model \
119 + --port $port \
120 + $server_args"
121 + fi
122 +
123 + # run the server
124 + echo "Running test case $test_name"
125 + echo "Server command: $server_command"
126 + eval "$server_command" &
127 +
128 + # wait until the server is alive
129 + wait_for_server
130 + if [ $? -eq 0 ]; then
131 + echo ""
132 + echo "vllm server is up and running."
133 + else
134 + echo ""
135 + echo "vllm failed to start within the timeout period."
136 + break
137 + fi
138 +
139 + # iterate over different QPS
140 + for qps in $qps_list; do
141 + # remove the surrounding single quote from qps
142 + if [[ "$qps" == *"inf"* ]]; then
143 + echo "qps was $qps"
144 + qps="inf"
145 + echo "now qps is $qps"
146 + fi
147 +
148 + new_test_name=$test_name"_qps_"$qps
149 +
150 + client_command="python3 benchmark_serving.py \
151 + --backend vllm \
152 + --model $model \
153 + --dataset-name $dataset_name \
154 + --dataset-path $dataset_path \
155 + --num-prompts $num_prompts \
156 + --port $port \
157 + --save-result \
158 + --result-dir $RESULTS_FOLDER \
159 + --result-filename ${new_test_name}.json \
160 + --request-rate $qps \
161 + $client_args"
162 +
163 + echo "Running test case $test_name with qps $qps"
164 + echo "Client command: $client_command"
165 +
166 + eval "$client_command"
167 +
168 + # record the benchmarking commands
169 + jq_output=$(jq -n \
170 + --arg server "$server_command" \
171 + --arg client "$client_command" \
172 + --arg gpu "$gpu_type" \
173 + --arg engine "vllm" \
174 + '{
175 + server_command: $server,
176 + client_command: $client,
177 + gpu_type: $gpu,
178 + engine: $engine
179 + }')
180 + echo "$jq_output" >"$RESULTS_FOLDER/${new_test_name}.commands"
181 +
182 + done
183 +
184 + # clean up
185 + kill_gpu_processes
186 + rm -rf /root/.cache/huggingface/*
187 + done
188 +}
189 +
190 +
191 +upload_to_buildkite() {
192 + # upload the benchmarking results to buildkite
193 +
194 + # if the agent binary is not found, skip uploading the results, exit 0
195 + if [ ! -f /workspace/buildkite-agent ]; then
196 + echo "buildkite-agent binary not found. Skip uploading the results."
197 + return 0
198 + fi
199 + # /workspace/buildkite-agent annotate --style "success" --context "benchmark-results" --append < $RESULTS_FOLDER/${CURRENT_LLM_SERVING_ENGINE}_nightly_results.md
200 + /workspace/buildkite-agent artifact upload "$RESULTS_FOLDER/*"
201 +}
202 +
203 +main() {
204 +
205 + check_gpus
206 + # enter vllm directory
207 + cd $VLLM_SOURCE_CODE_LOC/benchmarks
208 + declare -g RESULTS_FOLDER=results/
209 + mkdir -p $RESULTS_FOLDER
210 + BENCHMARK_ROOT=../.buildkite/nightly-benchmarks/
211 +
212 + export CURRENT_LLM_SERVING_ENGINE=vllm
213 + run_serving_tests $BENCHMARK_ROOT/tests/nightly-tests.json
214 +
215 + python3 -m pip install tabulate pandas
216 + python3 $BENCHMARK_ROOT/scripts/summary-nightly-results.py
217 + upload_to_buildkite
218 +
219 +}
220 +
221 +main "$@"
.buildkite/nightly-benchmarks/scripts/summary-nightly-results.py new
+76
@@ -0,0 +1,76 @@
1 +import datetime
2 +import json
3 +import os
4 +from pathlib import Path
5 +
6 +import pandas as pd
7 +from tabulate import tabulate
8 +
9 +results_folder = Path("results/")
10 +
11 +# serving results and the keys that will be printed into markdown
12 +serving_results = []
13 +serving_column_mapping = {
14 + "test_name": "Test name",
15 + "gpu_type": "GPU",
16 + "completed": "Successful req.",
17 + "request_throughput": "Tput (req/s)",
18 + "mean_ttft_ms": "Mean TTFT (ms)",
19 + "std_ttft_ms": "Std TTFT (ms)",
20 + "mean_itl_ms": "Mean ITL (ms)",
21 + "std_itl_ms": "Std ITL (ms)",
22 + "input_throughput": "Input Tput (tok/s)",
23 + "output_throughput": "Output Tput (tok/s)",
24 + "engine": "Engine",
25 +}
26 +
27 +if __name__ == "__main__":
28 +
29 + # collect results
30 + for test_file in results_folder.glob("*.json"):
31 +
32 + with open(test_file, "r") as f:
33 + raw_result = json.loads(f.read())
34 +
35 + # attach the benchmarking command to raw_result
36 + with open(test_file.with_suffix(".commands"), "r") as f:
37 + command = json.loads(f.read())
38 + raw_result.update(command)
39 +
40 + # update the test name of this result
41 + raw_result.update({"test_name": test_file.stem})
42 +
43 + # add the result to raw_result
44 + serving_results.append(raw_result)
45 + continue
46 +
47 + serving_results = pd.DataFrame.from_dict(serving_results)
48 +
49 + if not serving_results.empty:
50 + serving_results = serving_results[list(
51 + serving_column_mapping.keys())].rename(
52 + columns=serving_column_mapping)
53 +
54 + serving_md_table_with_headers = tabulate(serving_results,
55 + headers='keys',
56 + tablefmt='pipe',
57 + showindex=False)
58 + # remove the first line of header
59 + serving_md_table_lines = serving_md_table_with_headers.split('\n')
60 + serving_md_table_without_header = '\n'.join(serving_md_table_lines[2:])
61 +
62 + prefix = datetime.datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
63 + prefix = prefix + "_" + os.environ.get("CURRENT_LLM_SERVING_ENGINE")
64 +
65 + # document benchmarking results in markdown
66 + with open(results_folder / f"{prefix}_nightly_results.md", "w") as f:
67 + # document results with header.
68 + # for those who wants to reproduce our benchmark.
69 + f.write(serving_md_table_with_headers)
70 + f.write('\n')
71 +
72 + # document benchmarking results in json
73 + with open(results_folder / f"{prefix}_nightly_results.json", "w") as f:
74 +
75 + results = serving_results.to_dict(orient='records')
76 + f.write(json.dumps(results))
.buildkite/nightly-benchmarks/tests/nightly-tests.json new
+116
@@ -0,0 +1,116 @@
1 +[
2 + {
3 + "test_name": "llama8B_tp1",
4 + "qps_list": [4],
5 + "common_parameters": {
6 + "model": "meta-llama/Meta-Llama-3-8B",
7 + "tp": 1,
8 + "dataset_name": "sharegpt",
9 + "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
10 + "num_prompts": 500,
11 + "port": 8000
12 + },
13 + "lmdeploy_server_parameters": {
14 + },
15 + "lmdeploy_client_parameters": {
16 + },
17 + "tgi_server_parameters": {
18 + },
19 + "tgi_client_parameters": {
20 + "endpoint": "/generate_stream"
21 + },
22 + "trt_server_parameters": {
23 + "model_type": "llama",
24 + "model_dtype": "float16",
25 + "max_batch_size": 256,
26 + "max_input_len": 4096,
27 + "max_output_len": 4096,
28 + "trt_llm_version": "r24.04"
29 + },
30 + "trt_client_parameters": {
31 + "endpoint": "/v2/models/ensemble/generate_stream"
32 + },
33 + "vllm_server_parameters": {
34 + "disable_log_stats": "",
35 + "disable_log_requests": ""
36 + },
37 + "vllm_client_parameters": {
38 + }
39 + },
40 + {
41 + "test_name": "llama70B_tp4",
42 + "qps_list": [2],
43 + "common_parameters": {
44 + "model": "meta-llama/Meta-Llama-3-70B-Instruct",
45 + "tp": 4,
46 + "dataset_name": "sharegpt",
47 + "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
48 + "num_prompts": 500,
49 + "port": 8000
50 + },
51 + "lmdeploy_server_parameters": {
52 + },
53 + "lmdeploy_client_parameters": {
54 + },
55 + "tgi_server_parameters": {
56 + },
57 + "tgi_client_parameters": {
58 + "endpoint": "/generate_stream"
59 + },
60 + "trt_server_parameters": {
61 + "model_type": "llama",
62 + "model_dtype": "float16",
63 + "max_batch_size": 256,
64 + "max_input_len": 4096,
65 + "max_output_len": 4096,
66 + "trt_llm_version": "r24.04"
67 + },
68 + "trt_client_parameters": {
69 + "endpoint": "/v2/models/ensemble/generate_stream"
70 + },
71 + "vllm_server_parameters": {
72 + "disable_log_stats": "",
73 + "disable_log_requests": ""
74 + },
75 + "vllm_client_parameters": {
76 + }
77 + },
78 + {
79 + "test_name": "mixtral8x7B_tp2",
80 + "qps_list": [2],
81 + "common_parameters": {
82 + "model": "mistralai/Mixtral-8x7B-Instruct-v0.1",
83 + "tp": 2,
84 + "dataset_name": "sharegpt",
85 + "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
86 + "num_prompts": 500,
87 + "port": 8000
88 + },
89 + "lmdeploy_server_parameters": {
90 + },
91 + "lmdeploy_client_parameters": {
92 + },
93 + "tgi_server_parameters": {
94 + },
95 + "tgi_client_parameters": {
96 + "endpoint": "/generate_stream"
97 + },
98 + "trt_server_parameters": {
99 + "model_type": "llama",
100 + "model_dtype": "float16",
101 + "max_batch_size": 256,
102 + "max_input_len": 4096,
103 + "max_output_len": 4096,
104 + "trt_llm_version": "r24.04"
105 + },
106 + "trt_client_parameters": {
107 + "endpoint": "/v2/models/ensemble/generate_stream"
108 + },
109 + "vllm_server_parameters": {
110 + "disable_log_stats": "",
111 + "disable_log_requests": ""
112 + },
113 + "vllm_client_parameters": {
114 + }
115 + }
116 +]
\ No newline at end of file
README.md
+2
@@ -52,6 +52,8 @@ vLLM is fast with:
52 - Quantization: [GPTQ](https://arxiv.org/abs/2210.17323), [AWQ](https://arxiv.org/abs/2306.00978), [SqueezeLLM](https://arxiv.org/abs/2306.07629), FP8 KV Cache
53 - Optimized CUDA kernels
54
55 +**Performance benchmark**: We include a [performance benchmark](https://buildkite.com/vllm/performance-benchmark/builds/3924) that compares the performance of vllm against other LLM serving engines ([TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM), [text-generation-inference](https://github.com/huggingface/text-generation-inference) and [lmdeploy](https://github.com/InternLM/lmdeploy)).
56 +
57 vLLM is flexible and easy to use with:
58
59 - Seamless integration with popular Hugging Face models
benchmarks/benchmark_serving.py
+9
@@ -60,12 +60,15 @@ class BenchmarkMetrics:
60 output_throughput: float
61 mean_ttft_ms: float
62 median_ttft_ms: float
63 + std_ttft_ms: float
64 p99_ttft_ms: float
65 mean_tpot_ms: float
66 median_tpot_ms: float
67 + std_tpot_ms: float
68 p99_tpot_ms: float
69 mean_itl_ms: float
70 median_itl_ms: float
71 + std_itl_ms: float
72 p99_itl_ms: float
73
74
@@ -274,12 +277,15 @@ def calculate_metrics(
277 mean_ttft_ms=np.mean(ttfts or 0) *
278 1000, # ttfts is empty if streaming is not supported by backend
279 median_ttft_ms=np.median(ttfts or 0) * 1000,
280 + std_ttft_ms=np.std(ttfts or 0) * 1000,
281 p99_ttft_ms=np.percentile(ttfts or 0, 99) * 1000,
282 mean_tpot_ms=np.mean(tpots or 0) * 1000,
283 median_tpot_ms=np.median(tpots or 0) * 1000,
284 + std_tpot_ms=np.std(tpots or 0) * 1000,
285 p99_tpot_ms=np.percentile(tpots or 0, 99) * 1000,
286 mean_itl_ms=np.mean(itls or 0) * 1000,
287 median_itl_ms=np.median(itls or 0) * 1000,
288 + std_itl_ms=np.std(itls or 0) * 1000,
289 p99_itl_ms=np.percentile(itls or 0, 99) * 1000,
290 )
291
@@ -396,12 +402,15 @@ async def benchmark(
402 "output_throughput": metrics.output_throughput,
403 "mean_ttft_ms": metrics.mean_ttft_ms,
404 "median_ttft_ms": metrics.median_ttft_ms,
405 + "std_ttft_ms": metrics.std_ttft_ms,
406 "p99_ttft_ms": metrics.p99_ttft_ms,
407 "mean_tpot_ms": metrics.mean_tpot_ms,
408 "median_tpot_ms": metrics.median_tpot_ms,
409 + "std_tpot_ms": metrics.std_tpot_ms,
410 "p99_tpot_ms": metrics.p99_tpot_ms,
411 "mean_itl_ms": metrics.mean_itl_ms,
412 "median_itl_ms": metrics.median_itl_ms,
413 + "std_itl_ms": metrics.std_itl_ms,
414 "p99_itl_ms": metrics.p99_itl_ms,
415 "input_lens": [output.prompt_len for output in outputs],
416 "output_lens": actual_output_lens,