A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 10:45 UTC
[CI] Update performance benchmark: upgrade trt-llm to r24.07, and add SGLang (#7412)
Kuntai Du committed
Oct 4, 2024 at 14:01 UTC
fbb74420e7018bf0cc1bc81e6fd71a2392347227
18 files changed
+1152
-1276
.buildkite/nightly-benchmarks/nightly-annotation.md
new
+28
@@ -0,0 +1,28 @@
1
+
2
+## Description
3
+
4
+This file contains the downloading link for benchmarking results.
5
+
6
+- [benchmarking pipeline](artifact://nightly-pipeline.yaml)
7
+- [benchmarking results](artifact://results.zip)
8
+- [benchmarking code](artifact://nightly-benchmarks.zip)
9
+
10
+Please download the visualization scripts in the post
11
+
12
+
13
+## Results reproduction
14
+
15
+- Find the docker we use in `benchmarking pipeline`
16
+- Deploy the docker, and inside the docker:
17
+ - Download `nightly-benchmarks.zip`.
18
+ - In the same folder, run the following code
19
+```
20
+export HF_TOKEN=<your HF token>
21
+apt update
22
+apt install -y git
23
+unzip nightly-benchmarks.zip
24
+VLLM_SOURCE_CODE_LOC=./ bash .buildkite/nightly-benchmarks/scripts/run-nightly-benchmarks.sh
25
+```
26
+
27
+And the results will be inside `./benchmarks/results`.
28
+
.buildkite/nightly-benchmarks/nightly-descriptions.md
+36
-42
@@ -1,45 +1,39 @@
1
2
# Nightly benchmark
3
4
-The main goal of this benchmarking is two-fold:
5
-- Performance clarity: Provide clarity on which one (vllm, tensorrt-llm, lmdeploy and tgi) leads in performance in what workload.
6
-- Reproducible: one can run the exact same set of benchmarking commands inside the exact same docker by following reproducing instructions in [reproduce.md]().
7
-
8
-
9
-## Docker images
10
-
11
-We benchmark vllm, tensorrt-llm, lmdeploy and tgi using the following docker images:
12
-- vllm/vllm-openai:v0.5.0.post1
13
-- nvcr.io/nvidia/tritonserver:24.04-trtllm-python-py3
14
-- openmmlab/lmdeploy:v0.5.0
15
-- ghcr.io/huggingface/text-generation-inference:2.1
16
-
17
-<!-- Please check <a href="artifact://workspace/build/buildkite/vllm/performance-benchmark/.buildkite/nightly-benchmarks/nightly-pipeline.yaml">nightly-pipeline.yaml</a> artifact for more details on how we deploy the docker images. -->
18
-
19
-
20
-## Hardware
21
-
22
-One AWS node with 8x NVIDIA A100 GPUs.
23
-
24
-
25
-## Workload description
26
-
27
-We benchmark vllm, tensorrt-llm, lmdeploy and tgi using the following workload:
28
-
29
-- Input length: randomly sample 500 prompts from ShareGPT dataset (with fixed random seed).
30
-- Output length: the corresponding output length of these 500 prompts.
31
-- Models: llama-3 8B, llama-3 70B, mixtral 8x7B.
32
-- Average QPS (query per second): 4 for the small model (llama-3 8B) and 2 for other two models. For each QPS, the arrival time of each query is determined using a random Poisson process (with fixed random seed).
33
-- Evaluation metrics: Throughput (higher the better), TTFT (time to the first token, lower the better), ITL (inter-token latency, lower the better).
34
-
35
-<!-- Check <a href="artifact://workspace/build/buildkite/vllm/performance-benchmark/.buildkite/nightly-benchmarks/tests/nightly-tests.json">nightly-tests.json</a> artifact for more details. -->
36
-
37
-## Plots
38
-
39
-In the following plots, the dot shows the mean and the error bar shows the standard error of the mean. Value 0 means that the corresponding benchmark crashed.
40
-
41
-<img src="artifact://nightly_results.png" alt="Benchmarking results" height=250 >
42
-
43
-## Results
44
-
45
-{nightly_results_benchmarking_table}
4
+This benchmark aims to:
5
+- Provide performance clarity: Provide clarity on which one (vllm, tensorrt-llm, lmdeploy and SGLang) leads in performance in what workload.
6
+- Be reproducible: one can run the exact same set of benchmarking commands inside the exact same docker by following reproducing instructions.
7
+
8
+Latest results: [results link](https://blog.vllm.ai/2024/09/05/perf-update.html), scroll to the end.
9
+
10
+Latest reproduction guilde: [github issue link](https://github.com/vllm-project/vllm/issues/8176)
11
+
12
+
13
+## Setup
14
+
15
+- Docker images:
16
+ - vLLM: `vllm/vllm-openai:v0.6.2`
17
+ - SGLang: `lmsysorg/sglang:v0.3.2-cu121`
18
+ - LMDeploy: `openmmlab/lmdeploy:v0.6.1-cu12`
19
+ - TensorRT-LLM: `nvcr.io/nvidia/tritonserver:24.07-trtllm-python-py3`
20
+ - *NOTE: we uses r24.07 as the current implementation only works for this version. We are going to bump this up.*
21
+ - Check [nightly-pipeline.yaml](nightly-pipeline.yaml) for the concrete docker images, specs and commands we use for the benchmark.
22
+- Hardware
23
+ - 8x Nvidia A100 GPUs
24
+- Workload:
25
+ - Dataset
26
+ - ShareGPT dataset
27
+ - Prefill-heavy dataset (in average 462 input tokens, 16 tokens as output)
28
+ - Decode-heavy dataset (in average 462 input tokens, 256 output tokens)
29
+ - Check [nightly-tests.json](tests/nightly-tests.json) for the concrete configuration of datasets we use.
30
+ - Models: llama-3 8B, llama-3 70B.
31
+ - We do not use llama 3.1 as it is incompatible with trt-llm r24.07. ([issue](https://github.com/NVIDIA/TensorRT-LLM/issues/2105)).
32
+ - Average QPS (query per second): 2, 4, 8, 16, 32 and inf.
33
+ - Queries are randomly sampled, and arrival patterns are determined via Poisson process, but all with fixed random seed.
34
+ - Evaluation metrics: Throughput (higher the better), TTFT (time to the first token, lower the better), ITL (inter-token latency, lower the better).
35
+
36
+# Known issues
37
+
38
+- TRT-LLM crashes with Llama 3.1 8B [issue](https://github.com/NVIDIA/TensorRT-LLM/issues/2105).
39
+- TGI does not support `ignore-eos` flag.
\ No newline at end of file
.buildkite/nightly-benchmarks/nightly-pipeline.yaml
+87
-11
@@ -13,7 +13,7 @@ common_pod_spec: &common_pod_spec
13
14
common_container_settings: &common_container_settings
15
command:
16
- - bash .buildkite/nightly-benchmarks/run-nightly-suite.sh
16
+ - bash .buildkite/nightly-benchmarks/scripts/run-nightly-benchmarks.sh
17
resources:
18
limits:
19
nvidia.com/gpu: 8
@@ -37,7 +37,10 @@ common_container_settings: &common_container_settings
37
38
steps:
39
- block: ":rocket: Ready for comparing vllm against alternatives? This will take 4 hours."
40
- - label: "A100 trt benchmark"
40
+
41
+
42
+
43
+ - label: "A100 vllm step 10"
44
priority: 100
45
agents:
46
queue: A100
@@ -46,7 +49,21 @@ steps:
49
podSpec:
50
<<: *common_pod_spec
51
containers:
49
- - image: nvcr.io/nvidia/tritonserver:24.04-trtllm-python-py3
52
+ - image: vllm/vllm-openai:v0.6.2
53
+ <<: *common_container_settings
54
+
55
+
56
+
57
+ - label: "A100 sglang benchmark"
58
+ priority: 100
59
+ agents:
60
+ queue: A100
61
+ plugins:
62
+ - kubernetes:
63
+ podSpec:
64
+ <<: *common_pod_spec
65
+ containers:
66
+ - image: lmsysorg/sglang:v0.3.2-cu121
67
<<: *common_container_settings
68
69
- label: "A100 lmdeploy benchmark"
@@ -58,11 +75,13 @@ steps:
75
podSpec:
76
<<: *common_pod_spec
77
containers:
61
- - image: openmmlab/lmdeploy:v0.5.0
78
+ - image: openmmlab/lmdeploy:v0.6.1-cu12
79
<<: *common_container_settings
63
-
80
65
- - label: "A100 vllm benchmark"
81
+
82
+
83
+
84
+ - label: "A100 trt llama-8B"
85
priority: 100
86
agents:
87
queue: A100
@@ -71,10 +90,25 @@ steps:
90
podSpec:
91
<<: *common_pod_spec
92
containers:
74
- - image: vllm/vllm-openai:latest
93
+ - image: nvcr.io/nvidia/tritonserver:24.07-trtllm-python-py3
94
<<: *common_container_settings
95
+ env:
96
+ - name: VLLM_USAGE_SOURCE
97
+ value: ci-test
98
+ - name: HF_HOME
99
+ value: /root/.cache/huggingface
100
+ - name: VLLM_SOURCE_CODE_LOC
101
+ value: /workspace/build/buildkite/vllm/performance-benchmark
102
+ - name: HF_TOKEN
103
+ valueFrom:
104
+ secretKeyRef:
105
+ name: hf-token-secret
106
+ key: token
107
+ - name: TEST_SELECTOR
108
+ value: "llama8B"
109
77
- - label: "A100 tgi benchmark"
110
+
111
+ - label: "A100 trt llama-70B"
112
priority: 100
113
agents:
114
queue: A100
@@ -83,12 +117,54 @@ steps:
117
podSpec:
118
<<: *common_pod_spec
119
containers:
86
- - image: ghcr.io/huggingface/text-generation-inference:2.1
120
+ - image: nvcr.io/nvidia/tritonserver:24.07-trtllm-python-py3
121
<<: *common_container_settings
122
+ env:
123
+ - name: VLLM_USAGE_SOURCE
124
+ value: ci-test
125
+ - name: HF_HOME
126
+ value: /root/.cache/huggingface
127
+ - name: VLLM_SOURCE_CODE_LOC
128
+ value: /workspace/build/buildkite/vllm/performance-benchmark
129
+ - name: HF_TOKEN
130
+ valueFrom:
131
+ secretKeyRef:
132
+ name: hf-token-secret
133
+ key: token
134
+ - name: TEST_SELECTOR
135
+ value: "llama70B"
136
+
137
+
138
+ # FIXME(Kuntai): uncomment this after NVIDIA gives us their test docker image
139
+ # - label: "A100 trt benchmark"
140
+ # priority: 100
141
+ # agents:
142
+ # queue: A100
143
+ # plugins:
144
+ # - kubernetes:
145
+ # podSpec:
146
+ # <<: *common_pod_spec
147
+ # containers:
148
+ # - image: nvcr.io/nvidia/tritonserver:24.07-trtllm-python-py3
149
+ # <<: *common_container_settings
150
+
151
+
152
+ # FIXME(Kuntai): uncomment this after TGI supports `--ignore-eos`.
153
+ # - label: "A100 tgi benchmark"
154
+ # priority: 100
155
+ # agents:
156
+ # queue: A100
157
+ # plugins:
158
+ # - kubernetes:
159
+ # podSpec:
160
+ # <<: *common_pod_spec
161
+ # containers:
162
+ # - image: ghcr.io/huggingface/text-generation-inference:2.2.0
163
+ # <<: *common_container_settings
164
165
- wait
166
91
- - label: "Plot"
167
+ - label: "Collect the results"
168
priority: 100
169
agents:
170
queue: A100
@@ -117,4 +193,4 @@ steps:
193
name: hf-token-secret
194
key: token
195
120
- - wait
\ No newline at end of file
196
+ - block: ":rocket: check the results!"
\ No newline at end of file
.buildkite/nightly-benchmarks/run-nightly-suite.sh
deleted
-76
@@ -1,76 +0,0 @@
1
-#!/bin/bash
2
-
3
-set -o pipefail
4
-set -x
5
-
6
-check_gpus() {
7
- # check the number of GPUs and GPU type.
8
- declare -g gpu_count=$(nvidia-smi --list-gpus | wc -l)
9
- if [[ $gpu_count -gt 0 ]]; then
10
- echo "GPU found."
11
- else
12
- echo "Need at least 1 GPU to run benchmarking."
13
- exit 1
14
- fi
15
- declare -g gpu_type=$(echo $(nvidia-smi --query-gpu=name --format=csv,noheader) | awk '{print $2}')
16
- echo "GPU type is $gpu_type"
17
-}
18
-
19
-check_hf_token() {
20
- # check if HF_TOKEN is available and valid
21
- if [[ -z "$HF_TOKEN" ]]; then
22
- echo "Error: HF_TOKEN is not set."
23
- exit 1
24
- elif [[ ! "$HF_TOKEN" =~ ^hf_ ]]; then
25
- echo "Error: HF_TOKEN does not start with 'hf_'."
26
- exit 1
27
- else
28
- echo "HF_TOKEN is set and valid."
29
- fi
30
-}
31
-
32
-main() {
33
-
34
- check_gpus
35
- check_hf_token
36
-
37
- df -h
38
-
39
- (which wget && which curl) || (apt-get update && apt-get install -y wget curl)
40
- (which jq) || (apt-get update && apt-get -y install jq)
41
-
42
- cd $VLLM_SOURCE_CODE_LOC/benchmarks
43
- wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
44
-
45
-
46
- # run lmdeploy
47
- if which lmdeploy >/dev/null; then
48
- echo "lmdeploy is available, redirect to run-lmdeploy-nightly.sh"
49
- bash ../.buildkite/nightly-benchmarks/scripts/run-lmdeploy-nightly.sh
50
- exit 0
51
- fi
52
-
53
- # run tgi
54
- if [ -e /tgi-entrypoint.sh ]; then
55
- echo "tgi is available, redirect to run-tgi-nightly.sh"
56
- bash ../.buildkite/nightly-benchmarks/scripts/run-tgi-nightly.sh
57
- exit 0
58
- fi
59
-
60
- # run trt
61
- if which trtllm-build >/dev/null; then
62
- echo "trtllm is available, redirect to run-trt-nightly.sh"
63
- bash ../.buildkite/nightly-benchmarks/scripts/run-trt-nightly.sh
64
- exit 0
65
- fi
66
-
67
- # run vllm
68
- if [ -e /vllm-workspace ]; then
69
- echo "vllm is available, redirect to run-vllm-nightly.sh"
70
- bash ../.buildkite/nightly-benchmarks/scripts/run-vllm-nightly.sh
71
- exit 0
72
- fi
73
-
74
-}
75
-
76
-main "$@"
\ No newline at end of file
.buildkite/nightly-benchmarks/scripts/generate-nightly-markdown.py
new
+95
@@ -0,0 +1,95 @@
1
+import argparse
2
+import json
3
+from pathlib import Path
4
+
5
+import numpy as np
6
+import pandas as pd
7
+from tabulate import tabulate
8
+
9
+
10
+def parse_arguments():
11
+ parser = argparse.ArgumentParser(
12
+ description=
13
+ 'Parse command line arguments for summary-nightly-results script.')
14
+ parser.add_argument('--results-folder',
15
+ type=str,
16
+ required=True,
17
+ help='The folder where the results are stored.')
18
+ parser.add_argument('--description',
19
+ type=str,
20
+ required=True,
21
+ help='Description of the results.')
22
+
23
+ args = parser.parse_args()
24
+ return args
25
+
26
+
27
+def get_perf(df, method, model, metric):
28
+
29
+ means = []
30
+
31
+ for qps in [2, 4, 8, 16, "inf"]:
32
+ target = df['Test name'].str.contains(model)
33
+ target = target & df['Engine'].str.contains(method)
34
+ target = target & df['Test name'].str.contains("qps_" + str(qps))
35
+ filtered_df = df[target]
36
+
37
+ if filtered_df.empty:
38
+ means.append(0.)
39
+ else:
40
+ means.append(filtered_df[metric].values[0])
41
+
42
+ return np.array(means)
43
+
44
+
45
+def get_perf_w_std(df, method, model, metric):
46
+
47
+ if metric in ["TTFT", "ITL"]:
48
+ mean = get_perf(df, method, model, "Mean " + metric + " (ms)")
49
+ mean = mean.tolist()
50
+ std = get_perf(df, method, model, "Std " + metric + " (ms)")
51
+ if std.mean() == 0:
52
+ std = None
53
+ success = get_perf(df, method, model, "Successful req.")
54
+ if std is not None:
55
+ std = std / np.sqrt(success)
56
+ std = std.tolist()
57
+
58
+ else:
59
+ assert metric == "Tput"
60
+ mean = get_perf(df, method, model, "Input Tput (tok/s)") + get_perf(
61
+ df, method, model, "Output Tput (tok/s)")
62
+ mean = mean.tolist()
63
+ std = None
64
+
65
+ return mean, std
66
+
67
+
68
+def main(args):
69
+ results_folder = Path(args.results_folder)
70
+
71
+ results = []
72
+
73
+ # collect results
74
+ for test_file in results_folder.glob("*_nightly_results.json"):
75
+ with open(test_file, "r") as f:
76
+ results = results + json.loads(f.read())
77
+
78
+ # generate markdown table
79
+ df = pd.DataFrame.from_dict(results)
80
+
81
+ md_table = tabulate(df, headers='keys', tablefmt='pipe', showindex=False)
82
+
83
+ with open(args.description, "r") as f:
84
+ description = f.read()
85
+
86
+ description = description.format(
87
+ nightly_results_benchmarking_table=md_table)
88
+
89
+ with open("nightly_results.md", "w") as f:
90
+ f.write(description)
91
+
92
+
93
+if __name__ == '__main__':
94
+ args = parse_arguments()
95
+ main(args)
.buildkite/nightly-benchmarks/scripts/launch-server.sh
new
+241
@@ -0,0 +1,241 @@
1
+#!/bin/bash
2
+
3
+# Currently FP8 benchmark is NOT enabled.
4
+
5
+set -x
6
+server_params=$1
7
+common_params=$2
8
+
9
+json2args() {
10
+ # transforms the JSON string to command line args, and '_' is replaced to '-'
11
+ # example:
12
+ # input: { "model": "meta-llama/Llama-2-7b-chat-hf", "tensor_parallel_size": 1 }
13
+ # output: --model meta-llama/Llama-2-7b-chat-hf --tensor-parallel-size 1
14
+ local json_string=$1
15
+ local args=$(
16
+ echo "$json_string" | jq -r '
17
+ to_entries |
18
+ map("--" + (.key | gsub("_"; "-")) + " " + (.value | tostring)) |
19
+ join(" ")
20
+ '
21
+ )
22
+ echo "$args"
23
+}
24
+
25
+launch_trt_server() {
26
+
27
+ model_path=$(echo "$common_params" | jq -r '.model')
28
+ model_name="${model_path#*/}"
29
+ model_type=$(echo "$server_params" | jq -r '.model_type')
30
+ model_dtype=$(echo "$server_params" | jq -r '.model_dtype')
31
+ model_tp_size=$(echo "$common_params" | jq -r '.tp')
32
+ max_batch_size=$(echo "$server_params" | jq -r '.max_batch_size')
33
+ max_input_len=$(echo "$server_params" | jq -r '.max_input_len')
34
+ max_seq_len=$(echo "$server_params" | jq -r '.max_seq_len')
35
+ max_num_tokens=$(echo "$server_params" | jq -r '.max_num_tokens')
36
+ trt_llm_version=$(echo "$server_params" | jq -r '.trt_llm_version')
37
+
38
+ # create model caching directory
39
+ cd ~
40
+ rm -rf models
41
+ mkdir -p models
42
+ cd models
43
+ models_dir=$(pwd)
44
+ trt_model_path=${models_dir}/${model_name}-trt-ckpt
45
+ trt_engine_path=${models_dir}/${model_name}-trt-engine
46
+
47
+ # clone tensorrt backend
48
+ cd /
49
+ rm -rf tensorrtllm_backend
50
+ git clone https://github.com/triton-inference-server/tensorrtllm_backend.git
51
+ git lfs install
52
+ cd tensorrtllm_backend
53
+ git checkout $trt_llm_version
54
+ tensorrtllm_backend_dir=$(pwd)
55
+ git submodule update --init --recursive
56
+
57
+ # build trtllm engine
58
+ cd /tensorrtllm_backend
59
+ cd ./tensorrt_llm/examples/${model_type}
60
+ python3 convert_checkpoint.py \
61
+ --model_dir ${model_path} \
62
+ --dtype ${model_dtype} \
63
+ --tp_size ${model_tp_size} \
64
+ --output_dir ${trt_model_path}
65
+ trtllm-build \
66
+ --checkpoint_dir ${trt_model_path} \
67
+ --use_fused_mlp \
68
+ --reduce_fusion disable \
69
+ --workers 8 \
70
+ --gpt_attention_plugin ${model_dtype} \
71
+ --gemm_plugin ${model_dtype} \
72
+ --tp_size ${model_tp_size} \
73
+ --max_batch_size ${max_batch_size} \
74
+ --max_input_len ${max_input_len} \
75
+ --max_seq_len ${max_seq_len} \
76
+ --max_num_tokens ${max_num_tokens} \
77
+ --output_dir ${trt_engine_path}
78
+
79
+ # handle triton protobuf files and launch triton server
80
+ cd /tensorrtllm_backend
81
+ mkdir triton_model_repo
82
+ cp -r all_models/inflight_batcher_llm/* triton_model_repo/
83
+ cd triton_model_repo
84
+ rm -rf ./tensorrt_llm/1/*
85
+ cp -r ${trt_engine_path}/* ./tensorrt_llm/1
86
+ python3 ../tools/fill_template.py -i tensorrt_llm/config.pbtxt triton_backend:tensorrtllm,engine_dir:/tensorrtllm_backend/triton_model_repo/tensorrt_llm/1,decoupled_mode:true,batching_strategy:inflight_fused_batching,batch_scheduler_policy:guaranteed_no_evict,exclude_input_in_output:true,triton_max_batch_size:2048,max_queue_delay_microseconds:0,max_beam_width:1,max_queue_size:2048,enable_kv_cache_reuse:false
87
+ python3 ../tools/fill_template.py -i preprocessing/config.pbtxt triton_max_batch_size:2048,tokenizer_dir:$model_path,preprocessing_instance_count:5
88
+ python3 ../tools/fill_template.py -i postprocessing/config.pbtxt triton_max_batch_size:2048,tokenizer_dir:$model_path,postprocessing_instance_count:5,skip_special_tokens:false
89
+ python3 ../tools/fill_template.py -i ensemble/config.pbtxt triton_max_batch_size:$max_batch_size
90
+ python3 ../tools/fill_template.py -i tensorrt_llm_bls/config.pbtxt triton_max_batch_size:$max_batch_size,decoupled_mode:true,accumulate_tokens:"False",bls_instance_count:1
91
+ cd /tensorrtllm_backend
92
+ python3 scripts/launch_triton_server.py \
93
+ --world_size=${model_tp_size} \
94
+ --model_repo=/tensorrtllm_backend/triton_model_repo &
95
+
96
+}
97
+
98
+launch_tgi_server() {
99
+ model=$(echo "$common_params" | jq -r '.model')
100
+ tp=$(echo "$common_params" | jq -r '.tp')
101
+ dataset_name=$(echo "$common_params" | jq -r '.dataset_name')
102
+ dataset_path=$(echo "$common_params" | jq -r '.dataset_path')
103
+ port=$(echo "$common_params" | jq -r '.port')
104
+ num_prompts=$(echo "$common_params" | jq -r '.num_prompts')
105
+ server_args=$(json2args "$server_params")
106
+
107
+ if echo "$common_params" | jq -e 'has("fp8")' >/dev/null; then
108
+ echo "Key 'fp8' exists in common params."
109
+ server_command="/tgi-entrypoint.sh \
110
+ --model-id $model \
111
+ --num-shard $tp \
112
+ --port $port \
113
+ --quantize fp8 \
114
+ $server_args"
115
+ else
116
+ echo "Key 'fp8' does not exist in common params."
117
+ server_command="/tgi-entrypoint.sh \
118
+ --model-id $model \
119
+ --num-shard $tp \
120
+ --port $port \
121
+ $server_args"
122
+ fi
123
+
124
+ echo "Server command: $server_command"
125
+ eval "$server_command" &
126
+
127
+}
128
+
129
+launch_lmdeploy_server() {
130
+ model=$(echo "$common_params" | jq -r '.model')
131
+ tp=$(echo "$common_params" | jq -r '.tp')
132
+ dataset_name=$(echo "$common_params" | jq -r '.dataset_name')
133
+ dataset_path=$(echo "$common_params" | jq -r '.dataset_path')
134
+ port=$(echo "$common_params" | jq -r '.port')
135
+ num_prompts=$(echo "$common_params" | jq -r '.num_prompts')
136
+ server_args=$(json2args "$server_params")
137
+
138
+ server_command="lmdeploy serve api_server $model \
139
+ --tp $tp \
140
+ --server-port $port \
141
+ $server_args"
142
+
143
+ # run the server
144
+ echo "Server command: $server_command"
145
+ bash -c "$server_command" &
146
+}
147
+
148
+launch_sglang_server() {
149
+
150
+ model=$(echo "$common_params" | jq -r '.model')
151
+ tp=$(echo "$common_params" | jq -r '.tp')
152
+ dataset_name=$(echo "$common_params" | jq -r '.dataset_name')
153
+ dataset_path=$(echo "$common_params" | jq -r '.dataset_path')
154
+ port=$(echo "$common_params" | jq -r '.port')
155
+ num_prompts=$(echo "$common_params" | jq -r '.num_prompts')
156
+ server_args=$(json2args "$server_params")
157
+
158
+ if echo "$common_params" | jq -e 'has("fp8")' >/dev/null; then
159
+ echo "Key 'fp8' exists in common params. Use neuralmagic fp8 model for convenience."
160
+ model=$(echo "$common_params" | jq -r '.neuralmagic_quantized_model')
161
+ server_command="python3 \
162
+ -m sglang.launch_server \
163
+ --tp $tp \
164
+ --model-path $model \
165
+ --port $port \
166
+ $server_args"
167
+ else
168
+ echo "Key 'fp8' does not exist in common params."
169
+ server_command="python3 \
170
+ -m sglang.launch_server \
171
+ --tp $tp \
172
+ --model-path $model \
173
+ --port $port \
174
+ $server_args"
175
+ fi
176
+
177
+ # run the server
178
+ echo "Server command: $server_command"
179
+ eval "$server_command" &
180
+}
181
+
182
+launch_vllm_server() {
183
+
184
+ export VLLM_HOST_IP=$(hostname -I | awk '{print $1}')
185
+
186
+ model=$(echo "$common_params" | jq -r '.model')
187
+ tp=$(echo "$common_params" | jq -r '.tp')
188
+ dataset_name=$(echo "$common_params" | jq -r '.dataset_name')
189
+ dataset_path=$(echo "$common_params" | jq -r '.dataset_path')
190
+ port=$(echo "$common_params" | jq -r '.port')
191
+ num_prompts=$(echo "$common_params" | jq -r '.num_prompts')
192
+ server_args=$(json2args "$server_params")
193
+
194
+ if echo "$common_params" | jq -e 'has("fp8")' >/dev/null; then
195
+ echo "Key 'fp8' exists in common params. Use neuralmagic fp8 model for convenience."
196
+ model=$(echo "$common_params" | jq -r '.neuralmagic_quantized_model')
197
+ server_command="python3 \
198
+ -m vllm.entrypoints.openai.api_server \
199
+ -tp $tp \
200
+ --model $model \
201
+ --port $port \
202
+ $server_args"
203
+ else
204
+ echo "Key 'fp8' does not exist in common params."
205
+ server_command="python3 \
206
+ -m vllm.entrypoints.openai.api_server \
207
+ -tp $tp \
208
+ --model $model \
209
+ --port $port \
210
+ $server_args"
211
+ fi
212
+
213
+ # run the server
214
+ echo "Server command: $server_command"
215
+ eval "$server_command" &
216
+}
217
+
218
+main() {
219
+
220
+ if [[ $CURRENT_LLM_SERVING_ENGINE == "trt" ]]; then
221
+ launch_trt_server
222
+ fi
223
+
224
+ if [[ $CURRENT_LLM_SERVING_ENGINE == "tgi" ]]; then
225
+ launch_tgi_server
226
+ fi
227
+
228
+ if [[ $CURRENT_LLM_SERVING_ENGINE == "lmdeploy" ]]; then
229
+ launch_lmdeploy_server
230
+ fi
231
+
232
+ if [[ $CURRENT_LLM_SERVING_ENGINE == "sglang" ]]; then
233
+ launch_sglang_server
234
+ fi
235
+
236
+ if [[ "$CURRENT_LLM_SERVING_ENGINE" == *"vllm"* ]]; then
237
+ launch_vllm_server
238
+ fi
239
+}
240
+
241
+main
.buildkite/nightly-benchmarks/scripts/launch-trt-server.sh
deleted
-102
@@ -1,102 +0,0 @@
1
-#!/bin/bash
2
-
3
-
4
-server_params=$1
5
-common_params=$2
6
-
7
-
8
-
9
-model_path=$(echo "$common_params" | jq -r '.model')
10
-model_name="${model_path#*/}"
11
-model_type=$(echo "$server_params" | jq -r '.model_type')
12
-model_dtype=$(echo "$server_params" | jq -r '.model_dtype')
13
-model_tp_size=$(echo "$common_params" | jq -r '.tp')
14
-max_batch_size=$(echo "$server_params" | jq -r '.max_batch_size')
15
-max_input_len=$(echo "$server_params" | jq -r '.max_input_len')
16
-max_output_len=$(echo "$server_params" | jq -r '.max_output_len')
17
-trt_llm_version=$(echo "$server_params" | jq -r '.trt_llm_version')
18
-
19
-cd ~
20
-rm -rf models
21
-mkdir -p models
22
-cd models
23
-models_dir=$(pwd)
24
-trt_model_path=${models_dir}/${model_name}-trt-ckpt
25
-trt_engine_path=${models_dir}/${model_name}-trt-engine
26
-
27
-cd ~
28
-rm -rf tensorrt-demo
29
-git clone https://github.com/neuralmagic/tensorrt-demo.git
30
-cd tensorrt-demo
31
-tensorrt_demo_dir=$(pwd)
32
-
33
-# make sure the parameter inside tensorrt_demo is consistent to envvar
34
-sed -i.bak "/key: \"tokenizer_dir\"/,/string_value:/s|string_value: \".*\"|string_value: \"$model_path\"|" ./triton_model_repo/postprocessing/config.pbtxt
35
-sed -i.bak "/key: \"tokenizer_dir\"/,/string_value:/s|string_value: \".*\"|string_value: \"$model_path\"|" ./triton_model_repo/preprocessing/config.pbtxt
36
-sed -i.bak "s|\(max_batch_size:\s*\)[0-9]*|\1$max_batch_size|g" ./triton_model_repo/ensemble/config.pbtxt
37
-sed -i.bak "s|\(max_batch_size:\s*\)[0-9]*|\1$max_batch_size|g" ./triton_model_repo/preprocessing/config.pbtxt
38
-sed -i.bak "s|\(max_batch_size:\s*\)[0-9]*|\1$max_batch_size|g" ./triton_model_repo/postprocessing/config.pbtxt
39
-sed -i.bak "s|\(max_batch_size:\s*\)[0-9]*|\1$max_batch_size|g" ./triton_model_repo/tensorrt_llm_bls/config.pbtxt
40
-
41
-
42
-cd /
43
-rm -rf tensorrtllm_backend
44
-git clone https://github.com/triton-inference-server/tensorrtllm_backend.git
45
-git lfs install
46
-cd tensorrtllm_backend
47
-git checkout $trt_llm_version
48
-tensorrtllm_backend_dir=$(pwd)
49
-git submodule update --init --recursive
50
-cp -r ${tensorrt_demo_dir}/triton_model_repo ${tensorrtllm_backend_dir}/
51
-
52
-cd /tensorrtllm_backend
53
-cd ./tensorrt_llm/examples/${model_type}
54
-
55
-
56
-if echo "$common_params" | jq -e 'has("fp8")' > /dev/null; then
57
-
58
- echo "Key 'fp8' exists in common params. Use quantize.py instead of convert_checkpoint.py"
59
- echo "Reference: https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/llama/README.md"
60
- python ../quantization/quantize.py \
61
- --model_dir ${model_path} \
62
- --dtype ${model_dtype} \
63
- --tp_size ${model_tp_size} \
64
- --output_dir ${trt_model_path} \
65
- --qformat fp8 \
66
- --kv_cache_dtype fp8 \
67
- --calib_size 2
68
-
69
-else
70
-
71
- echo "Key 'fp8' does not exist in common params. Use convert_checkpoint.py"
72
- python3 convert_checkpoint.py \
73
- --model_dir ${model_path} \
74
- --dtype ${model_dtype} \
75
- --tp_size ${model_tp_size} \
76
- --output_dir ${trt_model_path}
77
-
78
-fi
79
-
80
-
81
-
82
-trtllm-build \
83
---checkpoint_dir=${trt_model_path} \
84
---gpt_attention_plugin=${model_dtype} \
85
---gemm_plugin=${model_dtype} \
86
---remove_input_padding=enable \
87
---paged_kv_cache=enable \
88
---tp_size=${model_tp_size} \
89
---max_batch_size=${max_batch_size} \
90
---max_input_len=${max_input_len} \
91
---max_output_len=${max_output_len} \
92
---max_num_tokens=${max_output_len} \
93
---opt_num_tokens=${max_output_len} \
94
---output_dir=${trt_engine_path}
95
-
96
-cd /tensorrtllm_backend/triton_model_repo
97
-rm -rf ./tensorrt_llm/1/*
98
-cp -r ${trt_engine_path}/* ./tensorrt_llm/1
99
-cd /tensorrtllm_backend
100
-python3 scripts/launch_triton_server.py \
101
---world_size=${model_tp_size} \
102
---model_repo=/tensorrtllm_backend/triton_model_repo &
\ No newline at end of file
.buildkite/nightly-benchmarks/scripts/nightly-annotate.sh
+48
-10
@@ -8,6 +8,7 @@ main() {
8
9
(which wget && which curl) || (apt-get update && apt-get install -y wget curl)
10
(which jq) || (apt-get update && apt-get -y install jq)
11
+ (which zip) || (apt-get install -y zip)
12
13
if [ ! -f /workspace/buildkite-agent ]; then
14
echo "buildkite-agent binary not found. Skip plotting the results."
@@ -24,17 +25,54 @@ main() {
25
ls
26
ls results/
27
27
- # generate figures
28
- python3 -m pip install tabulate pandas matplotlib
29
- python3 $VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/scripts/plot-nightly-results.py \
30
- --description $description \
31
- --results-folder results/
28
+ # upload benchmark results
29
+ zip -r results.zip results/
30
+ /workspace/buildkite-agent artifact upload "results.zip"
31
+
32
+ # upload benchmarking scripts
33
+ cd $VLLM_SOURCE_CODE_LOC/
34
+ zip -r nightly-benchmarks.zip .buildkite/ benchmarks/
35
+ /workspace/buildkite-agent artifact upload "nightly-benchmarks.zip"
36
+
37
+ cd $VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/
38
+ # upload benchmarking pipeline
39
+ /workspace/buildkite-agent artifact upload "nightly-pipeline.yaml"
40
+
41
+ cd $VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/
42
+ /workspace/buildkite-agent annotate --style "success" --context "nightly-benchmarks-results" --append < nightly-annotation.md
43
+
44
+
45
+
46
+ # The figures should be genereated by a separate process outside the CI/CD pipeline
47
+
48
+ # # generate figures
49
+ # python3 -m pip install tabulate pandas matplotlib
50
+
51
+ # python3 $VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/scripts/generate-nightly-markdown.py \
52
+ # --description $description \
53
+ # --results-folder results/
54
+
55
+
56
+ # python3 $VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/scripts/plot-nightly-results.py \
57
+ # --description $description \
58
+ # --results-folder results/ \
59
+ # --dataset sharegpt
60
+
61
+ # python3 $VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/scripts/plot-nightly-results.py \
62
+ # --description $description \
63
+ # --results-folder results/ \
64
+ # --dataset sonnet_2048_128
65
+
66
+ # python3 $VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/scripts/plot-nightly-results.py \
67
+ # --description $description \
68
+ # --results-folder results/ \
69
+ # --dataset sonnet_128_2048
70
33
- # upload results and figures
34
- /workspace/buildkite-agent artifact upload "nightly_results.png"
35
- /workspace/buildkite-agent artifact upload $VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/nightly-pipeline.yaml
36
- /workspace/buildkite-agent artifact upload $VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/tests/nightly-tests.json
37
- /workspace/buildkite-agent annotate --style "success" --context "nightly-benchmarks-results" --append < nightly_results.md
71
+ # # upload results and figures
72
+ # /workspace/buildkite-agent artifact upload "nightly_results*.png"
73
+ # /workspace/buildkite-agent artifact upload $VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/nightly-pipeline.yaml
74
+ # /workspace/buildkite-agent artifact upload $VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/tests/nightly-tests.json
75
+ # /workspace/buildkite-agent annotate --style "success" --context "nightly-benchmarks-results" --append < nightly_results.md
76
}
77
78
main "$@"
\ No newline at end of file
.buildkite/nightly-benchmarks/scripts/plot-nightly-results.py
deleted
-135
@@ -1,135 +0,0 @@
1
-import argparse
2
-import json
3
-import math
4
-from pathlib import Path
5
-
6
-import matplotlib.pyplot as plt
7
-import pandas as pd
8
-from tabulate import tabulate
9
-
10
-
11
-def parse_arguments():
12
- parser = argparse.ArgumentParser(
13
- description=
14
- 'Parse command line arguments for summary-nightly-results script.')
15
- parser.add_argument('--results-folder',
16
- type=str,
17
- required=True,
18
- help='The folder where the results are stored.')
19
- parser.add_argument('--description',
20
- type=str,
21
- required=True,
22
- help='Description of the results.')
23
-
24
- args = parser.parse_args()
25
- return args
26
-
27
-
28
-def main(args):
29
- bar_colors = ['#56B4E9', '#009E73', '#D55E00', '#E69F00']
30
- results_folder = Path(args.results_folder)
31
-
32
- results = []
33
-
34
- # collect results
35
- for test_file in results_folder.glob("*_nightly_results.json"):
36
- with open(test_file, "r") as f:
37
- results = results + json.loads(f.read())
38
-
39
- # generate markdown table
40
- df = pd.DataFrame.from_dict(results)
41
-
42
- md_table = tabulate(df, headers='keys', tablefmt='pipe', showindex=False)
43
-
44
- with open(args.description, "r") as f:
45
- description = f.read()
46
-
47
- description = description.format(
48
- nightly_results_benchmarking_table=md_table)
49
-
50
- with open("nightly_results.md", "w") as f:
51
- f.write(description)
52
-
53
- plt.rcParams.update({'font.size': 20})
54
-
55
- # plot results
56
- fig, axes = plt.subplots(3, 3, figsize=(16, 14))
57
- fig.subplots_adjust(hspace=1)
58
- methods = ["vllm", "trt", "lmdeploy", "tgi"]
59
- for i, model in enumerate(["llama8B", "llama70B", "mixtral8x7B"]):
60
- for j, metric in enumerate(["TTFT", "ITL"]):
61
- means, stds = [], []
62
- for method in methods:
63
- target = df['Test name'].str.contains(model)
64
- target = target & df['Engine'].str.contains(method)
65
- filtered_df = df[target]
66
-
67
- if filtered_df.empty:
68
- means.append(0.)
69
- stds.append(0.)
70
- else:
71
- means.append(filtered_df[f"Mean {metric} (ms)"].values[0])
72
- std = filtered_df[f"Std {metric} (ms)"].values[0]
73
- success = filtered_df["Successful req."].values[0]
74
- stds.append(std / math.sqrt(success))
75
-
76
- print(model, metric)
77
- print(means, stds)
78
-
79
- ax = axes[i, j + 1]
80
-
81
- bars = ax.bar(
82
- ["vllm", "trt", "lmdeploy", "tgi"],
83
- means,
84
- yerr=stds,
85
- capsize=10,
86
- )
87
- for idx, bar in enumerate(bars):
88
- bar.set_color(bar_colors[idx])
89
- ax.set_ylim(bottom=0)
90
-
91
- ax.set_ylabel(f"{metric} (ms)")
92
- ax.set_title(f"{model} {metric}")
93
- ax.grid(axis='y')
94
-
95
- metric = "Tput"
96
- j = 0
97
- if True:
98
- tputs = []
99
- for method in methods:
100
- target = df['Test name'].str.contains(model)
101
- target = target & df['Engine'].str.contains(method)
102
- filtered_df = df[target]
103
-
104
- if filtered_df.empty:
105
- tputs.append(0.)
106
- else:
107
- input_tput = filtered_df["Input Tput (tok/s)"].values[0]
108
- output_tput = filtered_df["Output Tput (tok/s)"].values[0]
109
- tputs.append(input_tput + output_tput)
110
-
111
- print(model, metric)
112
- print(tputs)
113
-
114
- ax = axes[i, j]
115
-
116
- bars = ax.bar(
117
- ["vllm", "trt", "lmdeploy", "tgi"],
118
- tputs,
119
- )
120
- for idx, bar in enumerate(bars):
121
- bar.set_color(bar_colors[idx])
122
-
123
- ax.set_ylim(bottom=0)
124
-
125
- ax.set_ylabel("Tput (token/s)")
126
- ax.set_title(f"{model} {metric}")
127
- ax.grid(axis='y')
128
-
129
- fig.tight_layout()
130
- fig.savefig("nightly_results.png", bbox_inches='tight', dpi=400)
131
-
132
-
133
-if __name__ == '__main__':
134
- args = parse_arguments()
135
- main(args)
.buildkite/nightly-benchmarks/scripts/run-lmdeploy-nightly.sh
deleted
-218
@@ -1,218 +0,0 @@
1
-#!/bin/bash
2
-
3
-set -o pipefail
4
-
5
-check_gpus() {
6
- # check the number of GPUs and GPU type.
7
- declare -g gpu_count=$(nvidia-smi --list-gpus | wc -l)
8
- if [[ $gpu_count -gt 0 ]]; then
9
- echo "GPU found."
10
- else
11
- echo "Need at least 1 GPU to run benchmarking."
12
- exit 1
13
- fi
14
- declare -g gpu_type=$(echo $(nvidia-smi --query-gpu=name --format=csv,noheader) | awk '{print $2}')
15
- echo "GPU type is $gpu_type"
16
-}
17
-
18
-kill_gpu_processes() {
19
- pkill lmdeploy || true
20
- # waiting for GPU processes to be fully killed
21
- sleep 10
22
- # Print the GPU memory usage
23
- # so that we know if all GPU processes are killed.
24
- gpu_memory_usage=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits -i 0)
25
- # The memory usage should be 0 MB.
26
- echo "GPU 0 Memory Usage: $gpu_memory_usage MB"
27
-}
28
-
29
-json2args() {
30
- # transforms the JSON string to command line args, and '_' is replaced to '-'
31
- # example:
32
- # input: { "model": "meta-llama/Llama-2-7b-chat-hf", "tensor_parallel_size": 1 }
33
- # output: --model meta-llama/Llama-2-7b-chat-hf --tensor-parallel-size 1
34
- local json_string=$1
35
- local args=$(
36
- echo "$json_string" | jq -r '
37
- to_entries |
38
- map("--" + (.key | gsub("_"; "-")) + " " + (.value | tostring)) |
39
- join(" ")
40
- '
41
- )
42
- echo "$args"
43
-}
44
-
45
-wait_for_server() {
46
- # wait for vllm server to start
47
- # return 1 if vllm server crashes
48
- timeout 1200 bash -c '
49
- until curl -s localhost:8000/v1/completions > /dev/null; do
50
- sleep 1
51
- done' && return 0 || return 1
52
-}
53
-
54
-run_serving_tests() {
55
- # run serving tests using `benchmark_serving.py`
56
- # $1: a json file specifying serving test cases
57
-
58
- local serving_test_file
59
- serving_test_file=$1
60
-
61
- # Iterate over serving tests
62
- jq -c '.[]' "$serving_test_file" | while read -r params; do
63
- # get the test name, and append the GPU type back to it.
64
- test_name=$(echo "$params" | jq -r '.test_name')
65
-
66
- # if TEST_SELECTOR is set, only run the test cases that match the selector
67
- if [[ -n "$TEST_SELECTOR" ]] && [[ ! "$test_name" =~ $TEST_SELECTOR ]]; then
68
- echo "Skip test case $test_name."
69
- continue
70
- fi
71
-
72
- # append lmdeploy to the test name
73
- test_name=lmdeploy_$test_name
74
-
75
- # get common parameters
76
- common_params=$(echo "$params" | jq -r '.common_parameters')
77
- model=$(echo "$common_params" | jq -r '.model')
78
- tp=$(echo "$common_params" | jq -r '.tp')
79
- dataset_name=$(echo "$common_params" | jq -r '.dataset_name')
80
- dataset_path=$(echo "$common_params" | jq -r '.dataset_path')
81
- port=$(echo "$common_params" | jq -r '.port')
82
- num_prompts=$(echo "$common_params" | jq -r '.num_prompts')
83
-
84
-
85
-
86
- # get client and server arguments
87
- server_params=$(echo "$params" | jq -r '.lmdeploy_server_parameters')
88
- client_params=$(echo "$params" | jq -r '.lmdeploy_client_parameters')
89
- server_args=$(json2args "$server_params")
90
- client_args=$(json2args "$client_params")
91
- qps_list=$(echo "$params" | jq -r '.qps_list')
92
- qps_list=$(echo "$qps_list" | jq -r '.[] | @sh')
93
- echo "Running over qps list $qps_list"
94
-
95
- # check if there is enough GPU to run the test
96
- if [[ $gpu_count -lt $tp ]]; then
97
- echo "Required tensor-parallel-size $tp but only $gpu_count GPU found. Skip testcase $test_name."
98
- continue
99
- fi
100
-
101
- # prepare tokenizer
102
- rm -rf /tokenizer_cache
103
- mkdir /tokenizer_cache
104
- python ../.buildkite/nightly-benchmarks/scripts/download-tokenizer.py \
105
- --model "$model" \
106
- --cachedir /tokenizer_cache
107
-
108
- server_command="lmdeploy serve api_server $model \
109
- --tp $tp \
110
- --server-port $port \
111
- $server_args"
112
-
113
- # run the server
114
- echo "Running test case $test_name"
115
- echo "Server command: $server_command"
116
- bash -c "$server_command" &
117
-
118
- # wait until the server is alive
119
- wait_for_server
120
- if [ $? -eq 0 ]; then
121
- echo ""
122
- echo "lmdeploy server is up and running."
123
- else
124
- echo ""
125
- echo "lmdeploy failed to start within the timeout period."
126
- break
127
- fi
128
-
129
- # get model name
130
- model_name=$(python ../.buildkite/nightly-benchmarks/scripts/get-lmdeploy-modelname.py)
131
-
132
- # iterate over different QPS
133
- for qps in $qps_list; do
134
- # remove the surrounding single quote from qps
135
- if [[ "$qps" == *"inf"* ]]; then
136
- echo "qps was $qps"
137
- qps="inf"
138
- echo "now qps is $qps"
139
- fi
140
-
141
- new_test_name=$test_name"_qps_"$qps
142
-
143
- client_command="python3 benchmark_serving.py \
144
- --backend lmdeploy \
145
- --tokenizer /tokenizer_cache \
146
- --dataset-name $dataset_name \
147
- --dataset-path $dataset_path \
148
- --num-prompts $num_prompts \
149
- --port $port \
150
- --save-result \
151
- --result-dir $RESULTS_FOLDER \
152
- --result-filename ${new_test_name}.json \
153
- --request-rate $qps \
154
- --model \"$model_name\" \
155
- $client_args"
156
-
157
- echo "Running test case $test_name with qps $qps"
158
- echo "Client command: $client_command"
159
-
160
- eval "$client_command"
161
-
162
- # record the benchmarking commands
163
- jq_output=$(jq -n \
164
- --arg server "$server_command" \
165
- --arg client "$client_command" \
166
- --arg gpu "$gpu_type" \
167
- --arg engine "lmdeploy" \
168
- '{
169
- server_command: $server,
170
- client_command: $client,
171
- gpu_type: $gpu,
172
- engine: $engine
173
- }')
174
- echo "$jq_output" >"$RESULTS_FOLDER/${new_test_name}.commands"
175
-
176
- done
177
-
178
- # clean up
179
- kill_gpu_processes
180
- rm -rf /root/.cache/huggingface/*
181
- done
182
-}
183
-
184
-
185
-upload_to_buildkite() {
186
- # upload the benchmarking results to buildkite
187
-
188
- # if the agent binary is not found, skip uploading the results, exit 0
189
- if [ ! -f /workspace/buildkite-agent ]; then
190
- echo "buildkite-agent binary not found. Skip uploading the results."
191
- return 0
192
- fi
193
- # /workspace/buildkite-agent annotate --style "success" --context "benchmark-results" --append < $RESULTS_FOLDER/${CURRENT_LLM_SERVING_ENGINE}_nightly_results.md
194
- /workspace/buildkite-agent artifact upload "$RESULTS_FOLDER/*"
195
-}
196
-
197
-
198
-main() {
199
-
200
- check_gpus
201
- # enter vllm directory
202
- cd $VLLM_SOURCE_CODE_LOC/benchmarks
203
-
204
- declare -g RESULTS_FOLDER=results/
205
- mkdir -p $RESULTS_FOLDER
206
- BENCHMARK_ROOT=../.buildkite/nightly-benchmarks/
207
-
208
- python -m pip install transformers==4.41.2
209
-
210
- export CURRENT_LLM_SERVING_ENGINE=lmdeploy
211
- run_serving_tests $BENCHMARK_ROOT/tests/nightly-tests.json
212
- python -m pip install tabulate pandas
213
- python $BENCHMARK_ROOT/scripts/summary-nightly-results.py
214
- upload_to_buildkite
215
-
216
-}
217
-
218
-main "$@"
.buildkite/nightly-benchmarks/scripts/run-nightly-benchmarks.sh
new
+357
@@ -0,0 +1,357 @@
1
+#!/bin/bash
2
+
3
+set -o pipefail
4
+set -x
5
+
6
+check_gpus() {
7
+ # check the number of GPUs and GPU type.
8
+ declare -g gpu_count=$(nvidia-smi --list-gpus | wc -l)
9
+ if [[ $gpu_count -gt 0 ]]; then
10
+ echo "GPU found."
11
+ else
12
+ echo "Need at least 1 GPU to run benchmarking."
13
+ exit 1
14
+ fi
15
+ declare -g gpu_type=$(echo $(nvidia-smi --query-gpu=name --format=csv,noheader) | awk '{print $2}')
16
+ echo "GPU type is $gpu_type"
17
+}
18
+
19
+check_hf_token() {
20
+ # check if HF_TOKEN is available and valid
21
+ if [[ -z "$HF_TOKEN" ]]; then
22
+ echo "Error: HF_TOKEN is not set."
23
+ exit 1
24
+ elif [[ ! "$HF_TOKEN" =~ ^hf_ ]]; then
25
+ echo "Error: HF_TOKEN does not start with 'hf_'."
26
+ exit 1
27
+ else
28
+ echo "HF_TOKEN is set and valid."
29
+ fi
30
+}
31
+
32
+
33
+upload_to_buildkite() {
34
+ # upload the benchmarking results to buildkite
35
+
36
+ # if the agent binary is not found, skip uploading the results, exit 0
37
+ if [ ! -f /workspace/buildkite-agent ]; then
38
+ echo "buildkite-agent binary not found. Skip uploading the results."
39
+ return 0
40
+ fi
41
+ # /workspace/buildkite-agent annotate --style "success" --context "benchmark-results" --append < $RESULTS_FOLDER/${CURRENT_LLM_SERVING_ENGINE}_nightly_results.md
42
+ /workspace/buildkite-agent artifact upload "$RESULTS_FOLDER/*"
43
+}
44
+
45
+
46
+get_current_llm_serving_engine() {
47
+
48
+ if which lmdeploy >/dev/null; then
49
+ echo "Container: lmdeploy"
50
+ export CURRENT_LLM_SERVING_ENGINE=lmdeploy
51
+ return
52
+ fi
53
+
54
+ if [ -e /tgi-entrypoint.sh ]; then
55
+ echo "Container: tgi"
56
+ export CURRENT_LLM_SERVING_ENGINE=tgi
57
+ return
58
+ fi
59
+
60
+ if which trtllm-build >/dev/null; then
61
+ echo "Container: tensorrt-llm"
62
+ export CURRENT_LLM_SERVING_ENGINE=trt
63
+ return
64
+ fi
65
+
66
+ if [ -e /sgl-workspace ]; then
67
+ echo "Container: sglang"
68
+ export CURRENT_LLM_SERVING_ENGINE=sglang
69
+ return
70
+ fi
71
+
72
+ if [ -e /vllm-workspace ]; then
73
+ echo "Container: vllm"
74
+ # move to a completely irrelevant directory, to avoid import vllm from current folder
75
+ export CURRENT_LLM_SERVING_ENGINE=vllm
76
+
77
+ return
78
+ fi
79
+}
80
+
81
+json2args() {
82
+ # transforms the JSON string to command line args, and '_' is replaced to '-'
83
+ # example:
84
+ # input: { "model": "meta-llama/Llama-2-7b-chat-hf", "tensor_parallel_size": 1 }
85
+ # output: --model meta-llama/Llama-2-7b-chat-hf --tensor-parallel-size 1
86
+ local json_string=$1
87
+ local args=$(
88
+ echo "$json_string" | jq -r '
89
+ to_entries |
90
+ map("--" + (.key | gsub("_"; "-")) + " " + (.value | tostring)) |
91
+ join(" ")
92
+ '
93
+ )
94
+ echo "$args"
95
+}
96
+
97
+kill_gpu_processes() {
98
+ pkill -f python
99
+ pkill -f python3
100
+ pkill -f tritonserver
101
+ pkill -f pt_main_thread
102
+ pkill -f text-generation
103
+ pkill -f lmdeploy
104
+
105
+ while [ $(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits | head -n 1) -ge 1000 ]; do
106
+ sleep 1
107
+ done
108
+}
109
+
110
+wait_for_server() {
111
+ # wait for vllm server to start
112
+ # return 1 if vllm server crashes
113
+ timeout 1200 bash -c '
114
+ until curl -s localhost:8000/v1/completions > /dev/null; do
115
+ sleep 1
116
+ done' && return 0 || return 1
117
+}
118
+
119
+ensure_installed() {
120
+ # Ensure that the given command is installed by apt-get
121
+ local cmd=$1
122
+ if ! which $cmd >/dev/null; then
123
+ apt-get update && apt-get install -y $cmd
124
+ fi
125
+}
126
+
127
+run_serving_tests() {
128
+ # run serving tests using `benchmark_serving.py`
129
+ # $1: a json file specifying serving test cases
130
+
131
+ local serving_test_file
132
+ serving_test_file=$1
133
+
134
+ # Iterate over serving tests
135
+ jq -c '.[]' "$serving_test_file" | while read -r params; do
136
+ # get the test name, and append the GPU type back to it.
137
+ test_name=$(echo "$params" | jq -r '.test_name')
138
+
139
+ # if TEST_SELECTOR is set, only run the test cases that match the selector
140
+ if [[ -n "$TEST_SELECTOR" ]] && [[ ! "$test_name" =~ $TEST_SELECTOR ]]; then
141
+ echo "Skip test case $test_name."
142
+ continue
143
+ fi
144
+
145
+ # prepend the current serving engine to the test name
146
+ test_name=${CURRENT_LLM_SERVING_ENGINE}_${test_name}
147
+
148
+ # get common parameters
149
+ common_params=$(echo "$params" | jq -r '.common_parameters')
150
+ model=$(echo "$common_params" | jq -r '.model')
151
+ tp=$(echo "$common_params" | jq -r '.tp')
152
+ dataset_name=$(echo "$common_params" | jq -r '.dataset_name')
153
+ dataset_path=$(echo "$common_params" | jq -r '.dataset_path')
154
+ port=$(echo "$common_params" | jq -r '.port')
155
+ num_prompts=$(echo "$common_params" | jq -r '.num_prompts')
156
+ reuse_server=$(echo "$common_params" | jq -r '.reuse_server')
157
+
158
+ # get client and server arguments
159
+ server_params=$(echo "$params" | jq -r ".${CURRENT_LLM_SERVING_ENGINE}_server_parameters")
160
+ client_params=$(echo "$params" | jq -r ".${CURRENT_LLM_SERVING_ENGINE}_client_parameters")
161
+ client_args=$(json2args "$client_params")
162
+ qps_list=$(echo "$params" | jq -r '.qps_list')
163
+ qps_list=$(echo "$qps_list" | jq -r '.[] | @sh')
164
+ echo "Running over qps list $qps_list"
165
+
166
+ # check if there is enough GPU to run the test
167
+ if [[ $gpu_count -lt $tp ]]; then
168
+ echo "Required num-shard $tp but only $gpu_count GPU found. Skip testcase $test_name."
169
+ continue
170
+ fi
171
+
172
+ if [[ $reuse_server == "true" ]]; then
173
+ echo "Reuse previous server for test case $test_name"
174
+ else
175
+ kill_gpu_processes
176
+ bash $VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/scripts/launch-server.sh \
177
+ "$server_params" "$common_params"
178
+ fi
179
+
180
+ wait_for_server
181
+
182
+ if [ $? -eq 0 ]; then
183
+ echo ""
184
+ echo "$CURRENT_LLM_SERVING_ENGINE server is up and running."
185
+ else
186
+ echo ""
187
+ echo "$CURRENT_LLM_SERVING_ENGINE failed to start within the timeout period."
188
+ break
189
+ fi
190
+
191
+ # prepare tokenizer
192
+ # this is required for lmdeploy.
193
+ cd $VLLM_SOURCE_CODE_LOC/benchmarks
194
+ rm -rf /tokenizer_cache
195
+ mkdir /tokenizer_cache
196
+ python3 ../.buildkite/nightly-benchmarks/scripts/download-tokenizer.py \
197
+ --model "$model" \
198
+ --cachedir /tokenizer_cache
199
+ cd $VLLM_SOURCE_CODE_LOC/benchmarks
200
+
201
+
202
+ # change model name for lmdeploy (it will not follow standard hf name)
203
+ if [[ "$CURRENT_LLM_SERVING_ENGINE" == "lmdeploy" ]]; then
204
+ model=$(python ../.buildkite/nightly-benchmarks/scripts/get-lmdeploy-modelname.py)
205
+ fi
206
+
207
+ # iterate over different QPS
208
+ for qps in $qps_list; do
209
+ # remove the surrounding single quote from qps
210
+ if [[ "$qps" == *"inf"* ]]; then
211
+ echo "qps was $qps"
212
+ qps="inf"
213
+ echo "now qps is $qps"
214
+ fi
215
+
216
+ new_test_name=$test_name"_qps_"$qps
217
+
218
+ backend=$CURRENT_LLM_SERVING_ENGINE
219
+
220
+ if [[ $backend = "trt" ]]; then
221
+ backend="tensorrt-llm"
222
+ fi
223
+
224
+ if [[ "$backend" == *"vllm"* ]]; then
225
+ backend="vllm"
226
+ fi
227
+
228
+ if [[ "$dataset_name" = "sharegpt" ]]; then
229
+
230
+ client_command="python3 benchmark_serving.py \
231
+ --backend $backend \
232
+ --tokenizer /tokenizer_cache \
233
+ --model $model \
234
+ --dataset-name $dataset_name \
235
+ --dataset-path $dataset_path \
236
+ --num-prompts $num_prompts \
237
+ --port $port \
238
+ --save-result \
239
+ --result-dir $RESULTS_FOLDER \
240
+ --result-filename ${new_test_name}.json \
241
+ --request-rate $qps \
242
+ --ignore-eos \
243
+ $client_args"
244
+
245
+ elif [[ "$dataset_name" = "sonnet" ]]; then
246
+
247
+ sonnet_input_len=$(echo "$common_params" | jq -r '.sonnet_input_len')
248
+ sonnet_output_len=$(echo "$common_params" | jq -r '.sonnet_output_len')
249
+ sonnet_prefix_len=$(echo "$common_params" | jq -r '.sonnet_prefix_len')
250
+
251
+ client_command="python3 benchmark_serving.py \
252
+ --backend $backend \
253
+ --tokenizer /tokenizer_cache \
254
+ --model $model \
255
+ --dataset-name $dataset_name \
256
+ --dataset-path $dataset_path \
257
+ --num-prompts $num_prompts \
258
+ --sonnet-input-len $sonnet_input_len \
259
+ --sonnet-output-len $sonnet_output_len \
260
+ --sonnet-prefix-len $sonnet_prefix_len \
261
+ --port $port \
262
+ --save-result \
263
+ --result-dir $RESULTS_FOLDER \
264
+ --result-filename ${new_test_name}.json \
265
+ --request-rate $qps \
266
+ --ignore-eos \
267
+ $client_args"
268
+
269
+ else
270
+
271
+ echo "The dataset name must be either 'sharegpt' or 'sonnet'. Got $dataset_name."
272
+ exit 1
273
+
274
+ fi
275
+
276
+
277
+
278
+ echo "Running test case $test_name with qps $qps"
279
+ echo "Client command: $client_command"
280
+
281
+ eval "$client_command"
282
+
283
+ server_command="None"
284
+
285
+ # record the benchmarking commands
286
+ jq_output=$(jq -n \
287
+ --arg server "$server_command" \
288
+ --arg client "$client_command" \
289
+ --arg gpu "$gpu_type" \
290
+ --arg engine "$CURRENT_LLM_SERVING_ENGINE" \
291
+ '{
292
+ server_command: $server,
293
+ client_command: $client,
294
+ gpu_type: $gpu,
295
+ engine: $engine
296
+ }')
297
+ echo "$jq_output" >"$RESULTS_FOLDER/${new_test_name}.commands"
298
+
299
+ done
300
+
301
+ done
302
+
303
+ kill_gpu_processes
304
+}
305
+
306
+
307
+prepare_dataset() {
308
+
309
+ # download sharegpt dataset
310
+ cd $VLLM_SOURCE_CODE_LOC/benchmarks
311
+ wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
312
+
313
+ # duplicate sonnet by 4x, to allow benchmarking with input length 2048
314
+ cd $VLLM_SOURCE_CODE_LOC/benchmarks
315
+ echo "" > sonnet_4x.txt
316
+ for _ in {1..4}
317
+ do
318
+ cat sonnet.txt >> sonnet_4x.txt
319
+ done
320
+
321
+}
322
+
323
+main() {
324
+
325
+ # check if the environment variable is successfully injected from yaml
326
+
327
+ check_gpus
328
+ check_hf_token
329
+ get_current_llm_serving_engine
330
+
331
+ pip install -U transformers
332
+
333
+ # check storage
334
+ df -h
335
+
336
+ ensure_installed wget
337
+ ensure_installed curl
338
+ ensure_installed jq
339
+
340
+ prepare_dataset
341
+
342
+ cd $VLLM_SOURCE_CODE_LOC/benchmarks
343
+ declare -g RESULTS_FOLDER=results/
344
+ mkdir -p $RESULTS_FOLDER
345
+ BENCHMARK_ROOT=$VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/
346
+
347
+ # run the test
348
+ run_serving_tests $BENCHMARK_ROOT/tests/nightly-tests.json
349
+
350
+ # upload benchmark results to buildkite
351
+ python3 -m pip install tabulate pandas
352
+ python3 $BENCHMARK_ROOT/scripts/summary-nightly-results.py
353
+ upload_to_buildkite
354
+
355
+}
356
+
357
+main "$@"
.buildkite/nightly-benchmarks/scripts/run-tgi-nightly.sh
deleted
-216
@@ -1,216 +0,0 @@
1
-#!/bin/bash
2
-
3
-set -o pipefail
4
-
5
-check_gpus() {
6
- # check the number of GPUs and GPU type.
7
- declare -g gpu_count=$(nvidia-smi --list-gpus | wc -l)
8
- if [[ $gpu_count -gt 0 ]]; then
9
- echo "GPU found."
10
- else
11
- echo "Need at least 1 GPU to run benchmarking."
12
- exit 1
13
- fi
14
- declare -g gpu_type=$(echo $(nvidia-smi --query-gpu=name --format=csv,noheader) | awk '{print $2}')
15
- echo "GPU type is $gpu_type"
16
-}
17
-
18
-kill_gpu_processes() {
19
- pkill text-generation || true
20
- # waiting for GPU processes to be fully killed
21
- sleep 10
22
- # Print the GPU memory usage
23
- # so that we know if all GPU processes are killed.
24
- gpu_memory_usage=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits -i 0)
25
- # The memory usage should be 0 MB.
26
- echo "GPU 0 Memory Usage: $gpu_memory_usage MB"
27
-}
28
-
29
-json2args() {
30
- # transforms the JSON string to command line args, and '_' is replaced to '-'
31
- # example:
32
- # input: { "model": "meta-llama/Llama-2-7b-chat-hf", "tensor_parallel_size": 1 }
33
- # output: --model meta-llama/Llama-2-7b-chat-hf --tensor-parallel-size 1
34
- local json_string=$1
35
- local args=$(
36
- echo "$json_string" | jq -r '
37
- to_entries |
38
- map("--" + (.key | gsub("_"; "-")) + " " + (.value | tostring)) |
39
- join(" ")
40
- '
41
- )
42
- echo "$args"
43
-}
44
-
45
-wait_for_server() {
46
- timeout 1200 bash -c '
47
- until curl -s localhost:8000/generate_stream > /dev/null; do
48
- sleep 1
49
- done' && return 0 || return 1
50
-}
51
-
52
-run_serving_tests() {
53
- # run serving tests using `benchmark_serving.py`
54
- # $1: a json file specifying serving test cases
55
-
56
- local serving_test_file
57
- serving_test_file=$1
58
-
59
- # Iterate over serving tests
60
- jq -c '.[]' "$serving_test_file" | while read -r params; do
61
- # get the test name, and append the GPU type back to it.
62
- test_name=$(echo "$params" | jq -r '.test_name')
63
-
64
-
65
- # if TEST_SELECTOR is set, only run the test cases that match the selector
66
- if [[ -n "$TEST_SELECTOR" ]] && [[ ! "$test_name" =~ $TEST_SELECTOR ]]; then
67
- echo "Skip test case $test_name."
68
- continue
69
- fi
70
-
71
- # append tgi to the test name
72
- test_name=tgi_$test_name
73
-
74
- # get common parameters
75
- common_params=$(echo "$params" | jq -r '.common_parameters')
76
- model=$(echo "$common_params" | jq -r '.model')
77
- tp=$(echo "$common_params" | jq -r '.tp')
78
- dataset_name=$(echo "$common_params" | jq -r '.dataset_name')
79
- dataset_path=$(echo "$common_params" | jq -r '.dataset_path')
80
- port=$(echo "$common_params" | jq -r '.port')
81
- num_prompts=$(echo "$common_params" | jq -r '.num_prompts')
82
-
83
- # get client and server arguments
84
- server_params=$(echo "$params" | jq -r '.tgi_server_parameters')
85
- client_params=$(echo "$params" | jq -r '.tgi_client_parameters')
86
- server_args=$(json2args "$server_params")
87
- client_args=$(json2args "$client_params")
88
- qps_list=$(echo "$params" | jq -r '.qps_list')
89
- qps_list=$(echo "$qps_list" | jq -r '.[] | @sh')
90
- echo "Running over qps list $qps_list"
91
-
92
- # check if there is enough GPU to run the test
93
- if [[ $gpu_count -lt $tp ]]; then
94
- echo "Required num-shard $tp but only $gpu_count GPU found. Skip testcase $test_name."
95
- continue
96
- fi
97
-
98
- if echo "$common_params" | jq -e 'has("fp8")' > /dev/null; then
99
- echo "Key 'fp8' exists in common params."
100
- server_command="/tgi-entrypoint.sh \
101
- --model-id $model \
102
- --num-shard $tp \
103
- --port $port \
104
- --quantize fp8 \
105
- $server_args"
106
- else
107
- echo "Key 'fp8' does not exist in common params."
108
- server_command="/tgi-entrypoint.sh \
109
- --model-id $model \
110
- --num-shard $tp \
111
- --port $port \
112
- $server_args"
113
- fi
114
-
115
-
116
-
117
-
118
- # run the server
119
- echo "Running test case $test_name"
120
- echo "Server command: $server_command"
121
- eval "$server_command" &
122
-
123
- # wait until the server is alive
124
- wait_for_server
125
- if [ $? -eq 0 ]; then
126
- echo ""
127
- echo "tgi server is up and running."
128
- else
129
- echo ""
130
- echo "tgi failed to start within the timeout period."
131
- break
132
- fi
133
-
134
- # iterate over different QPS
135
- for qps in $qps_list; do
136
- # remove the surrounding single quote from qps
137
- if [[ "$qps" == *"inf"* ]]; then
138
- echo "qps was $qps"
139
- qps="inf"
140
- echo "now qps is $qps"
141
- fi
142
-
143
- new_test_name=$test_name"_qps_"$qps
144
-
145
- client_command="python3 benchmark_serving.py \
146
- --backend tgi \
147
- --model $model \
148
- --dataset-name $dataset_name \
149
- --dataset-path $dataset_path \
150
- --num-prompts $num_prompts \
151
- --port $port \
152
- --save-result \
153
- --result-dir $RESULTS_FOLDER \
154
- --result-filename ${new_test_name}.json \
155
- --request-rate $qps \
156
- $client_args"
157
-
158
- echo "Running test case $test_name with qps $qps"
159
- echo "Client command: $client_command"
160
-
161
- eval "$client_command"
162
-
163
- # record the benchmarking commands
164
- jq_output=$(jq -n \
165
- --arg server "$server_command" \
166
- --arg client "$client_command" \
167
- --arg gpu "$gpu_type" \
168
- --arg engine "tgi" \
169
- '{
170
- server_command: $server,
171
- client_command: $client,
172
- gpu_type: $gpu,
173
- engine: $engine
174
- }')
175
- echo "$jq_output" >"$RESULTS_FOLDER/${new_test_name}.commands"
176
-
177
- done
178
-
179
- # clean up
180
- kill_gpu_processes
181
- rm -rf /root/.cache/huggingface/*
182
- done
183
-}
184
-
185
-
186
-
187
-upload_to_buildkite() {
188
- # upload the benchmarking results to buildkite
189
-
190
- # if the agent binary is not found, skip uploading the results, exit 0
191
- if [ ! -f /workspace/buildkite-agent ]; then
192
- echo "buildkite-agent binary not found. Skip uploading the results."
193
- return 0
194
- fi
195
- # /workspace/buildkite-agent annotate --style "success" --context "benchmark-results" --append < $RESULTS_FOLDER/${CURRENT_LLM_SERVING_ENGINE}_nightly_results.md
196
- /workspace/buildkite-agent artifact upload "$RESULTS_FOLDER/*"
197
-}
198
-
199
-main() {
200
-
201
- check_gpus
202
- # enter vllm directory
203
- cd $VLLM_SOURCE_CODE_LOC/benchmarks
204
- declare -g RESULTS_FOLDER=results/
205
- mkdir -p $RESULTS_FOLDER
206
- BENCHMARK_ROOT=../.buildkite/nightly-benchmarks/
207
-
208
- export CURRENT_LLM_SERVING_ENGINE=tgi
209
- run_serving_tests $BENCHMARK_ROOT/tests/nightly-tests.json
210
- python -m pip install tabulate pandas
211
- python $BENCHMARK_ROOT/scripts/summary-nightly-results.py
212
- upload_to_buildkite
213
-
214
-}
215
-
216
-main "$@"
.buildkite/nightly-benchmarks/scripts/run-trt-nightly.sh
deleted
-214
@@ -1,214 +0,0 @@
1
-#!/bin/bash
2
-
3
-set -o pipefail
4
-
5
-check_gpus() {
6
- # check the number of GPUs and GPU type.
7
- declare -g gpu_count=$(nvidia-smi --list-gpus | wc -l)
8
- if [[ $gpu_count -gt 0 ]]; then
9
- echo "GPU found."
10
- else
11
- echo "Need at least 1 GPU to run benchmarking."
12
- exit 1
13
- fi
14
- declare -g gpu_type=$(echo $(nvidia-smi --query-gpu=name --format=csv,noheader) | awk '{print $2}')
15
- echo "GPU type is $gpu_type"
16
-}
17
-
18
-kill_gpu_processes() {
19
- pkill tritonserver || true
20
- # waiting for GPU processes to be fully killed
21
- sleep 20
22
- # Print the GPU memory usage
23
- # so that we know if all GPU processes are killed.
24
- gpu_memory_usage=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits -i 0)
25
- # The memory usage should be 0 MB.
26
- echo "GPU 0 Memory Usage: $gpu_memory_usage MB"
27
-}
28
-
29
-json2args() {
30
- # transforms the JSON string to command line args, and '_' is replaced to '-'
31
- # example:
32
- # input: { "model": "meta-llama/Llama-2-7b-chat-hf", "tensor_parallel_size": 1 }
33
- # output: --model meta-llama/Llama-2-7b-chat-hf --tensor-parallel-size 1
34
- local json_string=$1
35
- local args=$(
36
- echo "$json_string" | jq -r '
37
- to_entries |
38
- map("--" + (.key | gsub("_"; "-")) + " " + (.value | tostring)) |
39
- join(" ")
40
- '
41
- )
42
- echo "$args"
43
-}
44
-
45
-wait_for_server() {
46
- timeout 1200 bash -c '
47
- until curl -s localhost:8000/generate_stream > /dev/null; do
48
- sleep 1
49
- done' && return 0 || return 1
50
-}
51
-
52
-run_serving_tests() {
53
- # run serving tests using `benchmark_serving.py`
54
- # $1: a json file specifying serving test cases
55
-
56
- local serving_test_file
57
- serving_test_file=$1
58
-
59
- # Iterate over serving tests
60
- jq -c '.[]' "$serving_test_file" | while read -r params; do
61
- # get the test name, and append the GPU type back to it.
62
- test_name=$(echo "$params" | jq -r '.test_name')
63
-
64
- # if TEST_SELECTOR is set, only run the test cases that match the selector
65
- if [[ -n "$TEST_SELECTOR" ]] && [[ ! "$test_name" =~ $TEST_SELECTOR ]]; then
66
- echo "Skip test case $test_name."
67
- continue
68
- fi
69
-
70
- # append trt to the test name
71
- test_name=trt_$test_name
72
-
73
- # get common parameters
74
- common_params=$(echo "$params" | jq -r '.common_parameters')
75
- model=$(echo "$common_params" | jq -r '.model')
76
- tp=$(echo "$common_params" | jq -r '.tp')
77
- dataset_name=$(echo "$common_params" | jq -r '.dataset_name')
78
- dataset_path=$(echo "$common_params" | jq -r '.dataset_path')
79
- port=$(echo "$common_params" | jq -r '.port')
80
- num_prompts=$(echo "$common_params" | jq -r '.num_prompts')
81
-
82
- # get client and server arguments
83
- server_params=$(echo "$params" | jq -r '.trt_server_parameters')
84
- client_params=$(echo "$params" | jq -r '.trt_client_parameters')
85
- client_args=$(json2args "$client_params")
86
- qps_list=$(echo "$params" | jq -r '.qps_list')
87
- qps_list=$(echo "$qps_list" | jq -r '.[] | @sh')
88
- echo "Running over qps list $qps_list"
89
-
90
- # check if there is enough GPU to run the test
91
- if [[ $gpu_count -lt $tp ]]; then
92
- echo "Required model_tp_size $tp but only $gpu_count GPU found. Skip testcase $test_name."
93
- continue
94
- fi
95
-
96
-
97
-
98
- cd $VLLM_SOURCE_CODE_LOC/benchmarks
99
-
100
-
101
- echo "Running test case $test_name"
102
- bash ../.buildkite/nightly-benchmarks/scripts/launch-trt-server.sh "$server_params" "$common_params"
103
-
104
- # wait until the server is alive
105
- wait_for_server
106
- if [ $? -eq 0 ]; then
107
- echo ""
108
- echo "trt server is up and running."
109
- else
110
- echo ""
111
- echo "trt failed to start within the timeout period."
112
- break
113
- fi
114
-
115
- # prepare tokenizer
116
- cd $VLLM_SOURCE_CODE_LOC/benchmarks
117
- rm -rf /tokenizer_cache
118
- mkdir /tokenizer_cache
119
- python ../.buildkite/nightly-benchmarks/scripts/download-tokenizer.py \
120
- --model "$model" \
121
- --cachedir /tokenizer_cache
122
- cd $VLLM_SOURCE_CODE_LOC/benchmarks
123
-
124
-
125
- # iterate over different QPS
126
- for qps in $qps_list; do
127
- # remove the surrounding single quote from qps
128
- if [[ "$qps" == *"inf"* ]]; then
129
- echo "qps was $qps"
130
- qps="inf"
131
- echo "now qps is $qps"
132
- fi
133
-
134
- new_test_name=$test_name"_qps_"$qps
135
-
136
- client_command="python3 benchmark_serving.py \
137
- --backend tensorrt-llm \
138
- --tokenizer /tokenizer_cache \
139
- --model $model \
140
- --dataset-name $dataset_name \
141
- --dataset-path $dataset_path \
142
- --num-prompts $num_prompts \
143
- --port $port \
144
- --save-result \
145
- --result-dir $RESULTS_FOLDER \
146
- --result-filename ${new_test_name}.json \
147
- --request-rate $qps \
148
- $client_args"
149
-
150
- echo "Running test case $test_name with qps $qps"
151
- echo "Client command: $client_command"
152
-
153
- eval "$client_command"
154
-
155
- server_command=""
156
- # record the benchmarking commands
157
- jq_output=$(jq -n \
158
- --arg server "$server_command" \
159
- --arg client "$client_command" \
160
- --arg gpu "$gpu_type" \
161
- --arg engine "trt" \
162
- '{
163
- server_command: $server,
164
- client_command: $client,
165
- gpu_type: $gpu,
166
- engine: $engine
167
- }')
168
- echo "$jq_output" >"$RESULTS_FOLDER/${new_test_name}.commands"
169
-
170
- done
171
-
172
- # clean up
173
- kill_gpu_processes
174
- rm -rf /root/.cache/huggingface/*
175
- done
176
-}
177
-
178
-upload_to_buildkite() {
179
- # upload the benchmarking results to buildkite
180
-
181
- # if the agent binary is not found, skip uploading the results, exit 0
182
- if [ ! -f /workspace/buildkite-agent ]; then
183
- echo "buildkite-agent binary not found. Skip uploading the results."
184
- return 0
185
- fi
186
- # /workspace/buildkite-agent annotate --style "success" --context "benchmark-results" --append < $RESULTS_FOLDER/${CURRENT_LLM_SERVING_ENGINE}_nightly_results.md
187
- /workspace/buildkite-agent artifact upload "$RESULTS_FOLDER/*"
188
-}
189
-
190
-
191
-main() {
192
-
193
- check_gpus
194
-
195
-
196
- # enter vllm directory
197
- cd $VLLM_SOURCE_CODE_LOC/benchmarks
198
-
199
- declare -g RESULTS_FOLDER=results/
200
- mkdir -p $RESULTS_FOLDER
201
- BENCHMARK_ROOT=../.buildkite/nightly-benchmarks/
202
-
203
- # update transformers package, to make sure mixtral tokenizer is available
204
- python -m pip install transformers -U
205
-
206
- export CURRENT_LLM_SERVING_ENGINE=trt
207
- run_serving_tests $BENCHMARK_ROOT/tests/nightly-tests.json
208
- python -m pip install tabulate pandas
209
- python $BENCHMARK_ROOT/scripts/summary-nightly-results.py
210
- upload_to_buildkite
211
-
212
-}
213
-
214
-main "$@"
.buildkite/nightly-benchmarks/scripts/run-vllm-nightly.sh
deleted
-221
@@ -1,221 +0,0 @@
1
-#!/bin/bash
2
-
3
-set -o pipefail
4
-
5
-check_gpus() {
6
- # check the number of GPUs and GPU type.
7
- declare -g gpu_count=$(nvidia-smi --list-gpus | wc -l)
8
- if [[ $gpu_count -gt 0 ]]; then
9
- echo "GPU found."
10
- else
11
- echo "Need at least 1 GPU to run benchmarking."
12
- exit 1
13
- fi
14
- declare -g gpu_type=$(echo $(nvidia-smi --query-gpu=name --format=csv,noheader) | awk '{print $2}')
15
- echo "GPU type is $gpu_type"
16
-}
17
-
18
-kill_gpu_processes() {
19
- # kill all processes on GPU.
20
- pkill pt_main_thread
21
- sleep 10
22
-
23
- # remove vllm config file
24
- rm -rf ~/.config/vllm
25
-
26
- # Print the GPU memory usage
27
- # so that we know if all GPU processes are killed.
28
- gpu_memory_usage=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits -i 0)
29
- # The memory usage should be 0 MB.
30
- echo "GPU 0 Memory Usage: $gpu_memory_usage MB"
31
-}
32
-
33
-json2args() {
34
- # transforms the JSON string to command line args, and '_' is replaced to '-'
35
- # example:
36
- # input: { "model": "meta-llama/Llama-2-7b-chat-hf", "tensor_parallel_size": 1 }
37
- # output: --model meta-llama/Llama-2-7b-chat-hf --tensor-parallel-size 1
38
- local json_string=$1
39
- local args=$(
40
- echo "$json_string" | jq -r '
41
- to_entries |
42
- map("--" + (.key | gsub("_"; "-")) + " " + (.value | tostring)) |
43
- join(" ")
44
- '
45
- )
46
- echo "$args"
47
-}
48
-
49
-wait_for_server() {
50
- # wait for vllm server to start
51
- # return 1 if vllm server crashes
52
- timeout 1200 bash -c '
53
- until curl -s localhost:8000/v1/completions > /dev/null; do
54
- sleep 1
55
- done' && return 0 || return 1
56
-}
57
-
58
-run_serving_tests() {
59
- # run serving tests using `benchmark_serving.py`
60
- # $1: a json file specifying serving test cases
61
-
62
- local serving_test_file
63
- serving_test_file=$1
64
-
65
- # Iterate over serving tests
66
- jq -c '.[]' "$serving_test_file" | while read -r params; do
67
- # get the test name, and append the GPU type back to it.
68
- test_name=$(echo "$params" | jq -r '.test_name')
69
-
70
- # if TEST_SELECTOR is set, only run the test cases that match the selector
71
- if [[ -n "$TEST_SELECTOR" ]] && [[ ! "$test_name" =~ $TEST_SELECTOR ]]; then
72
- echo "Skip test case $test_name."
73
- continue
74
- fi
75
-
76
- # append vllm to the test name
77
- test_name=vllm_$test_name
78
-
79
-
80
- # get common parameters
81
- common_params=$(echo "$params" | jq -r '.common_parameters')
82
- model=$(echo "$common_params" | jq -r '.model')
83
- tp=$(echo "$common_params" | jq -r '.tp')
84
- dataset_name=$(echo "$common_params" | jq -r '.dataset_name')
85
- dataset_path=$(echo "$common_params" | jq -r '.dataset_path')
86
- port=$(echo "$common_params" | jq -r '.port')
87
- num_prompts=$(echo "$common_params" | jq -r '.num_prompts')
88
-
89
- # get client and server arguments
90
- server_params=$(echo "$params" | jq -r '.vllm_server_parameters')
91
- client_params=$(echo "$params" | jq -r '.vllm_client_parameters')
92
- server_args=$(json2args "$server_params")
93
- client_args=$(json2args "$client_params")
94
- qps_list=$(echo "$params" | jq -r '.qps_list')
95
- qps_list=$(echo "$qps_list" | jq -r '.[] | @sh')
96
- echo "Running over qps list $qps_list"
97
-
98
- # check if there is enough GPU to run the test
99
- if [[ $gpu_count -lt $tp ]]; then
100
- echo "Required tensor-parallel-size $tp but only $gpu_count GPU found. Skip testcase $test_name."
101
- continue
102
- fi
103
-
104
- if echo "$common_params" | jq -e 'has("fp8")' > /dev/null; then
105
- echo "Key 'fp8' exists in common params. Use neuralmagic fp8 model for convenience."
106
- model=$(echo "$common_params" | jq -r '.neuralmagic_quantized_model')
107
- server_command="python3 \
108
- -m vllm.entrypoints.openai.api_server \
109
- -tp $tp \
110
- --model $model \
111
- --port $port \
112
- $server_args"
113
- else
114
- echo "Key 'fp8' does not exist in common params."
115
- server_command="python3 \
116
- -m vllm.entrypoints.openai.api_server \
117
- -tp $tp \
118
- --model $model \
119
- --port $port \
120
- $server_args"
121
- fi
122
-
123
- # run the server
124
- echo "Running test case $test_name"
125
- echo "Server command: $server_command"
126
- eval "$server_command" &
127
-
128
- # wait until the server is alive
129
- wait_for_server
130
- if [ $? -eq 0 ]; then
131
- echo ""
132
- echo "vllm server is up and running."
133
- else
134
- echo ""
135
- echo "vllm failed to start within the timeout period."
136
- break
137
- fi
138
-
139
- # iterate over different QPS
140
- for qps in $qps_list; do
141
- # remove the surrounding single quote from qps
142
- if [[ "$qps" == *"inf"* ]]; then
143
- echo "qps was $qps"
144
- qps="inf"
145
- echo "now qps is $qps"
146
- fi
147
-
148
- new_test_name=$test_name"_qps_"$qps
149
-
150
- client_command="python3 benchmark_serving.py \
151
- --backend vllm \
152
- --model $model \
153
- --dataset-name $dataset_name \
154
- --dataset-path $dataset_path \
155
- --num-prompts $num_prompts \
156
- --port $port \
157
- --save-result \
158
- --result-dir $RESULTS_FOLDER \
159
- --result-filename ${new_test_name}.json \
160
- --request-rate $qps \
161
- $client_args"
162
-
163
- echo "Running test case $test_name with qps $qps"
164
- echo "Client command: $client_command"
165
-
166
- eval "$client_command"
167
-
168
- # record the benchmarking commands
169
- jq_output=$(jq -n \
170
- --arg server "$server_command" \
171
- --arg client "$client_command" \
172
- --arg gpu "$gpu_type" \
173
- --arg engine "vllm" \
174
- '{
175
- server_command: $server,
176
- client_command: $client,
177
- gpu_type: $gpu,
178
- engine: $engine
179
- }')
180
- echo "$jq_output" >"$RESULTS_FOLDER/${new_test_name}.commands"
181
-
182
- done
183
-
184
- # clean up
185
- kill_gpu_processes
186
- rm -rf /root/.cache/huggingface/*
187
- done
188
-}
189
-
190
-
191
-upload_to_buildkite() {
192
- # upload the benchmarking results to buildkite
193
-
194
- # if the agent binary is not found, skip uploading the results, exit 0
195
- if [ ! -f /workspace/buildkite-agent ]; then
196
- echo "buildkite-agent binary not found. Skip uploading the results."
197
- return 0
198
- fi
199
- # /workspace/buildkite-agent annotate --style "success" --context "benchmark-results" --append < $RESULTS_FOLDER/${CURRENT_LLM_SERVING_ENGINE}_nightly_results.md
200
- /workspace/buildkite-agent artifact upload "$RESULTS_FOLDER/*"
201
-}
202
-
203
-main() {
204
-
205
- check_gpus
206
- # enter vllm directory
207
- cd $VLLM_SOURCE_CODE_LOC/benchmarks
208
- declare -g RESULTS_FOLDER=results/
209
- mkdir -p $RESULTS_FOLDER
210
- BENCHMARK_ROOT=../.buildkite/nightly-benchmarks/
211
-
212
- export CURRENT_LLM_SERVING_ENGINE=vllm
213
- run_serving_tests $BENCHMARK_ROOT/tests/nightly-tests.json
214
-
215
- python3 -m pip install tabulate pandas
216
- python3 $BENCHMARK_ROOT/scripts/summary-nightly-results.py
217
- upload_to_buildkite
218
-
219
-}
220
-
221
-main "$@"
.buildkite/nightly-benchmarks/scripts/summary-nightly-results.py
+8
-1
@@ -17,10 +17,17 @@ serving_column_mapping = {
17
"request_throughput": "Tput (req/s)",
18
"mean_ttft_ms": "Mean TTFT (ms)",
19
"std_ttft_ms": "Std TTFT (ms)",
20
+ "median_ttft_ms": "Median TTFT (ms)",
21
"mean_itl_ms": "Mean ITL (ms)",
22
"std_itl_ms": "Std ITL (ms)",
22
- "input_throughput": "Input Tput (tok/s)",
23
+ "median_itl_ms": "Median ITL (ms)",
24
+ "mean_tpot_ms": "Mean TPOT (ms)",
25
+ "std_tpot_ms": "Std TPOT (ms)",
26
+ "median_tpot_ms": "Median TPOT (ms)",
27
+ "total_token_throughput": "Total Token Tput (tok/s)",
28
"output_throughput": "Output Tput (tok/s)",
29
+ "total_input_tokens": "Total input tokens",
30
+ "total_output_tokens": "Total output tokens",
31
"engine": "Engine",
32
}
33
.buildkite/nightly-benchmarks/tests/nightly-tests.json
+237
-30
@@ -1,16 +1,18 @@
1
[
2
{
3
- "test_name": "llama8B_tp1",
4
- "qps_list": [4],
3
+ "test_name": "llama8B_tp1_sharegpt",
4
+ "qps_list": [4,8,16,32,"inf"],
5
"common_parameters": {
6
- "model": "meta-llama/Meta-Llama-3-8B",
6
+ "model": "meta-llama/Meta-Llama-3-8B-Instruct",
7
"tp": 1,
8
"dataset_name": "sharegpt",
9
"dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
10
"num_prompts": 500,
11
- "port": 8000
11
+ "port": 8000,
12
+ "reuse_server": false
13
},
14
"lmdeploy_server_parameters": {
15
+ "dtype": "bfloat16"
16
},
17
"lmdeploy_client_parameters": {
18
},
@@ -21,34 +23,158 @@
23
},
24
"trt_server_parameters": {
25
"model_type": "llama",
24
- "model_dtype": "float16",
25
- "max_batch_size": 256,
26
+ "model_dtype": "bfloat16",
27
+ "max_batch_size": 2048,
28
"max_input_len": 4096,
27
- "max_output_len": 4096,
28
- "trt_llm_version": "r24.04"
29
+ "max_seq_len": 6144,
30
+ "max_num_tokens": 16384,
31
+ "trt_llm_version": "v0.11.0"
32
},
33
"trt_client_parameters": {
34
"endpoint": "/v2/models/ensemble/generate_stream"
35
+ },
36
+ "vllm_server_parameters": {
37
+ "disable_log_stats": "",
38
+ "disable_log_requests": "",
39
+ "gpu_memory_utilization": 0.9,
40
+ "num_scheduler_steps": 10,
41
+ "max_num_seqs": 512,
42
+ "dtype": "bfloat16"
43
+ },
44
+ "vllm_client_parameters": {
45
+ },
46
+ "sglang_server_parameters": {
47
+ "disable_radix_cache": "",
48
+ "enable_torch_compile": "",
49
+ "dtype": "bfloat16"
50
+ },
51
+ "sglang_client_parameters": {
52
+ }
53
+ },
54
+ {
55
+ "test_name": "llama8B_tp1_sonnet_512_16",
56
+ "qps_list": [4,8,16,32,"inf"],
57
+ "common_parameters": {
58
+ "model": "meta-llama/Meta-Llama-3-8B-Instruct",
59
+ "tp": 1,
60
+ "dataset_name": "sonnet",
61
+ "dataset_path": "./sonnet_4x.txt",
62
+ "num_prompts": 500,
63
+ "port": 8000,
64
+ "sonnet_input_len": 512,
65
+ "sonnet_output_len": 16,
66
+ "sonnet_prefix_len": 50,
67
+ "reuse_server": true
68
+ },
69
+ "lmdeploy_server_parameters": {
70
+ "dtype": "bfloat16"
71
+ },
72
+ "lmdeploy_client_parameters": {
73
+ },
74
+ "tgi_server_parameters": {
75
+ },
76
+ "tgi_client_parameters": {
77
+ "endpoint": "/generate_stream"
78
+ },
79
+ "trt_server_parameters": {
80
+ "model_type": "llama",
81
+ "model_dtype": "bfloat16",
82
+ "max_batch_size": 2048,
83
+ "max_input_len": 4096,
84
+ "max_seq_len": 6144,
85
+ "max_num_tokens": 16384,
86
+ "trt_llm_version": "v0.11.0"
87
+ },
88
+ "trt_client_parameters": {
89
+ "endpoint": "/v2/models/ensemble/generate_stream"
90
+ },
91
+ "vllm_server_parameters": {
92
+ "disable_log_stats": "",
93
+ "disable_log_requests": "",
94
+ "gpu_memory_utilization": 0.9,
95
+ "num_scheduler_steps": 10,
96
+ "max_num_seqs": 512,
97
+ "dtype": "bfloat16"
98
+ },
99
+ "vllm_client_parameters": {
100
+ },
101
+ "sglang_server_parameters": {
102
+ "disable_radix_cache": "",
103
+ "enable_torch_compile": "",
104
+ "dtype": "bfloat16"
105
+ },
106
+ "sglang_client_parameters": {
107
+ }
108
+ },
109
+ {
110
+ "test_name": "llama8B_tp1_sonnet_512_256",
111
+ "qps_list": [4,8,16,32,"inf"],
112
+ "common_parameters": {
113
+ "model": "meta-llama/Meta-Llama-3-8B-Instruct",
114
+ "tp": 1,
115
+ "dataset_name": "sonnet",
116
+ "dataset_path": "./sonnet_4x.txt",
117
+ "num_prompts": 500,
118
+ "port": 8000,
119
+ "sonnet_input_len": 512,
120
+ "sonnet_output_len": 256,
121
+ "sonnet_prefix_len": 50,
122
+ "reuse_server": true
123
+ },
124
+ "lmdeploy_server_parameters": {
125
+ "dtype": "bfloat16"
126
+ },
127
+ "lmdeploy_client_parameters": {
128
+ },
129
+ "tgi_server_parameters": {
130
+ },
131
+ "tgi_client_parameters": {
132
+ "endpoint": "/generate_stream"
133
+ },
134
+ "trt_server_parameters": {
135
+ "model_type": "llama",
136
+ "model_dtype": "bfloat16",
137
+ "max_batch_size": 2048,
138
+ "max_input_len": 4096,
139
+ "max_seq_len": 6144,
140
+ "max_num_tokens": 16384,
141
+ "trt_llm_version": "v0.11.0"
142
},
143
+ "trt_client_parameters": {
144
+ "endpoint": "/v2/models/ensemble/generate_stream"
145
+ },
146
"vllm_server_parameters": {
147
"disable_log_stats": "",
35
- "disable_log_requests": ""
148
+ "disable_log_requests": "",
149
+ "gpu_memory_utilization": 0.9,
150
+ "num_scheduler_steps": 10,
151
+ "max_num_seqs": 512,
152
+ "dtype": "bfloat16"
153
},
154
"vllm_client_parameters": {
155
+ },
156
+ "sglang_server_parameters": {
157
+ "disable_radix_cache": "",
158
+ "enable_torch_compile": "",
159
+ "dtype": "bfloat16"
160
+ },
161
+ "sglang_client_parameters": {
162
}
163
},
164
{
41
- "test_name": "llama70B_tp4",
42
- "qps_list": [2],
165
+ "test_name": "llama70B_tp4_sharegpt",
166
+ "qps_list": [4,8,16,32,"inf"],
167
"common_parameters": {
168
"model": "meta-llama/Meta-Llama-3-70B-Instruct",
169
"tp": 4,
170
"dataset_name": "sharegpt",
171
"dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
172
"num_prompts": 500,
49
- "port": 8000
173
+ "port": 8000,
174
+ "reuse_server": false
175
},
176
"lmdeploy_server_parameters": {
177
+ "dtype": "bfloat16"
178
},
179
"lmdeploy_client_parameters": {
180
},
@@ -59,34 +185,50 @@
185
},
186
"trt_server_parameters": {
187
"model_type": "llama",
62
- "model_dtype": "float16",
63
- "max_batch_size": 256,
188
+ "model_dtype": "bfloat16",
189
+ "max_batch_size": 2048,
190
"max_input_len": 4096,
65
- "max_output_len": 4096,
66
- "trt_llm_version": "r24.04"
191
+ "max_seq_len": 6144,
192
+ "max_num_tokens": 16384,
193
+ "trt_llm_version": "v0.11.0"
194
},
195
"trt_client_parameters": {
196
"endpoint": "/v2/models/ensemble/generate_stream"
70
- },
197
+ },
198
"vllm_server_parameters": {
199
"disable_log_stats": "",
73
- "disable_log_requests": ""
200
+ "disable_log_requests": "",
201
+ "gpu_memory_utilization": 0.9,
202
+ "num_scheduler_steps": 10,
203
+ "max_num_seqs": 512,
204
+ "dtype": "bfloat16"
205
},
206
"vllm_client_parameters": {
207
+ },
208
+ "sglang_server_parameters": {
209
+ "disable_radix_cache": "",
210
+ "dtype": "bfloat16"
211
+ },
212
+ "sglang_client_parameters": {
213
}
214
},
215
{
79
- "test_name": "mixtral8x7B_tp2",
80
- "qps_list": [2],
216
+ "test_name": "llama70B_tp4_sonnet_512_16",
217
+ "qps_list": [4,8,16,32,"inf"],
218
"common_parameters": {
82
- "model": "mistralai/Mixtral-8x7B-Instruct-v0.1",
83
- "tp": 2,
84
- "dataset_name": "sharegpt",
85
- "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
219
+ "model": "meta-llama/Meta-Llama-3-70B-Instruct",
220
+ "tp": 4,
221
+ "dataset_name": "sonnet",
222
+ "dataset_path": "./sonnet_4x.txt",
223
"num_prompts": 500,
87
- "port": 8000
224
+ "port": 8000,
225
+ "sonnet_input_len": 512,
226
+ "sonnet_output_len": 16,
227
+ "sonnet_prefix_len": 50,
228
+ "reuse_server": true
229
},
230
"lmdeploy_server_parameters": {
231
+ "dtype": "bfloat16"
232
},
233
"lmdeploy_client_parameters": {
234
},
@@ -97,20 +239,85 @@
239
},
240
"trt_server_parameters": {
241
"model_type": "llama",
100
- "model_dtype": "float16",
101
- "max_batch_size": 256,
242
+ "model_dtype": "bfloat16",
243
+ "max_batch_size": 2048,
244
"max_input_len": 4096,
103
- "max_output_len": 4096,
104
- "trt_llm_version": "r24.04"
245
+ "max_seq_len": 6144,
246
+ "max_num_tokens": 16384,
247
+ "trt_llm_version": "v0.11.0"
248
},
249
"trt_client_parameters": {
250
"endpoint": "/v2/models/ensemble/generate_stream"
251
+ },
252
+ "vllm_server_parameters": {
253
+ "disable_log_stats": "",
254
+ "disable_log_requests": "",
255
+ "gpu_memory_utilization": 0.9,
256
+ "num_scheduler_steps": 10,
257
+ "max_num_seqs": 512,
258
+ "dtype": "bfloat16"
259
+ },
260
+ "vllm_client_parameters": {
261
},
262
+ "sglang_server_parameters": {
263
+ "disable_radix_cache": "",
264
+ "dtype": "bfloat16"
265
+ },
266
+ "sglang_client_parameters": {
267
+ }
268
+ },
269
+ {
270
+ "test_name": "llama70B_tp4_sonnet_512_256",
271
+ "qps_list": [4,8,16,32,"inf"],
272
+ "common_parameters": {
273
+ "model": "meta-llama/Meta-Llama-3-70B-Instruct",
274
+ "tp": 4,
275
+ "dataset_name": "sonnet",
276
+ "dataset_path": "./sonnet_4x.txt",
277
+ "num_prompts": 500,
278
+ "port": 8000,
279
+ "sonnet_input_len": 512,
280
+ "sonnet_output_len": 256,
281
+ "sonnet_prefix_len": 50,
282
+ "reuse_server": true
283
+ },
284
+ "lmdeploy_server_parameters": {
285
+ "dtype": "bfloat16"
286
+ },
287
+ "lmdeploy_client_parameters": {
288
+ },
289
+ "tgi_server_parameters": {
290
+ },
291
+ "tgi_client_parameters": {
292
+ "endpoint": "/generate_stream"
293
+ },
294
+ "trt_server_parameters": {
295
+ "model_type": "llama",
296
+ "model_dtype": "bfloat16",
297
+ "max_batch_size": 2048,
298
+ "max_input_len": 4096,
299
+ "max_seq_len": 6144,
300
+ "max_num_tokens": 16384,
301
+ "trt_llm_version": "v0.11.0"
302
+ },
303
+ "trt_client_parameters": {
304
+ "endpoint": "/v2/models/ensemble/generate_stream"
305
+ },
306
"vllm_server_parameters": {
307
"disable_log_stats": "",
111
- "disable_log_requests": ""
308
+ "disable_log_requests": "",
309
+ "gpu_memory_utilization": 0.9,
310
+ "num_scheduler_steps": 10,
311
+ "max_num_seqs": 512,
312
+ "dtype": "bfloat16"
313
},
314
"vllm_client_parameters": {
315
+ },
316
+ "sglang_server_parameters": {
317
+ "disable_radix_cache": "",
318
+ "dtype": "bfloat16"
319
+ },
320
+ "sglang_client_parameters": {
321
}
322
}
323
]
\ No newline at end of file
benchmarks/backend_request_func.py
+7
@@ -26,6 +26,7 @@ class RequestFuncInput:
26
use_beam_search: bool = False
27
logprobs: Optional[int] = None
28
multi_modal_content: Optional[dict] = None
29
+ ignore_eos: bool = False
30
31
32
@dataclass
@@ -55,6 +56,7 @@ async def async_request_tgi(
56
"do_sample": True,
57
"temperature": 0.01, # TGI does not accept 0.0 temperature.
58
"top_p": 0.99, # TGI does not accept 1.0 top_p.
59
+ # TGI does not accept ignore_eos flag.
60
}
61
payload = {
62
"inputs": request_func_input.prompt,
@@ -129,6 +131,8 @@ async def async_request_trt_llm(
131
"max_tokens": request_func_input.output_len,
132
"stream": True,
133
}
134
+ if request_func_input.ignore_eos:
135
+ payload["min_length"] = request_func_input.output_len
136
output = RequestFuncOutput()
137
output.prompt_len = request_func_input.prompt_len
138
@@ -240,6 +244,7 @@ async def async_request_openai_completions(
244
"max_tokens": request_func_input.output_len,
245
"logprobs": request_func_input.logprobs,
246
"stream": True,
247
+ "ignore_eos": request_func_input.ignore_eos,
248
}
249
headers = {
250
"Authorization": f"Bearer {os.environ.get('OPENAI_API_KEY')}"
@@ -327,6 +332,7 @@ async def async_request_openai_chat_completions(
332
"temperature": 0.0,
333
"max_tokens": request_func_input.output_len,
334
"stream": True,
335
+ "ignore_eos": request_func_input.ignore_eos,
336
}
337
headers = {
338
"Content-Type": "application/json",
@@ -430,4 +436,5 @@ ASYNC_REQUEST_FUNCS = {
436
"openai-chat": async_request_openai_chat_completions,
437
"tensorrt-llm": async_request_trt_llm,
438
"scalellm": async_request_openai_completions,
439
+ "sglang": async_request_openai_completions,
440
}
benchmarks/benchmark_serving.py
+8
@@ -397,6 +397,7 @@ async def benchmark(
397
profile: bool,
398
selected_percentile_metrics: List[str],
399
selected_percentiles: List[str],
400
+ ignore_eos: bool,
401
):
402
if backend in ASYNC_REQUEST_FUNCS:
403
request_func = ASYNC_REQUEST_FUNCS[backend]
@@ -420,6 +421,7 @@ async def benchmark(
421
best_of=best_of,
422
use_beam_search=use_beam_search,
423
multi_modal_content=test_mm_content,
424
+ ignore_eos=ignore_eos,
425
)
426
test_output = await request_func(request_func_input=test_input)
427
if not test_output.success:
@@ -685,6 +687,7 @@ def main(args: argparse.Namespace):
687
selected_percentiles=[
688
float(p) for p in args.metric_percentiles.split(",")
689
],
690
+ ignore_eos=args.ignore_eos,
691
))
692
693
# Save config and results to json
@@ -863,6 +866,11 @@ if __name__ == "__main__":
866
"{backend}-{args.request_rate}qps-{base_model_id}-{current_dt}.json"
867
" format.",
868
)
869
+ parser.add_argument(
870
+ "--ignore-eos",
871
+ action="store_true",
872
+ help="Set ignore_eos flag when sending the benchmark request."
873
+ "Warning: ignore_eos is not supported in deepspeed_mii and tgi.")
874
parser.add_argument(
875
"--percentile-metrics",
876
type=str,