@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / fbb74420e7
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 10:45 UTC

[CI] Update performance benchmark: upgrade trt-llm to r24.07, and add SGLang (#7412)

Kuntai Du committed Oct 4, 2024 at 14:01 UTC fbb74420e7018bf0cc1bc81e6fd71a2392347227
18 files changed +1152 -1276
.buildkite/nightly-benchmarks/nightly-annotation.md new
+28
@@ -0,0 +1,28 @@
1 +
2 +## Description
3 +
4 +This file contains the downloading link for benchmarking results.
5 +
6 +- [benchmarking pipeline](artifact://nightly-pipeline.yaml)
7 +- [benchmarking results](artifact://results.zip)
8 +- [benchmarking code](artifact://nightly-benchmarks.zip)
9 +
10 +Please download the visualization scripts in the post
11 +
12 +
13 +## Results reproduction
14 +
15 +- Find the docker we use in `benchmarking pipeline`
16 +- Deploy the docker, and inside the docker:
17 + - Download `nightly-benchmarks.zip`.
18 + - In the same folder, run the following code
19 +```
20 +export HF_TOKEN=<your HF token>
21 +apt update
22 +apt install -y git
23 +unzip nightly-benchmarks.zip
24 +VLLM_SOURCE_CODE_LOC=./ bash .buildkite/nightly-benchmarks/scripts/run-nightly-benchmarks.sh
25 +```
26 +
27 +And the results will be inside `./benchmarks/results`.
28 +
.buildkite/nightly-benchmarks/nightly-descriptions.md
+36 -42
@@ -1,45 +1,39 @@
1
2 # Nightly benchmark
3
4 -The main goal of this benchmarking is two-fold:
5 -- Performance clarity: Provide clarity on which one (vllm, tensorrt-llm, lmdeploy and tgi) leads in performance in what workload.
6 -- Reproducible: one can run the exact same set of benchmarking commands inside the exact same docker by following reproducing instructions in [reproduce.md]().
7 -
8 -
9 -## Docker images
10 -
11 -We benchmark vllm, tensorrt-llm, lmdeploy and tgi using the following docker images:
12 -- vllm/vllm-openai:v0.5.0.post1
13 -- nvcr.io/nvidia/tritonserver:24.04-trtllm-python-py3
14 -- openmmlab/lmdeploy:v0.5.0
15 -- ghcr.io/huggingface/text-generation-inference:2.1
16 -
17 -<!-- Please check <a href="artifact://workspace/build/buildkite/vllm/performance-benchmark/.buildkite/nightly-benchmarks/nightly-pipeline.yaml">nightly-pipeline.yaml</a> artifact for more details on how we deploy the docker images. -->
18 -
19 -
20 -## Hardware
21 -
22 -One AWS node with 8x NVIDIA A100 GPUs.
23 -
24 -
25 -## Workload description
26 -
27 -We benchmark vllm, tensorrt-llm, lmdeploy and tgi using the following workload:
28 -
29 -- Input length: randomly sample 500 prompts from ShareGPT dataset (with fixed random seed).
30 -- Output length: the corresponding output length of these 500 prompts.
31 -- Models: llama-3 8B, llama-3 70B, mixtral 8x7B.
32 -- Average QPS (query per second): 4 for the small model (llama-3 8B) and 2 for other two models. For each QPS, the arrival time of each query is determined using a random Poisson process (with fixed random seed).
33 -- Evaluation metrics: Throughput (higher the better), TTFT (time to the first token, lower the better), ITL (inter-token latency, lower the better).
34 -
35 -<!-- Check <a href="artifact://workspace/build/buildkite/vllm/performance-benchmark/.buildkite/nightly-benchmarks/tests/nightly-tests.json">nightly-tests.json</a> artifact for more details. -->
36 -
37 -## Plots
38 -
39 -In the following plots, the dot shows the mean and the error bar shows the standard error of the mean. Value 0 means that the corresponding benchmark crashed.
40 -
41 -<img src="artifact://nightly_results.png" alt="Benchmarking results" height=250 >
42 -
43 -## Results
44 -
45 -{nightly_results_benchmarking_table}
4 +This benchmark aims to:
5 +- Provide performance clarity: Provide clarity on which one (vllm, tensorrt-llm, lmdeploy and SGLang) leads in performance in what workload.
6 +- Be reproducible: one can run the exact same set of benchmarking commands inside the exact same docker by following reproducing instructions.
7 +
8 +Latest results: [results link](https://blog.vllm.ai/2024/09/05/perf-update.html), scroll to the end.
9 +
10 +Latest reproduction guilde: [github issue link](https://github.com/vllm-project/vllm/issues/8176)
11 +
12 +
13 +## Setup
14 +
15 +- Docker images:
16 + - vLLM: `vllm/vllm-openai:v0.6.2`
17 + - SGLang: `lmsysorg/sglang:v0.3.2-cu121`
18 + - LMDeploy: `openmmlab/lmdeploy:v0.6.1-cu12`
19 + - TensorRT-LLM: `nvcr.io/nvidia/tritonserver:24.07-trtllm-python-py3`
20 + - *NOTE: we uses r24.07 as the current implementation only works for this version. We are going to bump this up.*
21 + - Check [nightly-pipeline.yaml](nightly-pipeline.yaml) for the concrete docker images, specs and commands we use for the benchmark.
22 +- Hardware
23 + - 8x Nvidia A100 GPUs
24 +- Workload:
25 + - Dataset
26 + - ShareGPT dataset
27 + - Prefill-heavy dataset (in average 462 input tokens, 16 tokens as output)
28 + - Decode-heavy dataset (in average 462 input tokens, 256 output tokens)
29 + - Check [nightly-tests.json](tests/nightly-tests.json) for the concrete configuration of datasets we use.
30 + - Models: llama-3 8B, llama-3 70B.
31 + - We do not use llama 3.1 as it is incompatible with trt-llm r24.07. ([issue](https://github.com/NVIDIA/TensorRT-LLM/issues/2105)).
32 + - Average QPS (query per second): 2, 4, 8, 16, 32 and inf.
33 + - Queries are randomly sampled, and arrival patterns are determined via Poisson process, but all with fixed random seed.
34 + - Evaluation metrics: Throughput (higher the better), TTFT (time to the first token, lower the better), ITL (inter-token latency, lower the better).
35 +
36 +# Known issues
37 +
38 +- TRT-LLM crashes with Llama 3.1 8B [issue](https://github.com/NVIDIA/TensorRT-LLM/issues/2105).
39 +- TGI does not support `ignore-eos` flag.
\ No newline at end of file
.buildkite/nightly-benchmarks/nightly-pipeline.yaml
+87 -11
@@ -13,7 +13,7 @@ common_pod_spec: &common_pod_spec
13
14 common_container_settings: &common_container_settings
15 command:
16 - - bash .buildkite/nightly-benchmarks/run-nightly-suite.sh
16 + - bash .buildkite/nightly-benchmarks/scripts/run-nightly-benchmarks.sh
17 resources:
18 limits:
19 nvidia.com/gpu: 8
@@ -37,7 +37,10 @@ common_container_settings: &common_container_settings
37
38 steps:
39 - block: ":rocket: Ready for comparing vllm against alternatives? This will take 4 hours."
40 - - label: "A100 trt benchmark"
40 +
41 +
42 +
43 + - label: "A100 vllm step 10"
44 priority: 100
45 agents:
46 queue: A100
@@ -46,7 +49,21 @@ steps:
49 podSpec:
50 <<: *common_pod_spec
51 containers:
49 - - image: nvcr.io/nvidia/tritonserver:24.04-trtllm-python-py3
52 + - image: vllm/vllm-openai:v0.6.2
53 + <<: *common_container_settings
54 +
55 +
56 +
57 + - label: "A100 sglang benchmark"
58 + priority: 100
59 + agents:
60 + queue: A100
61 + plugins:
62 + - kubernetes:
63 + podSpec:
64 + <<: *common_pod_spec
65 + containers:
66 + - image: lmsysorg/sglang:v0.3.2-cu121
67 <<: *common_container_settings
68
69 - label: "A100 lmdeploy benchmark"
@@ -58,11 +75,13 @@ steps:
75 podSpec:
76 <<: *common_pod_spec
77 containers:
61 - - image: openmmlab/lmdeploy:v0.5.0
78 + - image: openmmlab/lmdeploy:v0.6.1-cu12
79 <<: *common_container_settings
63 -
80
65 - - label: "A100 vllm benchmark"
81 +
82 +
83 +
84 + - label: "A100 trt llama-8B"
85 priority: 100
86 agents:
87 queue: A100
@@ -71,10 +90,25 @@ steps:
90 podSpec:
91 <<: *common_pod_spec
92 containers:
74 - - image: vllm/vllm-openai:latest
93 + - image: nvcr.io/nvidia/tritonserver:24.07-trtllm-python-py3
94 <<: *common_container_settings
95 + env:
96 + - name: VLLM_USAGE_SOURCE
97 + value: ci-test
98 + - name: HF_HOME
99 + value: /root/.cache/huggingface
100 + - name: VLLM_SOURCE_CODE_LOC
101 + value: /workspace/build/buildkite/vllm/performance-benchmark
102 + - name: HF_TOKEN
103 + valueFrom:
104 + secretKeyRef:
105 + name: hf-token-secret
106 + key: token
107 + - name: TEST_SELECTOR
108 + value: "llama8B"
109
77 - - label: "A100 tgi benchmark"
110 +
111 + - label: "A100 trt llama-70B"
112 priority: 100
113 agents:
114 queue: A100
@@ -83,12 +117,54 @@ steps:
117 podSpec:
118 <<: *common_pod_spec
119 containers:
86 - - image: ghcr.io/huggingface/text-generation-inference:2.1
120 + - image: nvcr.io/nvidia/tritonserver:24.07-trtllm-python-py3
121 <<: *common_container_settings
122 + env:
123 + - name: VLLM_USAGE_SOURCE
124 + value: ci-test
125 + - name: HF_HOME
126 + value: /root/.cache/huggingface
127 + - name: VLLM_SOURCE_CODE_LOC
128 + value: /workspace/build/buildkite/vllm/performance-benchmark
129 + - name: HF_TOKEN
130 + valueFrom:
131 + secretKeyRef:
132 + name: hf-token-secret
133 + key: token
134 + - name: TEST_SELECTOR
135 + value: "llama70B"
136 +
137 +
138 + # FIXME(Kuntai): uncomment this after NVIDIA gives us their test docker image
139 + # - label: "A100 trt benchmark"
140 + # priority: 100
141 + # agents:
142 + # queue: A100
143 + # plugins:
144 + # - kubernetes:
145 + # podSpec:
146 + # <<: *common_pod_spec
147 + # containers:
148 + # - image: nvcr.io/nvidia/tritonserver:24.07-trtllm-python-py3
149 + # <<: *common_container_settings
150 +
151 +
152 + # FIXME(Kuntai): uncomment this after TGI supports `--ignore-eos`.
153 + # - label: "A100 tgi benchmark"
154 + # priority: 100
155 + # agents:
156 + # queue: A100
157 + # plugins:
158 + # - kubernetes:
159 + # podSpec:
160 + # <<: *common_pod_spec
161 + # containers:
162 + # - image: ghcr.io/huggingface/text-generation-inference:2.2.0
163 + # <<: *common_container_settings
164
165 - wait
166
91 - - label: "Plot"
167 + - label: "Collect the results"
168 priority: 100
169 agents:
170 queue: A100
@@ -117,4 +193,4 @@ steps:
193 name: hf-token-secret
194 key: token
195
120 - - wait
\ No newline at end of file
196 + - block: ":rocket: check the results!"
\ No newline at end of file
.buildkite/nightly-benchmarks/run-nightly-suite.sh deleted
-76
@@ -1,76 +0,0 @@
1 -#!/bin/bash
2 -
3 -set -o pipefail
4 -set -x
5 -
6 -check_gpus() {
7 - # check the number of GPUs and GPU type.
8 - declare -g gpu_count=$(nvidia-smi --list-gpus | wc -l)
9 - if [[ $gpu_count -gt 0 ]]; then
10 - echo "GPU found."
11 - else
12 - echo "Need at least 1 GPU to run benchmarking."
13 - exit 1
14 - fi
15 - declare -g gpu_type=$(echo $(nvidia-smi --query-gpu=name --format=csv,noheader) | awk '{print $2}')
16 - echo "GPU type is $gpu_type"
17 -}
18 -
19 -check_hf_token() {
20 - # check if HF_TOKEN is available and valid
21 - if [[ -z "$HF_TOKEN" ]]; then
22 - echo "Error: HF_TOKEN is not set."
23 - exit 1
24 - elif [[ ! "$HF_TOKEN" =~ ^hf_ ]]; then
25 - echo "Error: HF_TOKEN does not start with 'hf_'."
26 - exit 1
27 - else
28 - echo "HF_TOKEN is set and valid."
29 - fi
30 -}
31 -
32 -main() {
33 -
34 - check_gpus
35 - check_hf_token
36 -
37 - df -h
38 -
39 - (which wget && which curl) || (apt-get update && apt-get install -y wget curl)
40 - (which jq) || (apt-get update && apt-get -y install jq)
41 -
42 - cd $VLLM_SOURCE_CODE_LOC/benchmarks
43 - wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
44 -
45 -
46 - # run lmdeploy
47 - if which lmdeploy >/dev/null; then
48 - echo "lmdeploy is available, redirect to run-lmdeploy-nightly.sh"
49 - bash ../.buildkite/nightly-benchmarks/scripts/run-lmdeploy-nightly.sh
50 - exit 0
51 - fi
52 -
53 - # run tgi
54 - if [ -e /tgi-entrypoint.sh ]; then
55 - echo "tgi is available, redirect to run-tgi-nightly.sh"
56 - bash ../.buildkite/nightly-benchmarks/scripts/run-tgi-nightly.sh
57 - exit 0
58 - fi
59 -
60 - # run trt
61 - if which trtllm-build >/dev/null; then
62 - echo "trtllm is available, redirect to run-trt-nightly.sh"
63 - bash ../.buildkite/nightly-benchmarks/scripts/run-trt-nightly.sh
64 - exit 0
65 - fi
66 -
67 - # run vllm
68 - if [ -e /vllm-workspace ]; then
69 - echo "vllm is available, redirect to run-vllm-nightly.sh"
70 - bash ../.buildkite/nightly-benchmarks/scripts/run-vllm-nightly.sh
71 - exit 0
72 - fi
73 -
74 -}
75 -
76 -main "$@"
\ No newline at end of file
.buildkite/nightly-benchmarks/scripts/generate-nightly-markdown.py new
+95
@@ -0,0 +1,95 @@
1 +import argparse
2 +import json
3 +from pathlib import Path
4 +
5 +import numpy as np
6 +import pandas as pd
7 +from tabulate import tabulate
8 +
9 +
10 +def parse_arguments():
11 + parser = argparse.ArgumentParser(
12 + description=
13 + 'Parse command line arguments for summary-nightly-results script.')
14 + parser.add_argument('--results-folder',
15 + type=str,
16 + required=True,
17 + help='The folder where the results are stored.')
18 + parser.add_argument('--description',
19 + type=str,
20 + required=True,
21 + help='Description of the results.')
22 +
23 + args = parser.parse_args()
24 + return args
25 +
26 +
27 +def get_perf(df, method, model, metric):
28 +
29 + means = []
30 +
31 + for qps in [2, 4, 8, 16, "inf"]:
32 + target = df['Test name'].str.contains(model)
33 + target = target & df['Engine'].str.contains(method)
34 + target = target & df['Test name'].str.contains("qps_" + str(qps))
35 + filtered_df = df[target]
36 +
37 + if filtered_df.empty:
38 + means.append(0.)
39 + else:
40 + means.append(filtered_df[metric].values[0])
41 +
42 + return np.array(means)
43 +
44 +
45 +def get_perf_w_std(df, method, model, metric):
46 +
47 + if metric in ["TTFT", "ITL"]:
48 + mean = get_perf(df, method, model, "Mean " + metric + " (ms)")
49 + mean = mean.tolist()
50 + std = get_perf(df, method, model, "Std " + metric + " (ms)")
51 + if std.mean() == 0:
52 + std = None
53 + success = get_perf(df, method, model, "Successful req.")
54 + if std is not None:
55 + std = std / np.sqrt(success)
56 + std = std.tolist()
57 +
58 + else:
59 + assert metric == "Tput"
60 + mean = get_perf(df, method, model, "Input Tput (tok/s)") + get_perf(
61 + df, method, model, "Output Tput (tok/s)")
62 + mean = mean.tolist()
63 + std = None
64 +
65 + return mean, std
66 +
67 +
68 +def main(args):
69 + results_folder = Path(args.results_folder)
70 +
71 + results = []
72 +
73 + # collect results
74 + for test_file in results_folder.glob("*_nightly_results.json"):
75 + with open(test_file, "r") as f:
76 + results = results + json.loads(f.read())
77 +
78 + # generate markdown table
79 + df = pd.DataFrame.from_dict(results)
80 +
81 + md_table = tabulate(df, headers='keys', tablefmt='pipe', showindex=False)
82 +
83 + with open(args.description, "r") as f:
84 + description = f.read()
85 +
86 + description = description.format(
87 + nightly_results_benchmarking_table=md_table)
88 +
89 + with open("nightly_results.md", "w") as f:
90 + f.write(description)
91 +
92 +
93 +if __name__ == '__main__':
94 + args = parse_arguments()
95 + main(args)
.buildkite/nightly-benchmarks/scripts/launch-server.sh new
+241
@@ -0,0 +1,241 @@
1 +#!/bin/bash
2 +
3 +# Currently FP8 benchmark is NOT enabled.
4 +
5 +set -x
6 +server_params=$1
7 +common_params=$2
8 +
9 +json2args() {
10 + # transforms the JSON string to command line args, and '_' is replaced to '-'
11 + # example:
12 + # input: { "model": "meta-llama/Llama-2-7b-chat-hf", "tensor_parallel_size": 1 }
13 + # output: --model meta-llama/Llama-2-7b-chat-hf --tensor-parallel-size 1
14 + local json_string=$1
15 + local args=$(
16 + echo "$json_string" | jq -r '
17 + to_entries |
18 + map("--" + (.key | gsub("_"; "-")) + " " + (.value | tostring)) |
19 + join(" ")
20 + '
21 + )
22 + echo "$args"
23 +}
24 +
25 +launch_trt_server() {
26 +
27 + model_path=$(echo "$common_params" | jq -r '.model')
28 + model_name="${model_path#*/}"
29 + model_type=$(echo "$server_params" | jq -r '.model_type')
30 + model_dtype=$(echo "$server_params" | jq -r '.model_dtype')
31 + model_tp_size=$(echo "$common_params" | jq -r '.tp')
32 + max_batch_size=$(echo "$server_params" | jq -r '.max_batch_size')
33 + max_input_len=$(echo "$server_params" | jq -r '.max_input_len')
34 + max_seq_len=$(echo "$server_params" | jq -r '.max_seq_len')
35 + max_num_tokens=$(echo "$server_params" | jq -r '.max_num_tokens')
36 + trt_llm_version=$(echo "$server_params" | jq -r '.trt_llm_version')
37 +
38 + # create model caching directory
39 + cd ~
40 + rm -rf models
41 + mkdir -p models
42 + cd models
43 + models_dir=$(pwd)
44 + trt_model_path=${models_dir}/${model_name}-trt-ckpt
45 + trt_engine_path=${models_dir}/${model_name}-trt-engine
46 +
47 + # clone tensorrt backend
48 + cd /
49 + rm -rf tensorrtllm_backend
50 + git clone https://github.com/triton-inference-server/tensorrtllm_backend.git
51 + git lfs install
52 + cd tensorrtllm_backend
53 + git checkout $trt_llm_version
54 + tensorrtllm_backend_dir=$(pwd)
55 + git submodule update --init --recursive
56 +
57 + # build trtllm engine
58 + cd /tensorrtllm_backend
59 + cd ./tensorrt_llm/examples/${model_type}
60 + python3 convert_checkpoint.py \
61 + --model_dir ${model_path} \
62 + --dtype ${model_dtype} \
63 + --tp_size ${model_tp_size} \
64 + --output_dir ${trt_model_path}
65 + trtllm-build \
66 + --checkpoint_dir ${trt_model_path} \
67 + --use_fused_mlp \
68 + --reduce_fusion disable \
69 + --workers 8 \
70 + --gpt_attention_plugin ${model_dtype} \
71 + --gemm_plugin ${model_dtype} \
72 + --tp_size ${model_tp_size} \
73 + --max_batch_size ${max_batch_size} \
74 + --max_input_len ${max_input_len} \
75 + --max_seq_len ${max_seq_len} \
76 + --max_num_tokens ${max_num_tokens} \
77 + --output_dir ${trt_engine_path}
78 +
79 + # handle triton protobuf files and launch triton server
80 + cd /tensorrtllm_backend
81 + mkdir triton_model_repo
82 + cp -r all_models/inflight_batcher_llm/* triton_model_repo/
83 + cd triton_model_repo
84 + rm -rf ./tensorrt_llm/1/*
85 + cp -r ${trt_engine_path}/* ./tensorrt_llm/1
86 + python3 ../tools/fill_template.py -i tensorrt_llm/config.pbtxt triton_backend:tensorrtllm,engine_dir:/tensorrtllm_backend/triton_model_repo/tensorrt_llm/1,decoupled_mode:true,batching_strategy:inflight_fused_batching,batch_scheduler_policy:guaranteed_no_evict,exclude_input_in_output:true,triton_max_batch_size:2048,max_queue_delay_microseconds:0,max_beam_width:1,max_queue_size:2048,enable_kv_cache_reuse:false
87 + python3 ../tools/fill_template.py -i preprocessing/config.pbtxt triton_max_batch_size:2048,tokenizer_dir:$model_path,preprocessing_instance_count:5
88 + python3 ../tools/fill_template.py -i postprocessing/config.pbtxt triton_max_batch_size:2048,tokenizer_dir:$model_path,postprocessing_instance_count:5,skip_special_tokens:false
89 + python3 ../tools/fill_template.py -i ensemble/config.pbtxt triton_max_batch_size:$max_batch_size
90 + python3 ../tools/fill_template.py -i tensorrt_llm_bls/config.pbtxt triton_max_batch_size:$max_batch_size,decoupled_mode:true,accumulate_tokens:"False",bls_instance_count:1
91 + cd /tensorrtllm_backend
92 + python3 scripts/launch_triton_server.py \
93 + --world_size=${model_tp_size} \
94 + --model_repo=/tensorrtllm_backend/triton_model_repo &
95 +
96 +}
97 +
98 +launch_tgi_server() {
99 + model=$(echo "$common_params" | jq -r '.model')
100 + tp=$(echo "$common_params" | jq -r '.tp')
101 + dataset_name=$(echo "$common_params" | jq -r '.dataset_name')
102 + dataset_path=$(echo "$common_params" | jq -r '.dataset_path')
103 + port=$(echo "$common_params" | jq -r '.port')
104 + num_prompts=$(echo "$common_params" | jq -r '.num_prompts')
105 + server_args=$(json2args "$server_params")
106 +
107 + if echo "$common_params" | jq -e 'has("fp8")' >/dev/null; then
108 + echo "Key 'fp8' exists in common params."
109 + server_command="/tgi-entrypoint.sh \
110 + --model-id $model \
111 + --num-shard $tp \
112 + --port $port \
113 + --quantize fp8 \
114 + $server_args"
115 + else
116 + echo "Key 'fp8' does not exist in common params."
117 + server_command="/tgi-entrypoint.sh \
118 + --model-id $model \
119 + --num-shard $tp \
120 + --port $port \
121 + $server_args"
122 + fi
123 +
124 + echo "Server command: $server_command"
125 + eval "$server_command" &
126 +
127 +}
128 +
129 +launch_lmdeploy_server() {
130 + model=$(echo "$common_params" | jq -r '.model')
131 + tp=$(echo "$common_params" | jq -r '.tp')
132 + dataset_name=$(echo "$common_params" | jq -r '.dataset_name')
133 + dataset_path=$(echo "$common_params" | jq -r '.dataset_path')
134 + port=$(echo "$common_params" | jq -r '.port')
135 + num_prompts=$(echo "$common_params" | jq -r '.num_prompts')
136 + server_args=$(json2args "$server_params")
137 +
138 + server_command="lmdeploy serve api_server $model \
139 + --tp $tp \
140 + --server-port $port \
141 + $server_args"
142 +
143 + # run the server
144 + echo "Server command: $server_command"
145 + bash -c "$server_command" &
146 +}
147 +
148 +launch_sglang_server() {
149 +
150 + model=$(echo "$common_params" | jq -r '.model')
151 + tp=$(echo "$common_params" | jq -r '.tp')
152 + dataset_name=$(echo "$common_params" | jq -r '.dataset_name')
153 + dataset_path=$(echo "$common_params" | jq -r '.dataset_path')
154 + port=$(echo "$common_params" | jq -r '.port')
155 + num_prompts=$(echo "$common_params" | jq -r '.num_prompts')
156 + server_args=$(json2args "$server_params")
157 +
158 + if echo "$common_params" | jq -e 'has("fp8")' >/dev/null; then
159 + echo "Key 'fp8' exists in common params. Use neuralmagic fp8 model for convenience."
160 + model=$(echo "$common_params" | jq -r '.neuralmagic_quantized_model')
161 + server_command="python3 \
162 + -m sglang.launch_server \
163 + --tp $tp \
164 + --model-path $model \
165 + --port $port \
166 + $server_args"
167 + else
168 + echo "Key 'fp8' does not exist in common params."
169 + server_command="python3 \
170 + -m sglang.launch_server \
171 + --tp $tp \
172 + --model-path $model \
173 + --port $port \
174 + $server_args"
175 + fi
176 +
177 + # run the server
178 + echo "Server command: $server_command"
179 + eval "$server_command" &
180 +}
181 +
182 +launch_vllm_server() {
183 +
184 + export VLLM_HOST_IP=$(hostname -I | awk '{print $1}')
185 +
186 + model=$(echo "$common_params" | jq -r '.model')
187 + tp=$(echo "$common_params" | jq -r '.tp')
188 + dataset_name=$(echo "$common_params" | jq -r '.dataset_name')
189 + dataset_path=$(echo "$common_params" | jq -r '.dataset_path')
190 + port=$(echo "$common_params" | jq -r '.port')
191 + num_prompts=$(echo "$common_params" | jq -r '.num_prompts')
192 + server_args=$(json2args "$server_params")
193 +
194 + if echo "$common_params" | jq -e 'has("fp8")' >/dev/null; then
195 + echo "Key 'fp8' exists in common params. Use neuralmagic fp8 model for convenience."
196 + model=$(echo "$common_params" | jq -r '.neuralmagic_quantized_model')
197 + server_command="python3 \
198 + -m vllm.entrypoints.openai.api_server \
199 + -tp $tp \
200 + --model $model \
201 + --port $port \
202 + $server_args"
203 + else
204 + echo "Key 'fp8' does not exist in common params."
205 + server_command="python3 \
206 + -m vllm.entrypoints.openai.api_server \
207 + -tp $tp \
208 + --model $model \
209 + --port $port \
210 + $server_args"
211 + fi
212 +
213 + # run the server
214 + echo "Server command: $server_command"
215 + eval "$server_command" &
216 +}
217 +
218 +main() {
219 +
220 + if [[ $CURRENT_LLM_SERVING_ENGINE == "trt" ]]; then
221 + launch_trt_server
222 + fi
223 +
224 + if [[ $CURRENT_LLM_SERVING_ENGINE == "tgi" ]]; then
225 + launch_tgi_server
226 + fi
227 +
228 + if [[ $CURRENT_LLM_SERVING_ENGINE == "lmdeploy" ]]; then
229 + launch_lmdeploy_server
230 + fi
231 +
232 + if [[ $CURRENT_LLM_SERVING_ENGINE == "sglang" ]]; then
233 + launch_sglang_server
234 + fi
235 +
236 + if [[ "$CURRENT_LLM_SERVING_ENGINE" == *"vllm"* ]]; then
237 + launch_vllm_server
238 + fi
239 +}
240 +
241 +main
.buildkite/nightly-benchmarks/scripts/launch-trt-server.sh deleted
-102
@@ -1,102 +0,0 @@
1 -#!/bin/bash
2 -
3 -
4 -server_params=$1
5 -common_params=$2
6 -
7 -
8 -
9 -model_path=$(echo "$common_params" | jq -r '.model')
10 -model_name="${model_path#*/}"
11 -model_type=$(echo "$server_params" | jq -r '.model_type')
12 -model_dtype=$(echo "$server_params" | jq -r '.model_dtype')
13 -model_tp_size=$(echo "$common_params" | jq -r '.tp')
14 -max_batch_size=$(echo "$server_params" | jq -r '.max_batch_size')
15 -max_input_len=$(echo "$server_params" | jq -r '.max_input_len')
16 -max_output_len=$(echo "$server_params" | jq -r '.max_output_len')
17 -trt_llm_version=$(echo "$server_params" | jq -r '.trt_llm_version')
18 -
19 -cd ~
20 -rm -rf models
21 -mkdir -p models
22 -cd models
23 -models_dir=$(pwd)
24 -trt_model_path=${models_dir}/${model_name}-trt-ckpt
25 -trt_engine_path=${models_dir}/${model_name}-trt-engine
26 -
27 -cd ~
28 -rm -rf tensorrt-demo
29 -git clone https://github.com/neuralmagic/tensorrt-demo.git
30 -cd tensorrt-demo
31 -tensorrt_demo_dir=$(pwd)
32 -
33 -# make sure the parameter inside tensorrt_demo is consistent to envvar
34 -sed -i.bak "/key: \"tokenizer_dir\"/,/string_value:/s|string_value: \".*\"|string_value: \"$model_path\"|" ./triton_model_repo/postprocessing/config.pbtxt
35 -sed -i.bak "/key: \"tokenizer_dir\"/,/string_value:/s|string_value: \".*\"|string_value: \"$model_path\"|" ./triton_model_repo/preprocessing/config.pbtxt
36 -sed -i.bak "s|\(max_batch_size:\s*\)[0-9]*|\1$max_batch_size|g" ./triton_model_repo/ensemble/config.pbtxt
37 -sed -i.bak "s|\(max_batch_size:\s*\)[0-9]*|\1$max_batch_size|g" ./triton_model_repo/preprocessing/config.pbtxt
38 -sed -i.bak "s|\(max_batch_size:\s*\)[0-9]*|\1$max_batch_size|g" ./triton_model_repo/postprocessing/config.pbtxt
39 -sed -i.bak "s|\(max_batch_size:\s*\)[0-9]*|\1$max_batch_size|g" ./triton_model_repo/tensorrt_llm_bls/config.pbtxt
40 -
41 -
42 -cd /
43 -rm -rf tensorrtllm_backend
44 -git clone https://github.com/triton-inference-server/tensorrtllm_backend.git
45 -git lfs install
46 -cd tensorrtllm_backend
47 -git checkout $trt_llm_version
48 -tensorrtllm_backend_dir=$(pwd)
49 -git submodule update --init --recursive
50 -cp -r ${tensorrt_demo_dir}/triton_model_repo ${tensorrtllm_backend_dir}/
51 -
52 -cd /tensorrtllm_backend
53 -cd ./tensorrt_llm/examples/${model_type}
54 -
55 -
56 -if echo "$common_params" | jq -e 'has("fp8")' > /dev/null; then
57 -
58 - echo "Key 'fp8' exists in common params. Use quantize.py instead of convert_checkpoint.py"
59 - echo "Reference: https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/llama/README.md"
60 - python ../quantization/quantize.py \
61 - --model_dir ${model_path} \
62 - --dtype ${model_dtype} \
63 - --tp_size ${model_tp_size} \
64 - --output_dir ${trt_model_path} \
65 - --qformat fp8 \
66 - --kv_cache_dtype fp8 \
67 - --calib_size 2
68 -
69 -else
70 -
71 - echo "Key 'fp8' does not exist in common params. Use convert_checkpoint.py"
72 - python3 convert_checkpoint.py \
73 - --model_dir ${model_path} \
74 - --dtype ${model_dtype} \
75 - --tp_size ${model_tp_size} \
76 - --output_dir ${trt_model_path}
77 -
78 -fi
79 -
80 -
81 -
82 -trtllm-build \
83 ---checkpoint_dir=${trt_model_path} \
84 ---gpt_attention_plugin=${model_dtype} \
85 ---gemm_plugin=${model_dtype} \
86 ---remove_input_padding=enable \
87 ---paged_kv_cache=enable \
88 ---tp_size=${model_tp_size} \
89 ---max_batch_size=${max_batch_size} \
90 ---max_input_len=${max_input_len} \
91 ---max_output_len=${max_output_len} \
92 ---max_num_tokens=${max_output_len} \
93 ---opt_num_tokens=${max_output_len} \
94 ---output_dir=${trt_engine_path}
95 -
96 -cd /tensorrtllm_backend/triton_model_repo
97 -rm -rf ./tensorrt_llm/1/*
98 -cp -r ${trt_engine_path}/* ./tensorrt_llm/1
99 -cd /tensorrtllm_backend
100 -python3 scripts/launch_triton_server.py \
101 ---world_size=${model_tp_size} \
102 ---model_repo=/tensorrtllm_backend/triton_model_repo &
\ No newline at end of file
.buildkite/nightly-benchmarks/scripts/nightly-annotate.sh
+48 -10
@@ -8,6 +8,7 @@ main() {
8
9 (which wget && which curl) || (apt-get update && apt-get install -y wget curl)
10 (which jq) || (apt-get update && apt-get -y install jq)
11 + (which zip) || (apt-get install -y zip)
12
13 if [ ! -f /workspace/buildkite-agent ]; then
14 echo "buildkite-agent binary not found. Skip plotting the results."
@@ -24,17 +25,54 @@ main() {
25 ls
26 ls results/
27
27 - # generate figures
28 - python3 -m pip install tabulate pandas matplotlib
29 - python3 $VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/scripts/plot-nightly-results.py \
30 - --description $description \
31 - --results-folder results/
28 + # upload benchmark results
29 + zip -r results.zip results/
30 + /workspace/buildkite-agent artifact upload "results.zip"
31 +
32 + # upload benchmarking scripts
33 + cd $VLLM_SOURCE_CODE_LOC/
34 + zip -r nightly-benchmarks.zip .buildkite/ benchmarks/
35 + /workspace/buildkite-agent artifact upload "nightly-benchmarks.zip"
36 +
37 + cd $VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/
38 + # upload benchmarking pipeline
39 + /workspace/buildkite-agent artifact upload "nightly-pipeline.yaml"
40 +
41 + cd $VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/
42 + /workspace/buildkite-agent annotate --style "success" --context "nightly-benchmarks-results" --append < nightly-annotation.md
43 +
44 +
45 +
46 + # The figures should be genereated by a separate process outside the CI/CD pipeline
47 +
48 + # # generate figures
49 + # python3 -m pip install tabulate pandas matplotlib
50 +
51 + # python3 $VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/scripts/generate-nightly-markdown.py \
52 + # --description $description \
53 + # --results-folder results/
54 +
55 +
56 + # python3 $VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/scripts/plot-nightly-results.py \
57 + # --description $description \
58 + # --results-folder results/ \
59 + # --dataset sharegpt
60 +
61 + # python3 $VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/scripts/plot-nightly-results.py \
62 + # --description $description \
63 + # --results-folder results/ \
64 + # --dataset sonnet_2048_128
65 +
66 + # python3 $VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/scripts/plot-nightly-results.py \
67 + # --description $description \
68 + # --results-folder results/ \
69 + # --dataset sonnet_128_2048
70
33 - # upload results and figures
34 - /workspace/buildkite-agent artifact upload "nightly_results.png"
35 - /workspace/buildkite-agent artifact upload $VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/nightly-pipeline.yaml
36 - /workspace/buildkite-agent artifact upload $VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/tests/nightly-tests.json
37 - /workspace/buildkite-agent annotate --style "success" --context "nightly-benchmarks-results" --append < nightly_results.md
71 + # # upload results and figures
72 + # /workspace/buildkite-agent artifact upload "nightly_results*.png"
73 + # /workspace/buildkite-agent artifact upload $VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/nightly-pipeline.yaml
74 + # /workspace/buildkite-agent artifact upload $VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/tests/nightly-tests.json
75 + # /workspace/buildkite-agent annotate --style "success" --context "nightly-benchmarks-results" --append < nightly_results.md
76 }
77
78 main "$@"
\ No newline at end of file
.buildkite/nightly-benchmarks/scripts/plot-nightly-results.py deleted
-135
@@ -1,135 +0,0 @@
1 -import argparse
2 -import json
3 -import math
4 -from pathlib import Path
5 -
6 -import matplotlib.pyplot as plt
7 -import pandas as pd
8 -from tabulate import tabulate
9 -
10 -
11 -def parse_arguments():
12 - parser = argparse.ArgumentParser(
13 - description=
14 - 'Parse command line arguments for summary-nightly-results script.')
15 - parser.add_argument('--results-folder',
16 - type=str,
17 - required=True,
18 - help='The folder where the results are stored.')
19 - parser.add_argument('--description',
20 - type=str,
21 - required=True,
22 - help='Description of the results.')
23 -
24 - args = parser.parse_args()
25 - return args
26 -
27 -
28 -def main(args):
29 - bar_colors = ['#56B4E9', '#009E73', '#D55E00', '#E69F00']
30 - results_folder = Path(args.results_folder)
31 -
32 - results = []
33 -
34 - # collect results
35 - for test_file in results_folder.glob("*_nightly_results.json"):
36 - with open(test_file, "r") as f:
37 - results = results + json.loads(f.read())
38 -
39 - # generate markdown table
40 - df = pd.DataFrame.from_dict(results)
41 -
42 - md_table = tabulate(df, headers='keys', tablefmt='pipe', showindex=False)
43 -
44 - with open(args.description, "r") as f:
45 - description = f.read()
46 -
47 - description = description.format(
48 - nightly_results_benchmarking_table=md_table)
49 -
50 - with open("nightly_results.md", "w") as f:
51 - f.write(description)
52 -
53 - plt.rcParams.update({'font.size': 20})
54 -
55 - # plot results
56 - fig, axes = plt.subplots(3, 3, figsize=(16, 14))
57 - fig.subplots_adjust(hspace=1)
58 - methods = ["vllm", "trt", "lmdeploy", "tgi"]
59 - for i, model in enumerate(["llama8B", "llama70B", "mixtral8x7B"]):
60 - for j, metric in enumerate(["TTFT", "ITL"]):
61 - means, stds = [], []
62 - for method in methods:
63 - target = df['Test name'].str.contains(model)
64 - target = target & df['Engine'].str.contains(method)
65 - filtered_df = df[target]
66 -
67 - if filtered_df.empty:
68 - means.append(0.)
69 - stds.append(0.)
70 - else:
71 - means.append(filtered_df[f"Mean {metric} (ms)"].values[0])
72 - std = filtered_df[f"Std {metric} (ms)"].values[0]
73 - success = filtered_df["Successful req."].values[0]
74 - stds.append(std / math.sqrt(success))
75 -
76 - print(model, metric)
77 - print(means, stds)
78 -
79 - ax = axes[i, j + 1]
80 -
81 - bars = ax.bar(
82 - ["vllm", "trt", "lmdeploy", "tgi"],
83 - means,
84 - yerr=stds,
85 - capsize=10,
86 - )
87 - for idx, bar in enumerate(bars):
88 - bar.set_color(bar_colors[idx])
89 - ax.set_ylim(bottom=0)
90 -
91 - ax.set_ylabel(f"{metric} (ms)")
92 - ax.set_title(f"{model} {metric}")
93 - ax.grid(axis='y')
94 -
95 - metric = "Tput"
96 - j = 0
97 - if True:
98 - tputs = []
99 - for method in methods:
100 - target = df['Test name'].str.contains(model)
101 - target = target & df['Engine'].str.contains(method)
102 - filtered_df = df[target]
103 -
104 - if filtered_df.empty:
105 - tputs.append(0.)
106 - else:
107 - input_tput = filtered_df["Input Tput (tok/s)"].values[0]
108 - output_tput = filtered_df["Output Tput (tok/s)"].values[0]
109 - tputs.append(input_tput + output_tput)
110 -
111 - print(model, metric)
112 - print(tputs)
113 -
114 - ax = axes[i, j]
115 -
116 - bars = ax.bar(
117 - ["vllm", "trt", "lmdeploy", "tgi"],
118 - tputs,
119 - )
120 - for idx, bar in enumerate(bars):
121 - bar.set_color(bar_colors[idx])
122 -
123 - ax.set_ylim(bottom=0)
124 -
125 - ax.set_ylabel("Tput (token/s)")
126 - ax.set_title(f"{model} {metric}")
127 - ax.grid(axis='y')
128 -
129 - fig.tight_layout()
130 - fig.savefig("nightly_results.png", bbox_inches='tight', dpi=400)
131 -
132 -
133 -if __name__ == '__main__':
134 - args = parse_arguments()
135 - main(args)
.buildkite/nightly-benchmarks/scripts/run-lmdeploy-nightly.sh deleted
-218
@@ -1,218 +0,0 @@
1 -#!/bin/bash
2 -
3 -set -o pipefail
4 -
5 -check_gpus() {
6 - # check the number of GPUs and GPU type.
7 - declare -g gpu_count=$(nvidia-smi --list-gpus | wc -l)
8 - if [[ $gpu_count -gt 0 ]]; then
9 - echo "GPU found."
10 - else
11 - echo "Need at least 1 GPU to run benchmarking."
12 - exit 1
13 - fi
14 - declare -g gpu_type=$(echo $(nvidia-smi --query-gpu=name --format=csv,noheader) | awk '{print $2}')
15 - echo "GPU type is $gpu_type"
16 -}
17 -
18 -kill_gpu_processes() {
19 - pkill lmdeploy || true
20 - # waiting for GPU processes to be fully killed
21 - sleep 10
22 - # Print the GPU memory usage
23 - # so that we know if all GPU processes are killed.
24 - gpu_memory_usage=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits -i 0)
25 - # The memory usage should be 0 MB.
26 - echo "GPU 0 Memory Usage: $gpu_memory_usage MB"
27 -}
28 -
29 -json2args() {
30 - # transforms the JSON string to command line args, and '_' is replaced to '-'
31 - # example:
32 - # input: { "model": "meta-llama/Llama-2-7b-chat-hf", "tensor_parallel_size": 1 }
33 - # output: --model meta-llama/Llama-2-7b-chat-hf --tensor-parallel-size 1
34 - local json_string=$1
35 - local args=$(
36 - echo "$json_string" | jq -r '
37 - to_entries |
38 - map("--" + (.key | gsub("_"; "-")) + " " + (.value | tostring)) |
39 - join(" ")
40 - '
41 - )
42 - echo "$args"
43 -}
44 -
45 -wait_for_server() {
46 - # wait for vllm server to start
47 - # return 1 if vllm server crashes
48 - timeout 1200 bash -c '
49 - until curl -s localhost:8000/v1/completions > /dev/null; do
50 - sleep 1
51 - done' && return 0 || return 1
52 -}
53 -
54 -run_serving_tests() {
55 - # run serving tests using `benchmark_serving.py`
56 - # $1: a json file specifying serving test cases
57 -
58 - local serving_test_file
59 - serving_test_file=$1
60 -
61 - # Iterate over serving tests
62 - jq -c '.[]' "$serving_test_file" | while read -r params; do
63 - # get the test name, and append the GPU type back to it.
64 - test_name=$(echo "$params" | jq -r '.test_name')
65 -
66 - # if TEST_SELECTOR is set, only run the test cases that match the selector
67 - if [[ -n "$TEST_SELECTOR" ]] && [[ ! "$test_name" =~ $TEST_SELECTOR ]]; then
68 - echo "Skip test case $test_name."
69 - continue
70 - fi
71 -
72 - # append lmdeploy to the test name
73 - test_name=lmdeploy_$test_name
74 -
75 - # get common parameters
76 - common_params=$(echo "$params" | jq -r '.common_parameters')
77 - model=$(echo "$common_params" | jq -r '.model')
78 - tp=$(echo "$common_params" | jq -r '.tp')
79 - dataset_name=$(echo "$common_params" | jq -r '.dataset_name')
80 - dataset_path=$(echo "$common_params" | jq -r '.dataset_path')
81 - port=$(echo "$common_params" | jq -r '.port')
82 - num_prompts=$(echo "$common_params" | jq -r '.num_prompts')
83 -
84 -
85 -
86 - # get client and server arguments
87 - server_params=$(echo "$params" | jq -r '.lmdeploy_server_parameters')
88 - client_params=$(echo "$params" | jq -r '.lmdeploy_client_parameters')
89 - server_args=$(json2args "$server_params")
90 - client_args=$(json2args "$client_params")
91 - qps_list=$(echo "$params" | jq -r '.qps_list')
92 - qps_list=$(echo "$qps_list" | jq -r '.[] | @sh')
93 - echo "Running over qps list $qps_list"
94 -
95 - # check if there is enough GPU to run the test
96 - if [[ $gpu_count -lt $tp ]]; then
97 - echo "Required tensor-parallel-size $tp but only $gpu_count GPU found. Skip testcase $test_name."
98 - continue
99 - fi
100 -
101 - # prepare tokenizer
102 - rm -rf /tokenizer_cache
103 - mkdir /tokenizer_cache
104 - python ../.buildkite/nightly-benchmarks/scripts/download-tokenizer.py \
105 - --model "$model" \
106 - --cachedir /tokenizer_cache
107 -
108 - server_command="lmdeploy serve api_server $model \
109 - --tp $tp \
110 - --server-port $port \
111 - $server_args"
112 -
113 - # run the server
114 - echo "Running test case $test_name"
115 - echo "Server command: $server_command"
116 - bash -c "$server_command" &
117 -
118 - # wait until the server is alive
119 - wait_for_server
120 - if [ $? -eq 0 ]; then
121 - echo ""
122 - echo "lmdeploy server is up and running."
123 - else
124 - echo ""
125 - echo "lmdeploy failed to start within the timeout period."
126 - break
127 - fi
128 -
129 - # get model name
130 - model_name=$(python ../.buildkite/nightly-benchmarks/scripts/get-lmdeploy-modelname.py)
131 -
132 - # iterate over different QPS
133 - for qps in $qps_list; do
134 - # remove the surrounding single quote from qps
135 - if [[ "$qps" == *"inf"* ]]; then
136 - echo "qps was $qps"
137 - qps="inf"
138 - echo "now qps is $qps"
139 - fi
140 -
141 - new_test_name=$test_name"_qps_"$qps
142 -
143 - client_command="python3 benchmark_serving.py \
144 - --backend lmdeploy \
145 - --tokenizer /tokenizer_cache \
146 - --dataset-name $dataset_name \
147 - --dataset-path $dataset_path \
148 - --num-prompts $num_prompts \
149 - --port $port \
150 - --save-result \
151 - --result-dir $RESULTS_FOLDER \
152 - --result-filename ${new_test_name}.json \
153 - --request-rate $qps \
154 - --model \"$model_name\" \
155 - $client_args"
156 -
157 - echo "Running test case $test_name with qps $qps"
158 - echo "Client command: $client_command"
159 -
160 - eval "$client_command"
161 -
162 - # record the benchmarking commands
163 - jq_output=$(jq -n \
164 - --arg server "$server_command" \
165 - --arg client "$client_command" \
166 - --arg gpu "$gpu_type" \
167 - --arg engine "lmdeploy" \
168 - '{
169 - server_command: $server,
170 - client_command: $client,
171 - gpu_type: $gpu,
172 - engine: $engine
173 - }')
174 - echo "$jq_output" >"$RESULTS_FOLDER/${new_test_name}.commands"
175 -
176 - done
177 -
178 - # clean up
179 - kill_gpu_processes
180 - rm -rf /root/.cache/huggingface/*
181 - done
182 -}
183 -
184 -
185 -upload_to_buildkite() {
186 - # upload the benchmarking results to buildkite
187 -
188 - # if the agent binary is not found, skip uploading the results, exit 0
189 - if [ ! -f /workspace/buildkite-agent ]; then
190 - echo "buildkite-agent binary not found. Skip uploading the results."
191 - return 0
192 - fi
193 - # /workspace/buildkite-agent annotate --style "success" --context "benchmark-results" --append < $RESULTS_FOLDER/${CURRENT_LLM_SERVING_ENGINE}_nightly_results.md
194 - /workspace/buildkite-agent artifact upload "$RESULTS_FOLDER/*"
195 -}
196 -
197 -
198 -main() {
199 -
200 - check_gpus
201 - # enter vllm directory
202 - cd $VLLM_SOURCE_CODE_LOC/benchmarks
203 -
204 - declare -g RESULTS_FOLDER=results/
205 - mkdir -p $RESULTS_FOLDER
206 - BENCHMARK_ROOT=../.buildkite/nightly-benchmarks/
207 -
208 - python -m pip install transformers==4.41.2
209 -
210 - export CURRENT_LLM_SERVING_ENGINE=lmdeploy
211 - run_serving_tests $BENCHMARK_ROOT/tests/nightly-tests.json
212 - python -m pip install tabulate pandas
213 - python $BENCHMARK_ROOT/scripts/summary-nightly-results.py
214 - upload_to_buildkite
215 -
216 -}
217 -
218 -main "$@"
.buildkite/nightly-benchmarks/scripts/run-nightly-benchmarks.sh new
+357
@@ -0,0 +1,357 @@
1 +#!/bin/bash
2 +
3 +set -o pipefail
4 +set -x
5 +
6 +check_gpus() {
7 + # check the number of GPUs and GPU type.
8 + declare -g gpu_count=$(nvidia-smi --list-gpus | wc -l)
9 + if [[ $gpu_count -gt 0 ]]; then
10 + echo "GPU found."
11 + else
12 + echo "Need at least 1 GPU to run benchmarking."
13 + exit 1
14 + fi
15 + declare -g gpu_type=$(echo $(nvidia-smi --query-gpu=name --format=csv,noheader) | awk '{print $2}')
16 + echo "GPU type is $gpu_type"
17 +}
18 +
19 +check_hf_token() {
20 + # check if HF_TOKEN is available and valid
21 + if [[ -z "$HF_TOKEN" ]]; then
22 + echo "Error: HF_TOKEN is not set."
23 + exit 1
24 + elif [[ ! "$HF_TOKEN" =~ ^hf_ ]]; then
25 + echo "Error: HF_TOKEN does not start with 'hf_'."
26 + exit 1
27 + else
28 + echo "HF_TOKEN is set and valid."
29 + fi
30 +}
31 +
32 +
33 +upload_to_buildkite() {
34 + # upload the benchmarking results to buildkite
35 +
36 + # if the agent binary is not found, skip uploading the results, exit 0
37 + if [ ! -f /workspace/buildkite-agent ]; then
38 + echo "buildkite-agent binary not found. Skip uploading the results."
39 + return 0
40 + fi
41 + # /workspace/buildkite-agent annotate --style "success" --context "benchmark-results" --append < $RESULTS_FOLDER/${CURRENT_LLM_SERVING_ENGINE}_nightly_results.md
42 + /workspace/buildkite-agent artifact upload "$RESULTS_FOLDER/*"
43 +}
44 +
45 +
46 +get_current_llm_serving_engine() {
47 +
48 + if which lmdeploy >/dev/null; then
49 + echo "Container: lmdeploy"
50 + export CURRENT_LLM_SERVING_ENGINE=lmdeploy
51 + return
52 + fi
53 +
54 + if [ -e /tgi-entrypoint.sh ]; then
55 + echo "Container: tgi"
56 + export CURRENT_LLM_SERVING_ENGINE=tgi
57 + return
58 + fi
59 +
60 + if which trtllm-build >/dev/null; then
61 + echo "Container: tensorrt-llm"
62 + export CURRENT_LLM_SERVING_ENGINE=trt
63 + return
64 + fi
65 +
66 + if [ -e /sgl-workspace ]; then
67 + echo "Container: sglang"
68 + export CURRENT_LLM_SERVING_ENGINE=sglang
69 + return
70 + fi
71 +
72 + if [ -e /vllm-workspace ]; then
73 + echo "Container: vllm"
74 + # move to a completely irrelevant directory, to avoid import vllm from current folder
75 + export CURRENT_LLM_SERVING_ENGINE=vllm
76 +
77 + return
78 + fi
79 +}
80 +
81 +json2args() {
82 + # transforms the JSON string to command line args, and '_' is replaced to '-'
83 + # example:
84 + # input: { "model": "meta-llama/Llama-2-7b-chat-hf", "tensor_parallel_size": 1 }
85 + # output: --model meta-llama/Llama-2-7b-chat-hf --tensor-parallel-size 1
86 + local json_string=$1
87 + local args=$(
88 + echo "$json_string" | jq -r '
89 + to_entries |
90 + map("--" + (.key | gsub("_"; "-")) + " " + (.value | tostring)) |
91 + join(" ")
92 + '
93 + )
94 + echo "$args"
95 +}
96 +
97 +kill_gpu_processes() {
98 + pkill -f python
99 + pkill -f python3
100 + pkill -f tritonserver
101 + pkill -f pt_main_thread
102 + pkill -f text-generation
103 + pkill -f lmdeploy
104 +
105 + while [ $(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits | head -n 1) -ge 1000 ]; do
106 + sleep 1
107 + done
108 +}
109 +
110 +wait_for_server() {
111 + # wait for vllm server to start
112 + # return 1 if vllm server crashes
113 + timeout 1200 bash -c '
114 + until curl -s localhost:8000/v1/completions > /dev/null; do
115 + sleep 1
116 + done' && return 0 || return 1
117 +}
118 +
119 +ensure_installed() {
120 + # Ensure that the given command is installed by apt-get
121 + local cmd=$1
122 + if ! which $cmd >/dev/null; then
123 + apt-get update && apt-get install -y $cmd
124 + fi
125 +}
126 +
127 +run_serving_tests() {
128 + # run serving tests using `benchmark_serving.py`
129 + # $1: a json file specifying serving test cases
130 +
131 + local serving_test_file
132 + serving_test_file=$1
133 +
134 + # Iterate over serving tests
135 + jq -c '.[]' "$serving_test_file" | while read -r params; do
136 + # get the test name, and append the GPU type back to it.
137 + test_name=$(echo "$params" | jq -r '.test_name')
138 +
139 + # if TEST_SELECTOR is set, only run the test cases that match the selector
140 + if [[ -n "$TEST_SELECTOR" ]] && [[ ! "$test_name" =~ $TEST_SELECTOR ]]; then
141 + echo "Skip test case $test_name."
142 + continue
143 + fi
144 +
145 + # prepend the current serving engine to the test name
146 + test_name=${CURRENT_LLM_SERVING_ENGINE}_${test_name}
147 +
148 + # get common parameters
149 + common_params=$(echo "$params" | jq -r '.common_parameters')
150 + model=$(echo "$common_params" | jq -r '.model')
151 + tp=$(echo "$common_params" | jq -r '.tp')
152 + dataset_name=$(echo "$common_params" | jq -r '.dataset_name')
153 + dataset_path=$(echo "$common_params" | jq -r '.dataset_path')
154 + port=$(echo "$common_params" | jq -r '.port')
155 + num_prompts=$(echo "$common_params" | jq -r '.num_prompts')
156 + reuse_server=$(echo "$common_params" | jq -r '.reuse_server')
157 +
158 + # get client and server arguments
159 + server_params=$(echo "$params" | jq -r ".${CURRENT_LLM_SERVING_ENGINE}_server_parameters")
160 + client_params=$(echo "$params" | jq -r ".${CURRENT_LLM_SERVING_ENGINE}_client_parameters")
161 + client_args=$(json2args "$client_params")
162 + qps_list=$(echo "$params" | jq -r '.qps_list')
163 + qps_list=$(echo "$qps_list" | jq -r '.[] | @sh')
164 + echo "Running over qps list $qps_list"
165 +
166 + # check if there is enough GPU to run the test
167 + if [[ $gpu_count -lt $tp ]]; then
168 + echo "Required num-shard $tp but only $gpu_count GPU found. Skip testcase $test_name."
169 + continue
170 + fi
171 +
172 + if [[ $reuse_server == "true" ]]; then
173 + echo "Reuse previous server for test case $test_name"
174 + else
175 + kill_gpu_processes
176 + bash $VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/scripts/launch-server.sh \
177 + "$server_params" "$common_params"
178 + fi
179 +
180 + wait_for_server
181 +
182 + if [ $? -eq 0 ]; then
183 + echo ""
184 + echo "$CURRENT_LLM_SERVING_ENGINE server is up and running."
185 + else
186 + echo ""
187 + echo "$CURRENT_LLM_SERVING_ENGINE failed to start within the timeout period."
188 + break
189 + fi
190 +
191 + # prepare tokenizer
192 + # this is required for lmdeploy.
193 + cd $VLLM_SOURCE_CODE_LOC/benchmarks
194 + rm -rf /tokenizer_cache
195 + mkdir /tokenizer_cache
196 + python3 ../.buildkite/nightly-benchmarks/scripts/download-tokenizer.py \
197 + --model "$model" \
198 + --cachedir /tokenizer_cache
199 + cd $VLLM_SOURCE_CODE_LOC/benchmarks
200 +
201 +
202 + # change model name for lmdeploy (it will not follow standard hf name)
203 + if [[ "$CURRENT_LLM_SERVING_ENGINE" == "lmdeploy" ]]; then
204 + model=$(python ../.buildkite/nightly-benchmarks/scripts/get-lmdeploy-modelname.py)
205 + fi
206 +
207 + # iterate over different QPS
208 + for qps in $qps_list; do
209 + # remove the surrounding single quote from qps
210 + if [[ "$qps" == *"inf"* ]]; then
211 + echo "qps was $qps"
212 + qps="inf"
213 + echo "now qps is $qps"
214 + fi
215 +
216 + new_test_name=$test_name"_qps_"$qps
217 +
218 + backend=$CURRENT_LLM_SERVING_ENGINE
219 +
220 + if [[ $backend = "trt" ]]; then
221 + backend="tensorrt-llm"
222 + fi
223 +
224 + if [[ "$backend" == *"vllm"* ]]; then
225 + backend="vllm"
226 + fi
227 +
228 + if [[ "$dataset_name" = "sharegpt" ]]; then
229 +
230 + client_command="python3 benchmark_serving.py \
231 + --backend $backend \
232 + --tokenizer /tokenizer_cache \
233 + --model $model \
234 + --dataset-name $dataset_name \
235 + --dataset-path $dataset_path \
236 + --num-prompts $num_prompts \
237 + --port $port \
238 + --save-result \
239 + --result-dir $RESULTS_FOLDER \
240 + --result-filename ${new_test_name}.json \
241 + --request-rate $qps \
242 + --ignore-eos \
243 + $client_args"
244 +
245 + elif [[ "$dataset_name" = "sonnet" ]]; then
246 +
247 + sonnet_input_len=$(echo "$common_params" | jq -r '.sonnet_input_len')
248 + sonnet_output_len=$(echo "$common_params" | jq -r '.sonnet_output_len')
249 + sonnet_prefix_len=$(echo "$common_params" | jq -r '.sonnet_prefix_len')
250 +
251 + client_command="python3 benchmark_serving.py \
252 + --backend $backend \
253 + --tokenizer /tokenizer_cache \
254 + --model $model \
255 + --dataset-name $dataset_name \
256 + --dataset-path $dataset_path \
257 + --num-prompts $num_prompts \
258 + --sonnet-input-len $sonnet_input_len \
259 + --sonnet-output-len $sonnet_output_len \
260 + --sonnet-prefix-len $sonnet_prefix_len \
261 + --port $port \
262 + --save-result \
263 + --result-dir $RESULTS_FOLDER \
264 + --result-filename ${new_test_name}.json \
265 + --request-rate $qps \
266 + --ignore-eos \
267 + $client_args"
268 +
269 + else
270 +
271 + echo "The dataset name must be either 'sharegpt' or 'sonnet'. Got $dataset_name."
272 + exit 1
273 +
274 + fi
275 +
276 +
277 +
278 + echo "Running test case $test_name with qps $qps"
279 + echo "Client command: $client_command"
280 +
281 + eval "$client_command"
282 +
283 + server_command="None"
284 +
285 + # record the benchmarking commands
286 + jq_output=$(jq -n \
287 + --arg server "$server_command" \
288 + --arg client "$client_command" \
289 + --arg gpu "$gpu_type" \
290 + --arg engine "$CURRENT_LLM_SERVING_ENGINE" \
291 + '{
292 + server_command: $server,
293 + client_command: $client,
294 + gpu_type: $gpu,
295 + engine: $engine
296 + }')
297 + echo "$jq_output" >"$RESULTS_FOLDER/${new_test_name}.commands"
298 +
299 + done
300 +
301 + done
302 +
303 + kill_gpu_processes
304 +}
305 +
306 +
307 +prepare_dataset() {
308 +
309 + # download sharegpt dataset
310 + cd $VLLM_SOURCE_CODE_LOC/benchmarks
311 + wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
312 +
313 + # duplicate sonnet by 4x, to allow benchmarking with input length 2048
314 + cd $VLLM_SOURCE_CODE_LOC/benchmarks
315 + echo "" > sonnet_4x.txt
316 + for _ in {1..4}
317 + do
318 + cat sonnet.txt >> sonnet_4x.txt
319 + done
320 +
321 +}
322 +
323 +main() {
324 +
325 + # check if the environment variable is successfully injected from yaml
326 +
327 + check_gpus
328 + check_hf_token
329 + get_current_llm_serving_engine
330 +
331 + pip install -U transformers
332 +
333 + # check storage
334 + df -h
335 +
336 + ensure_installed wget
337 + ensure_installed curl
338 + ensure_installed jq
339 +
340 + prepare_dataset
341 +
342 + cd $VLLM_SOURCE_CODE_LOC/benchmarks
343 + declare -g RESULTS_FOLDER=results/
344 + mkdir -p $RESULTS_FOLDER
345 + BENCHMARK_ROOT=$VLLM_SOURCE_CODE_LOC/.buildkite/nightly-benchmarks/
346 +
347 + # run the test
348 + run_serving_tests $BENCHMARK_ROOT/tests/nightly-tests.json
349 +
350 + # upload benchmark results to buildkite
351 + python3 -m pip install tabulate pandas
352 + python3 $BENCHMARK_ROOT/scripts/summary-nightly-results.py
353 + upload_to_buildkite
354 +
355 +}
356 +
357 +main "$@"
.buildkite/nightly-benchmarks/scripts/run-tgi-nightly.sh deleted
-216
@@ -1,216 +0,0 @@
1 -#!/bin/bash
2 -
3 -set -o pipefail
4 -
5 -check_gpus() {
6 - # check the number of GPUs and GPU type.
7 - declare -g gpu_count=$(nvidia-smi --list-gpus | wc -l)
8 - if [[ $gpu_count -gt 0 ]]; then
9 - echo "GPU found."
10 - else
11 - echo "Need at least 1 GPU to run benchmarking."
12 - exit 1
13 - fi
14 - declare -g gpu_type=$(echo $(nvidia-smi --query-gpu=name --format=csv,noheader) | awk '{print $2}')
15 - echo "GPU type is $gpu_type"
16 -}
17 -
18 -kill_gpu_processes() {
19 - pkill text-generation || true
20 - # waiting for GPU processes to be fully killed
21 - sleep 10
22 - # Print the GPU memory usage
23 - # so that we know if all GPU processes are killed.
24 - gpu_memory_usage=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits -i 0)
25 - # The memory usage should be 0 MB.
26 - echo "GPU 0 Memory Usage: $gpu_memory_usage MB"
27 -}
28 -
29 -json2args() {
30 - # transforms the JSON string to command line args, and '_' is replaced to '-'
31 - # example:
32 - # input: { "model": "meta-llama/Llama-2-7b-chat-hf", "tensor_parallel_size": 1 }
33 - # output: --model meta-llama/Llama-2-7b-chat-hf --tensor-parallel-size 1
34 - local json_string=$1
35 - local args=$(
36 - echo "$json_string" | jq -r '
37 - to_entries |
38 - map("--" + (.key | gsub("_"; "-")) + " " + (.value | tostring)) |
39 - join(" ")
40 - '
41 - )
42 - echo "$args"
43 -}
44 -
45 -wait_for_server() {
46 - timeout 1200 bash -c '
47 - until curl -s localhost:8000/generate_stream > /dev/null; do
48 - sleep 1
49 - done' && return 0 || return 1
50 -}
51 -
52 -run_serving_tests() {
53 - # run serving tests using `benchmark_serving.py`
54 - # $1: a json file specifying serving test cases
55 -
56 - local serving_test_file
57 - serving_test_file=$1
58 -
59 - # Iterate over serving tests
60 - jq -c '.[]' "$serving_test_file" | while read -r params; do
61 - # get the test name, and append the GPU type back to it.
62 - test_name=$(echo "$params" | jq -r '.test_name')
63 -
64 -
65 - # if TEST_SELECTOR is set, only run the test cases that match the selector
66 - if [[ -n "$TEST_SELECTOR" ]] && [[ ! "$test_name" =~ $TEST_SELECTOR ]]; then
67 - echo "Skip test case $test_name."
68 - continue
69 - fi
70 -
71 - # append tgi to the test name
72 - test_name=tgi_$test_name
73 -
74 - # get common parameters
75 - common_params=$(echo "$params" | jq -r '.common_parameters')
76 - model=$(echo "$common_params" | jq -r '.model')
77 - tp=$(echo "$common_params" | jq -r '.tp')
78 - dataset_name=$(echo "$common_params" | jq -r '.dataset_name')
79 - dataset_path=$(echo "$common_params" | jq -r '.dataset_path')
80 - port=$(echo "$common_params" | jq -r '.port')
81 - num_prompts=$(echo "$common_params" | jq -r '.num_prompts')
82 -
83 - # get client and server arguments
84 - server_params=$(echo "$params" | jq -r '.tgi_server_parameters')
85 - client_params=$(echo "$params" | jq -r '.tgi_client_parameters')
86 - server_args=$(json2args "$server_params")
87 - client_args=$(json2args "$client_params")
88 - qps_list=$(echo "$params" | jq -r '.qps_list')
89 - qps_list=$(echo "$qps_list" | jq -r '.[] | @sh')
90 - echo "Running over qps list $qps_list"
91 -
92 - # check if there is enough GPU to run the test
93 - if [[ $gpu_count -lt $tp ]]; then
94 - echo "Required num-shard $tp but only $gpu_count GPU found. Skip testcase $test_name."
95 - continue
96 - fi
97 -
98 - if echo "$common_params" | jq -e 'has("fp8")' > /dev/null; then
99 - echo "Key 'fp8' exists in common params."
100 - server_command="/tgi-entrypoint.sh \
101 - --model-id $model \
102 - --num-shard $tp \
103 - --port $port \
104 - --quantize fp8 \
105 - $server_args"
106 - else
107 - echo "Key 'fp8' does not exist in common params."
108 - server_command="/tgi-entrypoint.sh \
109 - --model-id $model \
110 - --num-shard $tp \
111 - --port $port \
112 - $server_args"
113 - fi
114 -
115 -
116 -
117 -
118 - # run the server
119 - echo "Running test case $test_name"
120 - echo "Server command: $server_command"
121 - eval "$server_command" &
122 -
123 - # wait until the server is alive
124 - wait_for_server
125 - if [ $? -eq 0 ]; then
126 - echo ""
127 - echo "tgi server is up and running."
128 - else
129 - echo ""
130 - echo "tgi failed to start within the timeout period."
131 - break
132 - fi
133 -
134 - # iterate over different QPS
135 - for qps in $qps_list; do
136 - # remove the surrounding single quote from qps
137 - if [[ "$qps" == *"inf"* ]]; then
138 - echo "qps was $qps"
139 - qps="inf"
140 - echo "now qps is $qps"
141 - fi
142 -
143 - new_test_name=$test_name"_qps_"$qps
144 -
145 - client_command="python3 benchmark_serving.py \
146 - --backend tgi \
147 - --model $model \
148 - --dataset-name $dataset_name \
149 - --dataset-path $dataset_path \
150 - --num-prompts $num_prompts \
151 - --port $port \
152 - --save-result \
153 - --result-dir $RESULTS_FOLDER \
154 - --result-filename ${new_test_name}.json \
155 - --request-rate $qps \
156 - $client_args"
157 -
158 - echo "Running test case $test_name with qps $qps"
159 - echo "Client command: $client_command"
160 -
161 - eval "$client_command"
162 -
163 - # record the benchmarking commands
164 - jq_output=$(jq -n \
165 - --arg server "$server_command" \
166 - --arg client "$client_command" \
167 - --arg gpu "$gpu_type" \
168 - --arg engine "tgi" \
169 - '{
170 - server_command: $server,
171 - client_command: $client,
172 - gpu_type: $gpu,
173 - engine: $engine
174 - }')
175 - echo "$jq_output" >"$RESULTS_FOLDER/${new_test_name}.commands"
176 -
177 - done
178 -
179 - # clean up
180 - kill_gpu_processes
181 - rm -rf /root/.cache/huggingface/*
182 - done
183 -}
184 -
185 -
186 -
187 -upload_to_buildkite() {
188 - # upload the benchmarking results to buildkite
189 -
190 - # if the agent binary is not found, skip uploading the results, exit 0
191 - if [ ! -f /workspace/buildkite-agent ]; then
192 - echo "buildkite-agent binary not found. Skip uploading the results."
193 - return 0
194 - fi
195 - # /workspace/buildkite-agent annotate --style "success" --context "benchmark-results" --append < $RESULTS_FOLDER/${CURRENT_LLM_SERVING_ENGINE}_nightly_results.md
196 - /workspace/buildkite-agent artifact upload "$RESULTS_FOLDER/*"
197 -}
198 -
199 -main() {
200 -
201 - check_gpus
202 - # enter vllm directory
203 - cd $VLLM_SOURCE_CODE_LOC/benchmarks
204 - declare -g RESULTS_FOLDER=results/
205 - mkdir -p $RESULTS_FOLDER
206 - BENCHMARK_ROOT=../.buildkite/nightly-benchmarks/
207 -
208 - export CURRENT_LLM_SERVING_ENGINE=tgi
209 - run_serving_tests $BENCHMARK_ROOT/tests/nightly-tests.json
210 - python -m pip install tabulate pandas
211 - python $BENCHMARK_ROOT/scripts/summary-nightly-results.py
212 - upload_to_buildkite
213 -
214 -}
215 -
216 -main "$@"
.buildkite/nightly-benchmarks/scripts/run-trt-nightly.sh deleted
-214
@@ -1,214 +0,0 @@
1 -#!/bin/bash
2 -
3 -set -o pipefail
4 -
5 -check_gpus() {
6 - # check the number of GPUs and GPU type.
7 - declare -g gpu_count=$(nvidia-smi --list-gpus | wc -l)
8 - if [[ $gpu_count -gt 0 ]]; then
9 - echo "GPU found."
10 - else
11 - echo "Need at least 1 GPU to run benchmarking."
12 - exit 1
13 - fi
14 - declare -g gpu_type=$(echo $(nvidia-smi --query-gpu=name --format=csv,noheader) | awk '{print $2}')
15 - echo "GPU type is $gpu_type"
16 -}
17 -
18 -kill_gpu_processes() {
19 - pkill tritonserver || true
20 - # waiting for GPU processes to be fully killed
21 - sleep 20
22 - # Print the GPU memory usage
23 - # so that we know if all GPU processes are killed.
24 - gpu_memory_usage=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits -i 0)
25 - # The memory usage should be 0 MB.
26 - echo "GPU 0 Memory Usage: $gpu_memory_usage MB"
27 -}
28 -
29 -json2args() {
30 - # transforms the JSON string to command line args, and '_' is replaced to '-'
31 - # example:
32 - # input: { "model": "meta-llama/Llama-2-7b-chat-hf", "tensor_parallel_size": 1 }
33 - # output: --model meta-llama/Llama-2-7b-chat-hf --tensor-parallel-size 1
34 - local json_string=$1
35 - local args=$(
36 - echo "$json_string" | jq -r '
37 - to_entries |
38 - map("--" + (.key | gsub("_"; "-")) + " " + (.value | tostring)) |
39 - join(" ")
40 - '
41 - )
42 - echo "$args"
43 -}
44 -
45 -wait_for_server() {
46 - timeout 1200 bash -c '
47 - until curl -s localhost:8000/generate_stream > /dev/null; do
48 - sleep 1
49 - done' && return 0 || return 1
50 -}
51 -
52 -run_serving_tests() {
53 - # run serving tests using `benchmark_serving.py`
54 - # $1: a json file specifying serving test cases
55 -
56 - local serving_test_file
57 - serving_test_file=$1
58 -
59 - # Iterate over serving tests
60 - jq -c '.[]' "$serving_test_file" | while read -r params; do
61 - # get the test name, and append the GPU type back to it.
62 - test_name=$(echo "$params" | jq -r '.test_name')
63 -
64 - # if TEST_SELECTOR is set, only run the test cases that match the selector
65 - if [[ -n "$TEST_SELECTOR" ]] && [[ ! "$test_name" =~ $TEST_SELECTOR ]]; then
66 - echo "Skip test case $test_name."
67 - continue
68 - fi
69 -
70 - # append trt to the test name
71 - test_name=trt_$test_name
72 -
73 - # get common parameters
74 - common_params=$(echo "$params" | jq -r '.common_parameters')
75 - model=$(echo "$common_params" | jq -r '.model')
76 - tp=$(echo "$common_params" | jq -r '.tp')
77 - dataset_name=$(echo "$common_params" | jq -r '.dataset_name')
78 - dataset_path=$(echo "$common_params" | jq -r '.dataset_path')
79 - port=$(echo "$common_params" | jq -r '.port')
80 - num_prompts=$(echo "$common_params" | jq -r '.num_prompts')
81 -
82 - # get client and server arguments
83 - server_params=$(echo "$params" | jq -r '.trt_server_parameters')
84 - client_params=$(echo "$params" | jq -r '.trt_client_parameters')
85 - client_args=$(json2args "$client_params")
86 - qps_list=$(echo "$params" | jq -r '.qps_list')
87 - qps_list=$(echo "$qps_list" | jq -r '.[] | @sh')
88 - echo "Running over qps list $qps_list"
89 -
90 - # check if there is enough GPU to run the test
91 - if [[ $gpu_count -lt $tp ]]; then
92 - echo "Required model_tp_size $tp but only $gpu_count GPU found. Skip testcase $test_name."
93 - continue
94 - fi
95 -
96 -
97 -
98 - cd $VLLM_SOURCE_CODE_LOC/benchmarks
99 -
100 -
101 - echo "Running test case $test_name"
102 - bash ../.buildkite/nightly-benchmarks/scripts/launch-trt-server.sh "$server_params" "$common_params"
103 -
104 - # wait until the server is alive
105 - wait_for_server
106 - if [ $? -eq 0 ]; then
107 - echo ""
108 - echo "trt server is up and running."
109 - else
110 - echo ""
111 - echo "trt failed to start within the timeout period."
112 - break
113 - fi
114 -
115 - # prepare tokenizer
116 - cd $VLLM_SOURCE_CODE_LOC/benchmarks
117 - rm -rf /tokenizer_cache
118 - mkdir /tokenizer_cache
119 - python ../.buildkite/nightly-benchmarks/scripts/download-tokenizer.py \
120 - --model "$model" \
121 - --cachedir /tokenizer_cache
122 - cd $VLLM_SOURCE_CODE_LOC/benchmarks
123 -
124 -
125 - # iterate over different QPS
126 - for qps in $qps_list; do
127 - # remove the surrounding single quote from qps
128 - if [[ "$qps" == *"inf"* ]]; then
129 - echo "qps was $qps"
130 - qps="inf"
131 - echo "now qps is $qps"
132 - fi
133 -
134 - new_test_name=$test_name"_qps_"$qps
135 -
136 - client_command="python3 benchmark_serving.py \
137 - --backend tensorrt-llm \
138 - --tokenizer /tokenizer_cache \
139 - --model $model \
140 - --dataset-name $dataset_name \
141 - --dataset-path $dataset_path \
142 - --num-prompts $num_prompts \
143 - --port $port \
144 - --save-result \
145 - --result-dir $RESULTS_FOLDER \
146 - --result-filename ${new_test_name}.json \
147 - --request-rate $qps \
148 - $client_args"
149 -
150 - echo "Running test case $test_name with qps $qps"
151 - echo "Client command: $client_command"
152 -
153 - eval "$client_command"
154 -
155 - server_command=""
156 - # record the benchmarking commands
157 - jq_output=$(jq -n \
158 - --arg server "$server_command" \
159 - --arg client "$client_command" \
160 - --arg gpu "$gpu_type" \
161 - --arg engine "trt" \
162 - '{
163 - server_command: $server,
164 - client_command: $client,
165 - gpu_type: $gpu,
166 - engine: $engine
167 - }')
168 - echo "$jq_output" >"$RESULTS_FOLDER/${new_test_name}.commands"
169 -
170 - done
171 -
172 - # clean up
173 - kill_gpu_processes
174 - rm -rf /root/.cache/huggingface/*
175 - done
176 -}
177 -
178 -upload_to_buildkite() {
179 - # upload the benchmarking results to buildkite
180 -
181 - # if the agent binary is not found, skip uploading the results, exit 0
182 - if [ ! -f /workspace/buildkite-agent ]; then
183 - echo "buildkite-agent binary not found. Skip uploading the results."
184 - return 0
185 - fi
186 - # /workspace/buildkite-agent annotate --style "success" --context "benchmark-results" --append < $RESULTS_FOLDER/${CURRENT_LLM_SERVING_ENGINE}_nightly_results.md
187 - /workspace/buildkite-agent artifact upload "$RESULTS_FOLDER/*"
188 -}
189 -
190 -
191 -main() {
192 -
193 - check_gpus
194 -
195 -
196 - # enter vllm directory
197 - cd $VLLM_SOURCE_CODE_LOC/benchmarks
198 -
199 - declare -g RESULTS_FOLDER=results/
200 - mkdir -p $RESULTS_FOLDER
201 - BENCHMARK_ROOT=../.buildkite/nightly-benchmarks/
202 -
203 - # update transformers package, to make sure mixtral tokenizer is available
204 - python -m pip install transformers -U
205 -
206 - export CURRENT_LLM_SERVING_ENGINE=trt
207 - run_serving_tests $BENCHMARK_ROOT/tests/nightly-tests.json
208 - python -m pip install tabulate pandas
209 - python $BENCHMARK_ROOT/scripts/summary-nightly-results.py
210 - upload_to_buildkite
211 -
212 -}
213 -
214 -main "$@"
.buildkite/nightly-benchmarks/scripts/run-vllm-nightly.sh deleted
-221
@@ -1,221 +0,0 @@
1 -#!/bin/bash
2 -
3 -set -o pipefail
4 -
5 -check_gpus() {
6 - # check the number of GPUs and GPU type.
7 - declare -g gpu_count=$(nvidia-smi --list-gpus | wc -l)
8 - if [[ $gpu_count -gt 0 ]]; then
9 - echo "GPU found."
10 - else
11 - echo "Need at least 1 GPU to run benchmarking."
12 - exit 1
13 - fi
14 - declare -g gpu_type=$(echo $(nvidia-smi --query-gpu=name --format=csv,noheader) | awk '{print $2}')
15 - echo "GPU type is $gpu_type"
16 -}
17 -
18 -kill_gpu_processes() {
19 - # kill all processes on GPU.
20 - pkill pt_main_thread
21 - sleep 10
22 -
23 - # remove vllm config file
24 - rm -rf ~/.config/vllm
25 -
26 - # Print the GPU memory usage
27 - # so that we know if all GPU processes are killed.
28 - gpu_memory_usage=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits -i 0)
29 - # The memory usage should be 0 MB.
30 - echo "GPU 0 Memory Usage: $gpu_memory_usage MB"
31 -}
32 -
33 -json2args() {
34 - # transforms the JSON string to command line args, and '_' is replaced to '-'
35 - # example:
36 - # input: { "model": "meta-llama/Llama-2-7b-chat-hf", "tensor_parallel_size": 1 }
37 - # output: --model meta-llama/Llama-2-7b-chat-hf --tensor-parallel-size 1
38 - local json_string=$1
39 - local args=$(
40 - echo "$json_string" | jq -r '
41 - to_entries |
42 - map("--" + (.key | gsub("_"; "-")) + " " + (.value | tostring)) |
43 - join(" ")
44 - '
45 - )
46 - echo "$args"
47 -}
48 -
49 -wait_for_server() {
50 - # wait for vllm server to start
51 - # return 1 if vllm server crashes
52 - timeout 1200 bash -c '
53 - until curl -s localhost:8000/v1/completions > /dev/null; do
54 - sleep 1
55 - done' && return 0 || return 1
56 -}
57 -
58 -run_serving_tests() {
59 - # run serving tests using `benchmark_serving.py`
60 - # $1: a json file specifying serving test cases
61 -
62 - local serving_test_file
63 - serving_test_file=$1
64 -
65 - # Iterate over serving tests
66 - jq -c '.[]' "$serving_test_file" | while read -r params; do
67 - # get the test name, and append the GPU type back to it.
68 - test_name=$(echo "$params" | jq -r '.test_name')
69 -
70 - # if TEST_SELECTOR is set, only run the test cases that match the selector
71 - if [[ -n "$TEST_SELECTOR" ]] && [[ ! "$test_name" =~ $TEST_SELECTOR ]]; then
72 - echo "Skip test case $test_name."
73 - continue
74 - fi
75 -
76 - # append vllm to the test name
77 - test_name=vllm_$test_name
78 -
79 -
80 - # get common parameters
81 - common_params=$(echo "$params" | jq -r '.common_parameters')
82 - model=$(echo "$common_params" | jq -r '.model')
83 - tp=$(echo "$common_params" | jq -r '.tp')
84 - dataset_name=$(echo "$common_params" | jq -r '.dataset_name')
85 - dataset_path=$(echo "$common_params" | jq -r '.dataset_path')
86 - port=$(echo "$common_params" | jq -r '.port')
87 - num_prompts=$(echo "$common_params" | jq -r '.num_prompts')
88 -
89 - # get client and server arguments
90 - server_params=$(echo "$params" | jq -r '.vllm_server_parameters')
91 - client_params=$(echo "$params" | jq -r '.vllm_client_parameters')
92 - server_args=$(json2args "$server_params")
93 - client_args=$(json2args "$client_params")
94 - qps_list=$(echo "$params" | jq -r '.qps_list')
95 - qps_list=$(echo "$qps_list" | jq -r '.[] | @sh')
96 - echo "Running over qps list $qps_list"
97 -
98 - # check if there is enough GPU to run the test
99 - if [[ $gpu_count -lt $tp ]]; then
100 - echo "Required tensor-parallel-size $tp but only $gpu_count GPU found. Skip testcase $test_name."
101 - continue
102 - fi
103 -
104 - if echo "$common_params" | jq -e 'has("fp8")' > /dev/null; then
105 - echo "Key 'fp8' exists in common params. Use neuralmagic fp8 model for convenience."
106 - model=$(echo "$common_params" | jq -r '.neuralmagic_quantized_model')
107 - server_command="python3 \
108 - -m vllm.entrypoints.openai.api_server \
109 - -tp $tp \
110 - --model $model \
111 - --port $port \
112 - $server_args"
113 - else
114 - echo "Key 'fp8' does not exist in common params."
115 - server_command="python3 \
116 - -m vllm.entrypoints.openai.api_server \
117 - -tp $tp \
118 - --model $model \
119 - --port $port \
120 - $server_args"
121 - fi
122 -
123 - # run the server
124 - echo "Running test case $test_name"
125 - echo "Server command: $server_command"
126 - eval "$server_command" &
127 -
128 - # wait until the server is alive
129 - wait_for_server
130 - if [ $? -eq 0 ]; then
131 - echo ""
132 - echo "vllm server is up and running."
133 - else
134 - echo ""
135 - echo "vllm failed to start within the timeout period."
136 - break
137 - fi
138 -
139 - # iterate over different QPS
140 - for qps in $qps_list; do
141 - # remove the surrounding single quote from qps
142 - if [[ "$qps" == *"inf"* ]]; then
143 - echo "qps was $qps"
144 - qps="inf"
145 - echo "now qps is $qps"
146 - fi
147 -
148 - new_test_name=$test_name"_qps_"$qps
149 -
150 - client_command="python3 benchmark_serving.py \
151 - --backend vllm \
152 - --model $model \
153 - --dataset-name $dataset_name \
154 - --dataset-path $dataset_path \
155 - --num-prompts $num_prompts \
156 - --port $port \
157 - --save-result \
158 - --result-dir $RESULTS_FOLDER \
159 - --result-filename ${new_test_name}.json \
160 - --request-rate $qps \
161 - $client_args"
162 -
163 - echo "Running test case $test_name with qps $qps"
164 - echo "Client command: $client_command"
165 -
166 - eval "$client_command"
167 -
168 - # record the benchmarking commands
169 - jq_output=$(jq -n \
170 - --arg server "$server_command" \
171 - --arg client "$client_command" \
172 - --arg gpu "$gpu_type" \
173 - --arg engine "vllm" \
174 - '{
175 - server_command: $server,
176 - client_command: $client,
177 - gpu_type: $gpu,
178 - engine: $engine
179 - }')
180 - echo "$jq_output" >"$RESULTS_FOLDER/${new_test_name}.commands"
181 -
182 - done
183 -
184 - # clean up
185 - kill_gpu_processes
186 - rm -rf /root/.cache/huggingface/*
187 - done
188 -}
189 -
190 -
191 -upload_to_buildkite() {
192 - # upload the benchmarking results to buildkite
193 -
194 - # if the agent binary is not found, skip uploading the results, exit 0
195 - if [ ! -f /workspace/buildkite-agent ]; then
196 - echo "buildkite-agent binary not found. Skip uploading the results."
197 - return 0
198 - fi
199 - # /workspace/buildkite-agent annotate --style "success" --context "benchmark-results" --append < $RESULTS_FOLDER/${CURRENT_LLM_SERVING_ENGINE}_nightly_results.md
200 - /workspace/buildkite-agent artifact upload "$RESULTS_FOLDER/*"
201 -}
202 -
203 -main() {
204 -
205 - check_gpus
206 - # enter vllm directory
207 - cd $VLLM_SOURCE_CODE_LOC/benchmarks
208 - declare -g RESULTS_FOLDER=results/
209 - mkdir -p $RESULTS_FOLDER
210 - BENCHMARK_ROOT=../.buildkite/nightly-benchmarks/
211 -
212 - export CURRENT_LLM_SERVING_ENGINE=vllm
213 - run_serving_tests $BENCHMARK_ROOT/tests/nightly-tests.json
214 -
215 - python3 -m pip install tabulate pandas
216 - python3 $BENCHMARK_ROOT/scripts/summary-nightly-results.py
217 - upload_to_buildkite
218 -
219 -}
220 -
221 -main "$@"
.buildkite/nightly-benchmarks/scripts/summary-nightly-results.py
+8 -1
@@ -17,10 +17,17 @@ serving_column_mapping = {
17 "request_throughput": "Tput (req/s)",
18 "mean_ttft_ms": "Mean TTFT (ms)",
19 "std_ttft_ms": "Std TTFT (ms)",
20 + "median_ttft_ms": "Median TTFT (ms)",
21 "mean_itl_ms": "Mean ITL (ms)",
22 "std_itl_ms": "Std ITL (ms)",
22 - "input_throughput": "Input Tput (tok/s)",
23 + "median_itl_ms": "Median ITL (ms)",
24 + "mean_tpot_ms": "Mean TPOT (ms)",
25 + "std_tpot_ms": "Std TPOT (ms)",
26 + "median_tpot_ms": "Median TPOT (ms)",
27 + "total_token_throughput": "Total Token Tput (tok/s)",
28 "output_throughput": "Output Tput (tok/s)",
29 + "total_input_tokens": "Total input tokens",
30 + "total_output_tokens": "Total output tokens",
31 "engine": "Engine",
32 }
33
.buildkite/nightly-benchmarks/tests/nightly-tests.json
+237 -30
@@ -1,16 +1,18 @@
1 [
2 {
3 - "test_name": "llama8B_tp1",
4 - "qps_list": [4],
3 + "test_name": "llama8B_tp1_sharegpt",
4 + "qps_list": [4,8,16,32,"inf"],
5 "common_parameters": {
6 - "model": "meta-llama/Meta-Llama-3-8B",
6 + "model": "meta-llama/Meta-Llama-3-8B-Instruct",
7 "tp": 1,
8 "dataset_name": "sharegpt",
9 "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
10 "num_prompts": 500,
11 - "port": 8000
11 + "port": 8000,
12 + "reuse_server": false
13 },
14 "lmdeploy_server_parameters": {
15 + "dtype": "bfloat16"
16 },
17 "lmdeploy_client_parameters": {
18 },
@@ -21,34 +23,158 @@
23 },
24 "trt_server_parameters": {
25 "model_type": "llama",
24 - "model_dtype": "float16",
25 - "max_batch_size": 256,
26 + "model_dtype": "bfloat16",
27 + "max_batch_size": 2048,
28 "max_input_len": 4096,
27 - "max_output_len": 4096,
28 - "trt_llm_version": "r24.04"
29 + "max_seq_len": 6144,
30 + "max_num_tokens": 16384,
31 + "trt_llm_version": "v0.11.0"
32 },
33 "trt_client_parameters": {
34 "endpoint": "/v2/models/ensemble/generate_stream"
35 + },
36 + "vllm_server_parameters": {
37 + "disable_log_stats": "",
38 + "disable_log_requests": "",
39 + "gpu_memory_utilization": 0.9,
40 + "num_scheduler_steps": 10,
41 + "max_num_seqs": 512,
42 + "dtype": "bfloat16"
43 + },
44 + "vllm_client_parameters": {
45 + },
46 + "sglang_server_parameters": {
47 + "disable_radix_cache": "",
48 + "enable_torch_compile": "",
49 + "dtype": "bfloat16"
50 + },
51 + "sglang_client_parameters": {
52 + }
53 + },
54 + {
55 + "test_name": "llama8B_tp1_sonnet_512_16",
56 + "qps_list": [4,8,16,32,"inf"],
57 + "common_parameters": {
58 + "model": "meta-llama/Meta-Llama-3-8B-Instruct",
59 + "tp": 1,
60 + "dataset_name": "sonnet",
61 + "dataset_path": "./sonnet_4x.txt",
62 + "num_prompts": 500,
63 + "port": 8000,
64 + "sonnet_input_len": 512,
65 + "sonnet_output_len": 16,
66 + "sonnet_prefix_len": 50,
67 + "reuse_server": true
68 + },
69 + "lmdeploy_server_parameters": {
70 + "dtype": "bfloat16"
71 + },
72 + "lmdeploy_client_parameters": {
73 + },
74 + "tgi_server_parameters": {
75 + },
76 + "tgi_client_parameters": {
77 + "endpoint": "/generate_stream"
78 + },
79 + "trt_server_parameters": {
80 + "model_type": "llama",
81 + "model_dtype": "bfloat16",
82 + "max_batch_size": 2048,
83 + "max_input_len": 4096,
84 + "max_seq_len": 6144,
85 + "max_num_tokens": 16384,
86 + "trt_llm_version": "v0.11.0"
87 + },
88 + "trt_client_parameters": {
89 + "endpoint": "/v2/models/ensemble/generate_stream"
90 + },
91 + "vllm_server_parameters": {
92 + "disable_log_stats": "",
93 + "disable_log_requests": "",
94 + "gpu_memory_utilization": 0.9,
95 + "num_scheduler_steps": 10,
96 + "max_num_seqs": 512,
97 + "dtype": "bfloat16"
98 + },
99 + "vllm_client_parameters": {
100 + },
101 + "sglang_server_parameters": {
102 + "disable_radix_cache": "",
103 + "enable_torch_compile": "",
104 + "dtype": "bfloat16"
105 + },
106 + "sglang_client_parameters": {
107 + }
108 + },
109 + {
110 + "test_name": "llama8B_tp1_sonnet_512_256",
111 + "qps_list": [4,8,16,32,"inf"],
112 + "common_parameters": {
113 + "model": "meta-llama/Meta-Llama-3-8B-Instruct",
114 + "tp": 1,
115 + "dataset_name": "sonnet",
116 + "dataset_path": "./sonnet_4x.txt",
117 + "num_prompts": 500,
118 + "port": 8000,
119 + "sonnet_input_len": 512,
120 + "sonnet_output_len": 256,
121 + "sonnet_prefix_len": 50,
122 + "reuse_server": true
123 + },
124 + "lmdeploy_server_parameters": {
125 + "dtype": "bfloat16"
126 + },
127 + "lmdeploy_client_parameters": {
128 + },
129 + "tgi_server_parameters": {
130 + },
131 + "tgi_client_parameters": {
132 + "endpoint": "/generate_stream"
133 + },
134 + "trt_server_parameters": {
135 + "model_type": "llama",
136 + "model_dtype": "bfloat16",
137 + "max_batch_size": 2048,
138 + "max_input_len": 4096,
139 + "max_seq_len": 6144,
140 + "max_num_tokens": 16384,
141 + "trt_llm_version": "v0.11.0"
142 },
143 + "trt_client_parameters": {
144 + "endpoint": "/v2/models/ensemble/generate_stream"
145 + },
146 "vllm_server_parameters": {
147 "disable_log_stats": "",
35 - "disable_log_requests": ""
148 + "disable_log_requests": "",
149 + "gpu_memory_utilization": 0.9,
150 + "num_scheduler_steps": 10,
151 + "max_num_seqs": 512,
152 + "dtype": "bfloat16"
153 },
154 "vllm_client_parameters": {
155 + },
156 + "sglang_server_parameters": {
157 + "disable_radix_cache": "",
158 + "enable_torch_compile": "",
159 + "dtype": "bfloat16"
160 + },
161 + "sglang_client_parameters": {
162 }
163 },
164 {
41 - "test_name": "llama70B_tp4",
42 - "qps_list": [2],
165 + "test_name": "llama70B_tp4_sharegpt",
166 + "qps_list": [4,8,16,32,"inf"],
167 "common_parameters": {
168 "model": "meta-llama/Meta-Llama-3-70B-Instruct",
169 "tp": 4,
170 "dataset_name": "sharegpt",
171 "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
172 "num_prompts": 500,
49 - "port": 8000
173 + "port": 8000,
174 + "reuse_server": false
175 },
176 "lmdeploy_server_parameters": {
177 + "dtype": "bfloat16"
178 },
179 "lmdeploy_client_parameters": {
180 },
@@ -59,34 +185,50 @@
185 },
186 "trt_server_parameters": {
187 "model_type": "llama",
62 - "model_dtype": "float16",
63 - "max_batch_size": 256,
188 + "model_dtype": "bfloat16",
189 + "max_batch_size": 2048,
190 "max_input_len": 4096,
65 - "max_output_len": 4096,
66 - "trt_llm_version": "r24.04"
191 + "max_seq_len": 6144,
192 + "max_num_tokens": 16384,
193 + "trt_llm_version": "v0.11.0"
194 },
195 "trt_client_parameters": {
196 "endpoint": "/v2/models/ensemble/generate_stream"
70 - },
197 + },
198 "vllm_server_parameters": {
199 "disable_log_stats": "",
73 - "disable_log_requests": ""
200 + "disable_log_requests": "",
201 + "gpu_memory_utilization": 0.9,
202 + "num_scheduler_steps": 10,
203 + "max_num_seqs": 512,
204 + "dtype": "bfloat16"
205 },
206 "vllm_client_parameters": {
207 + },
208 + "sglang_server_parameters": {
209 + "disable_radix_cache": "",
210 + "dtype": "bfloat16"
211 + },
212 + "sglang_client_parameters": {
213 }
214 },
215 {
79 - "test_name": "mixtral8x7B_tp2",
80 - "qps_list": [2],
216 + "test_name": "llama70B_tp4_sonnet_512_16",
217 + "qps_list": [4,8,16,32,"inf"],
218 "common_parameters": {
82 - "model": "mistralai/Mixtral-8x7B-Instruct-v0.1",
83 - "tp": 2,
84 - "dataset_name": "sharegpt",
85 - "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
219 + "model": "meta-llama/Meta-Llama-3-70B-Instruct",
220 + "tp": 4,
221 + "dataset_name": "sonnet",
222 + "dataset_path": "./sonnet_4x.txt",
223 "num_prompts": 500,
87 - "port": 8000
224 + "port": 8000,
225 + "sonnet_input_len": 512,
226 + "sonnet_output_len": 16,
227 + "sonnet_prefix_len": 50,
228 + "reuse_server": true
229 },
230 "lmdeploy_server_parameters": {
231 + "dtype": "bfloat16"
232 },
233 "lmdeploy_client_parameters": {
234 },
@@ -97,20 +239,85 @@
239 },
240 "trt_server_parameters": {
241 "model_type": "llama",
100 - "model_dtype": "float16",
101 - "max_batch_size": 256,
242 + "model_dtype": "bfloat16",
243 + "max_batch_size": 2048,
244 "max_input_len": 4096,
103 - "max_output_len": 4096,
104 - "trt_llm_version": "r24.04"
245 + "max_seq_len": 6144,
246 + "max_num_tokens": 16384,
247 + "trt_llm_version": "v0.11.0"
248 },
249 "trt_client_parameters": {
250 "endpoint": "/v2/models/ensemble/generate_stream"
251 + },
252 + "vllm_server_parameters": {
253 + "disable_log_stats": "",
254 + "disable_log_requests": "",
255 + "gpu_memory_utilization": 0.9,
256 + "num_scheduler_steps": 10,
257 + "max_num_seqs": 512,
258 + "dtype": "bfloat16"
259 + },
260 + "vllm_client_parameters": {
261 },
262 + "sglang_server_parameters": {
263 + "disable_radix_cache": "",
264 + "dtype": "bfloat16"
265 + },
266 + "sglang_client_parameters": {
267 + }
268 + },
269 + {
270 + "test_name": "llama70B_tp4_sonnet_512_256",
271 + "qps_list": [4,8,16,32,"inf"],
272 + "common_parameters": {
273 + "model": "meta-llama/Meta-Llama-3-70B-Instruct",
274 + "tp": 4,
275 + "dataset_name": "sonnet",
276 + "dataset_path": "./sonnet_4x.txt",
277 + "num_prompts": 500,
278 + "port": 8000,
279 + "sonnet_input_len": 512,
280 + "sonnet_output_len": 256,
281 + "sonnet_prefix_len": 50,
282 + "reuse_server": true
283 + },
284 + "lmdeploy_server_parameters": {
285 + "dtype": "bfloat16"
286 + },
287 + "lmdeploy_client_parameters": {
288 + },
289 + "tgi_server_parameters": {
290 + },
291 + "tgi_client_parameters": {
292 + "endpoint": "/generate_stream"
293 + },
294 + "trt_server_parameters": {
295 + "model_type": "llama",
296 + "model_dtype": "bfloat16",
297 + "max_batch_size": 2048,
298 + "max_input_len": 4096,
299 + "max_seq_len": 6144,
300 + "max_num_tokens": 16384,
301 + "trt_llm_version": "v0.11.0"
302 + },
303 + "trt_client_parameters": {
304 + "endpoint": "/v2/models/ensemble/generate_stream"
305 + },
306 "vllm_server_parameters": {
307 "disable_log_stats": "",
111 - "disable_log_requests": ""
308 + "disable_log_requests": "",
309 + "gpu_memory_utilization": 0.9,
310 + "num_scheduler_steps": 10,
311 + "max_num_seqs": 512,
312 + "dtype": "bfloat16"
313 },
314 "vllm_client_parameters": {
315 + },
316 + "sglang_server_parameters": {
317 + "disable_radix_cache": "",
318 + "dtype": "bfloat16"
319 + },
320 + "sglang_client_parameters": {
321 }
322 }
323 ]
\ No newline at end of file
benchmarks/backend_request_func.py
+7
@@ -26,6 +26,7 @@ class RequestFuncInput:
26 use_beam_search: bool = False
27 logprobs: Optional[int] = None
28 multi_modal_content: Optional[dict] = None
29 + ignore_eos: bool = False
30
31
32 @dataclass
@@ -55,6 +56,7 @@ async def async_request_tgi(
56 "do_sample": True,
57 "temperature": 0.01, # TGI does not accept 0.0 temperature.
58 "top_p": 0.99, # TGI does not accept 1.0 top_p.
59 + # TGI does not accept ignore_eos flag.
60 }
61 payload = {
62 "inputs": request_func_input.prompt,
@@ -129,6 +131,8 @@ async def async_request_trt_llm(
131 "max_tokens": request_func_input.output_len,
132 "stream": True,
133 }
134 + if request_func_input.ignore_eos:
135 + payload["min_length"] = request_func_input.output_len
136 output = RequestFuncOutput()
137 output.prompt_len = request_func_input.prompt_len
138
@@ -240,6 +244,7 @@ async def async_request_openai_completions(
244 "max_tokens": request_func_input.output_len,
245 "logprobs": request_func_input.logprobs,
246 "stream": True,
247 + "ignore_eos": request_func_input.ignore_eos,
248 }
249 headers = {
250 "Authorization": f"Bearer {os.environ.get('OPENAI_API_KEY')}"
@@ -327,6 +332,7 @@ async def async_request_openai_chat_completions(
332 "temperature": 0.0,
333 "max_tokens": request_func_input.output_len,
334 "stream": True,
335 + "ignore_eos": request_func_input.ignore_eos,
336 }
337 headers = {
338 "Content-Type": "application/json",
@@ -430,4 +436,5 @@ ASYNC_REQUEST_FUNCS = {
436 "openai-chat": async_request_openai_chat_completions,
437 "tensorrt-llm": async_request_trt_llm,
438 "scalellm": async_request_openai_completions,
439 + "sglang": async_request_openai_completions,
440 }
benchmarks/benchmark_serving.py
+8
@@ -397,6 +397,7 @@ async def benchmark(
397 profile: bool,
398 selected_percentile_metrics: List[str],
399 selected_percentiles: List[str],
400 + ignore_eos: bool,
401 ):
402 if backend in ASYNC_REQUEST_FUNCS:
403 request_func = ASYNC_REQUEST_FUNCS[backend]
@@ -420,6 +421,7 @@ async def benchmark(
421 best_of=best_of,
422 use_beam_search=use_beam_search,
423 multi_modal_content=test_mm_content,
424 + ignore_eos=ignore_eos,
425 )
426 test_output = await request_func(request_func_input=test_input)
427 if not test_output.success:
@@ -685,6 +687,7 @@ def main(args: argparse.Namespace):
687 selected_percentiles=[
688 float(p) for p in args.metric_percentiles.split(",")
689 ],
690 + ignore_eos=args.ignore_eos,
691 ))
692
693 # Save config and results to json
@@ -863,6 +866,11 @@ if __name__ == "__main__":
866 "{backend}-{args.request_rate}qps-{base_model_id}-{current_dt}.json"
867 " format.",
868 )
869 + parser.add_argument(
870 + "--ignore-eos",
871 + action="store_true",
872 + help="Set ignore_eos flag when sending the benchmark request."
873 + "Warning: ignore_eos is not supported in deepspeed_mii and tgi.")
874 parser.add_argument(
875 "--percentile-metrics",
876 type=str,