A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 03:30 UTC
Enable CPU nightly performance benchmark and its Markdown report (#18444)
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
Louie Tsai committed
Jul 2, 2025 at 18:50 UTC
9965c47d0d072a74252878ec3c9963a540e9a690
9 files changed
+494
-44
.buildkite/nightly-benchmarks/README.md
+40
-2
@@ -11,7 +11,7 @@ See [vLLM performance dashboard](https://perf.vllm.ai) for the latest performanc
11
12
## Performance benchmark quick overview
13
14
-**Benchmarking Coverage**: latency, throughput and fix-qps serving on A100 (the support for FP8 benchmark on H100 is coming!), with different models.
14
+**Benchmarking Coverage**: latency, throughput and fix-qps serving on A100 (the support for FP8 benchmark on H100 is coming!) and Intel® Xeon® Processors, with different models.
15
16
**Benchmarking Duration**: about 1hr.
17
@@ -31,13 +31,27 @@ Performance benchmark will be triggered when:
31
- A PR being merged into vllm.
32
- Every commit for those PRs with `perf-benchmarks` label AND `ready` label.
33
34
+Manually Trigger the benchmark
35
+
36
+```bash
37
+bash .buildkite/nightly-benchmarks/scripts/run-performance-benchmarks.sh
38
+```
39
+
40
+Runtime environment variables:
41
+- `ON_CPU`: set the value to '1' on Intel® Xeon® Processors. Default value is 0.
42
+- `SERVING_JSON`: JSON file to use for the serving tests. Default value is empty string (use default file).
43
+- `LATENCY_JSON`: JSON file to use for the latency tests. Default value is empty string (use default file).
44
+- `THROUGHPUT_JSON`: JSON file to use for the throughout tests. Default value is empty string (use default file).
45
+- `REMOTE_HOST`: IP for the remote vLLM service to benchmark. Default value is empty string.
46
+- `REMOTE_PORT`: Port for the remote vLLM service to benchmark. Default value is empty string.
47
+
48
Nightly benchmark will be triggered when:
49
- Every commit for those PRs with `perf-benchmarks` label and `nightly-benchmarks` label.
50
51
## Performance benchmark details
52
53
See [performance-benchmarks-descriptions.md](performance-benchmarks-descriptions.md) for detailed descriptions, and use `tests/latency-tests.json`, `tests/throughput-tests.json`, `tests/serving-tests.json` to configure the test cases.
40
-
54
+> NOTE: For Intel® Xeon® Processors, use `tests/latency-tests-cpu.json`, `tests/throughput-tests-cpu.json`, `tests/serving-tests-cpu.json` instead.
55
### Latency test
56
57
Here is an example of one test inside `latency-tests.json`:
@@ -119,6 +133,30 @@ If you do not see the table, please wait till the benchmark finish running.
133
The json version of the table (together with the json version of the benchmark) will be also attached to the markdown file.
134
The raw benchmarking results (in the format of json files) are in the `Artifacts` tab of the benchmarking.
135
136
+The `compare-json-results.py` helps to compare benchmark results JSON files converted using `convert-results-json-to-markdown.py`.
137
+When run, benchmark script generates results under `benchmark/results` folder, along with the `benchmark_results.md` and `benchmark_results.json`.
138
+`compare-json-results.py` compares two `benchmark_results.json` files and provides performance ratio e.g. for Output Tput, Median TTFT and Median TPOT.
139
+
140
+Here is an example using the script to compare result_a and result_b without detail test name.
141
+`python3 compare-json-results.py -f results_a/benchmark_results.json -f results_b/benchmark_results.json --ignore_test_name`
142
+
143
+| | results_a/benchmark_results.json | results_b/benchmark_results.json | perf_ratio |
144
+|----|----------------------------------------|----------------------------------------|----------|
145
+| 0 | 142.633982 | 156.526018 | 1.097396 |
146
+| 1 | 241.620334 | 294.018783 | 1.216863 |
147
+| 2 | 218.298905 | 262.664916 | 1.203235 |
148
+| 3 | 242.743860 | 299.816190 | 1.235113 |
149
+
150
+Here is an example using the script to compare result_a and result_b with detail test name.
151
+`python3 compare-json-results.py -f results_a/benchmark_results.json -f results_b/benchmark_results.json`
152
+| | results_a/benchmark_results.json_name | results_a/benchmark_results.json | results_b/benchmark_results.json_name | results_b/benchmark_results.json | perf_ratio |
153
+|---|---------------------------------------------|----------------------------------------|---------------------------------------------|----------------------------------------|----------|
154
+| 0 | serving_llama8B_tp1_sharegpt_qps_1 | 142.633982 | serving_llama8B_tp1_sharegpt_qps_1 | 156.526018 | 1.097396 |
155
+| 1 | serving_llama8B_tp1_sharegpt_qps_16 | 241.620334 | serving_llama8B_tp1_sharegpt_qps_16 | 294.018783 | 1.216863 |
156
+| 2 | serving_llama8B_tp1_sharegpt_qps_4 | 218.298905 | serving_llama8B_tp1_sharegpt_qps_4 | 262.664916 | 1.203235 |
157
+| 3 | serving_llama8B_tp1_sharegpt_qps_inf | 242.743860 | serving_llama8B_tp1_sharegpt_qps_inf | 299.816190 | 1.235113 |
158
+| 4 | serving_llama8B_tp2_random_1024_128_qps_1 | 96.613390 | serving_llama8B_tp4_random_1024_128_qps_1 | 108.404853 | 1.122048 |
159
+
160
## Nightly test details
161
162
See [nightly-descriptions.md](nightly-descriptions.md) for the detailed description on test workload, models and docker containers of benchmarking other llm engines.
.buildkite/nightly-benchmarks/performance-benchmarks-descriptions.md
+12
-4
@@ -4,7 +4,8 @@
4
- Input length: 32 tokens.
5
- Output length: 128 tokens.
6
- Batch size: fixed (8).
7
-- Models: llama-3.1 8B, llama-3 70B, mixtral 8x7B.
7
+- GPU Models: llama-3.1 8B, llama-3 70B, mixtral 8x7B.
8
+- CPU Models: llama-3.1 8B.
9
- Evaluation metrics: end-to-end latency (mean, median, p99).
10
11
{latency_tests_markdown_table}
@@ -14,7 +15,8 @@
15
- Input length: randomly sample 200 prompts from ShareGPT dataset (with fixed random seed).
16
- Output length: the corresponding output length of these 200 prompts.
17
- Batch size: dynamically determined by vllm to achieve maximum throughput.
17
-- Models: llama-3.1 8B, llama-3 70B, mixtral 8x7B.
18
+- GPU Models: llama-3.1 8B, llama-3 70B, mixtral 8x7B.
19
+- CPU Models: llama-3.1 8B.
20
- Evaluation metrics: throughput.
21
22
{throughput_tests_markdown_table}
@@ -25,12 +27,18 @@
27
- Output length: the corresponding output length of these 200 prompts.
28
- Batch size: dynamically determined by vllm and the arrival pattern of the requests.
29
- **Average QPS (query per second)**: 1, 4, 16 and inf. QPS = inf means all requests come at once. For other QPS values, the arrival time of each query is determined using a random Poisson process (with fixed random seed).
28
-- Models: llama-3.1 8B, llama-3 70B, mixtral 8x7B.
29
-- We also added a speculative decoding test for llama-3 70B, under QPS 2
30
+- GPU Models: llama-3.1 8B, llama-3 70B, mixtral 8x7B.
31
+- We also added a speculative decoding test for llama-3 70B on GPU, under QPS 2
32
+- CPU Models: llama-3.1 8B.
33
- Evaluation metrics: throughput, TTFT (time to the first token, with mean, median and p99), ITL (inter-token latency, with mean, median and p99).
34
+- For CPU, we added random dataset tests to benchmark fixed input/output length with 100 prompts.
35
36
{serving_tests_markdown_table}
37
38
+## Platform Information
39
+
40
+{platform_markdown_table}
41
+
42
## json version of the benchmarking tables
43
44
This section contains the data of the markdown tables above in JSON format.
.buildkite/nightly-benchmarks/scripts/compare-json-results.py
new
+66
@@ -0,0 +1,66 @@
1
+# SPDX-License-Identifier: Apache-2.0
2
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
+import argparse
4
+
5
+import pandas as pd
6
+
7
+
8
+def compare_data_columns(
9
+ files, name_column, data_column, drop_column, ignore_test_name=False
10
+):
11
+ print("\ncompare_data_column: " + data_column)
12
+ frames = []
13
+ compare_frames = []
14
+ for file in files:
15
+ data_df = pd.read_json(file)
16
+ serving_df = data_df.dropna(subset=[drop_column], ignore_index=True)
17
+ if ignore_test_name is False:
18
+ serving_df = serving_df.rename(columns={name_column: file + "_name"})
19
+ frames.append(serving_df[file + "_name"])
20
+ serving_df = serving_df.rename(columns={data_column: file})
21
+ frames.append(serving_df[file])
22
+ compare_frames.append(serving_df[file])
23
+ if len(compare_frames) >= 2:
24
+ # Compare numbers among two files
25
+ ratio_df = compare_frames[1] / compare_frames[0]
26
+ frames.append(ratio_df)
27
+ compare_frames.pop(1)
28
+
29
+ concat_df = pd.concat(frames, axis=1)
30
+ return concat_df
31
+
32
+
33
+if __name__ == "__main__":
34
+ parser = argparse.ArgumentParser()
35
+ parser.add_argument(
36
+ "-f", "--file", action="append", type=str, help="input file name"
37
+ )
38
+ parser.add_argument(
39
+ "--ignore_test_name", action="store_true", help="ignore_test_name or not"
40
+ )
41
+ args = parser.parse_args()
42
+ files = args.file
43
+ print("comparing : " + ", ".join(files))
44
+
45
+ drop_column = "P99"
46
+ name_column = "Test name"
47
+ data_cols_to_compare = ["Output Tput (tok/s)", "Median TTFT (ms)", "Median"]
48
+ html_msgs_for_data_cols = [
49
+ "Compare Output Tokens /n",
50
+ "Median TTFT /n",
51
+ "Median TPOT /n",
52
+ ]
53
+ ignore_test_name = args.ignore_test_name
54
+ with open("perf_comparison.html", "w") as text_file:
55
+ for i in range(len(data_cols_to_compare)):
56
+ output_df = compare_data_columns(
57
+ files,
58
+ name_column,
59
+ data_cols_to_compare[i],
60
+ drop_column,
61
+ ignore_test_name=ignore_test_name,
62
+ )
63
+ print(output_df)
64
+ html = output_df.to_html()
65
+ text_file.write(html_msgs_for_data_cols[i])
66
+ text_file.write(html)
.buildkite/nightly-benchmarks/scripts/convert-results-json-to-markdown.py
+53
-10
@@ -3,9 +3,11 @@
3
4
import json
5
import os
6
+from importlib import util
7
from pathlib import Path
8
9
import pandas as pd
10
+import psutil
11
from tabulate import tabulate
12
13
results_folder = Path("results/")
@@ -29,11 +31,11 @@ throughput_results = []
31
throughput_results_column_mapping = {
32
"test_name": "Test name",
33
"gpu_type": "GPU",
32
- # "num_requests": "# of req.",
33
- # "total_num_tokens": "Total # of tokens",
34
- # "elapsed_time": "Elapsed time (s)",
34
+ "num_requests": "# of req.",
35
+ "total_num_tokens": "Total # of tokens",
36
+ "elapsed_time": "Elapsed time (s)",
37
"requests_per_second": "Tput (req/s)",
36
- # "tokens_per_second": "Tput (tok/s)",
38
+ "tokens_per_second": "Tput (tok/s)",
39
}
40
41
# serving results and the keys that will be printed into markdown
@@ -41,16 +43,18 @@ serving_results = []
43
serving_column_mapping = {
44
"test_name": "Test name",
45
"gpu_type": "GPU",
44
- # "completed": "# of req.",
46
+ "completed": "# of req.",
47
"request_throughput": "Tput (req/s)",
46
- # "input_throughput": "Input Tput (tok/s)",
47
- # "output_throughput": "Output Tput (tok/s)",
48
+ "total_token_throughput": "Total Token Tput (tok/s)",
49
+ "output_throughput": "Output Tput (tok/s)",
50
+ "total_input_tokens": "Total input tokens",
51
+ "total_output_tokens": "Total output tokens",
52
"mean_ttft_ms": "Mean TTFT (ms)",
53
"median_ttft_ms": "Median TTFT (ms)",
54
"p99_ttft_ms": "P99 TTFT (ms)",
51
- # "mean_tpot_ms": "Mean TPOT (ms)",
52
- # "median_tpot_ms": "Median",
53
- # "p99_tpot_ms": "P99",
55
+ "mean_tpot_ms": "Mean TPOT (ms)",
56
+ "median_tpot_ms": "Median",
57
+ "p99_tpot_ms": "P99",
58
"mean_itl_ms": "Mean ITL (ms)",
59
"median_itl_ms": "Median ITL (ms)",
60
"p99_itl_ms": "P99 ITL (ms)",
@@ -75,6 +79,20 @@ def results_to_json(latency, throughput, serving):
79
)
80
81
82
+def get_size_with_unit(bytes, suffix="B"):
83
+ """
84
+ Scale bytes to its proper format
85
+ e.g:
86
+ 1253656 => '1.20MB'
87
+ 1253656678 => '1.17GB'
88
+ """
89
+ factor = 1024
90
+ for unit in ["", "K", "M", "G", "T", "P"]:
91
+ if bytes < factor:
92
+ return f"{bytes:.2f}{unit}{suffix}"
93
+ bytes /= factor
94
+
95
+
96
if __name__ == "__main__":
97
# collect results
98
for test_file in results_folder.glob("*.json"):
@@ -155,6 +173,27 @@ if __name__ == "__main__":
173
serving_results = pd.DataFrame.from_dict(serving_results)
174
throughput_results = pd.DataFrame.from_dict(throughput_results)
175
176
+ svmem = psutil.virtual_memory()
177
+ platform_data = {
178
+ "Physical cores": [psutil.cpu_count(logical=False)],
179
+ "Total cores": [psutil.cpu_count(logical=True)],
180
+ "Total Memory": [get_size_with_unit(svmem.total)],
181
+ }
182
+
183
+ if util.find_spec("numa") is not None:
184
+ from numa import info
185
+
186
+ platform_data["Total NUMA nodes"] = [info.get_num_configured_nodes()]
187
+
188
+ if util.find_spec("cpuinfo") is not None:
189
+ from cpuinfo import get_cpu_info
190
+
191
+ platform_data["CPU Brand"] = [get_cpu_info()["brand_raw"]]
192
+
193
+ platform_results = pd.DataFrame.from_dict(
194
+ platform_data, orient="index", columns=["Platform Info"]
195
+ )
196
+
197
raw_results_json = results_to_json(
198
latency_results, throughput_results, serving_results
199
)
@@ -200,6 +239,9 @@ if __name__ == "__main__":
239
throughput_md_table = tabulate(
240
throughput_results, headers="keys", tablefmt="pipe", showindex=False
241
)
242
+ platform_md_table = tabulate(
243
+ platform_results, headers="keys", tablefmt="pipe", showindex=True
244
+ )
245
246
# document the result
247
with open(results_folder / "benchmark_results.md", "w") as f:
@@ -211,6 +253,7 @@ if __name__ == "__main__":
253
latency_tests_markdown_table=latency_md_table,
254
throughput_tests_markdown_table=throughput_md_table,
255
serving_tests_markdown_table=serving_md_table,
256
+ platform_markdown_table=platform_md_table,
257
benchmarking_results_in_json_string=processed_results_json,
258
)
259
f.write(results)
.buildkite/nightly-benchmarks/scripts/run-performance-benchmarks.sh
+101
-27
@@ -31,6 +31,20 @@ check_gpus() {
31
echo "GPU type is $gpu_type"
32
}
33
34
+check_cpus() {
35
+ # check the number of CPUs and NUMA Node and GPU type.
36
+ declare -g numa_count=$(python3 -c "from numa import info;numa_size = info.get_num_configured_nodes(); print(numa_size)")
37
+ if [[ $numa_count -gt 0 ]]; then
38
+ echo "NUMA found."
39
+ echo $numa_count
40
+ else
41
+ echo "Need at least 1 NUMA to run benchmarking."
42
+ exit 1
43
+ fi
44
+ declare -g gpu_type="cpu"
45
+ echo "GPU type is $gpu_type"
46
+}
47
+
48
check_hf_token() {
49
# check if HF_TOKEN is available and valid
50
if [[ -z "$HF_TOKEN" ]]; then
@@ -69,6 +83,22 @@ json2args() {
83
echo "$args"
84
}
85
86
+json2envs() {
87
+ # transforms the JSON string to environment variables.
88
+ # example:
89
+ # input: { "VLLM_CPU_KVCACHE_SPACE": 5 }
90
+ # output: VLLM_CPU_KVCACHE_SPACE=5
91
+ local json_string=$1
92
+ local args=$(
93
+ echo "$json_string" | jq -r '
94
+ to_entries |
95
+ map((.key ) + "=" + (.value | tostring)) |
96
+ join(" ")
97
+ '
98
+ )
99
+ echo "$args"
100
+}
101
+
102
wait_for_server() {
103
# wait for vllm server to start
104
# return 1 if vllm server crashes
@@ -158,15 +188,24 @@ run_latency_tests() {
188
# get arguments
189
latency_params=$(echo "$params" | jq -r '.parameters')
190
latency_args=$(json2args "$latency_params")
191
+ latency_environment_variables=$(echo "$params" | jq -r '.environment_variables')
192
+ latency_envs=$(json2envs "$latency_environment_variables")
193
194
# check if there is enough GPU to run the test
195
tp=$(echo "$latency_params" | jq -r '.tensor_parallel_size')
164
- if [[ $gpu_count -lt $tp ]]; then
165
- echo "Required tensor-parallel-size $tp but only $gpu_count GPU found. Skip testcase $test_name."
166
- continue
196
+ if [ "$ON_CPU" == "1" ];then
197
+ if [[ $numa_count -lt $tp ]]; then
198
+ echo "Required tensor-parallel-size $tp but only $numa_count NUMA nodes found. Skip testcase $test_name."
199
+ continue
200
+ fi
201
+ else
202
+ if [[ $gpu_count -lt $tp ]]; then
203
+ echo "Required tensor-parallel-size $tp but only $gpu_count GPU found. Skip testcase $test_name."
204
+ continue
205
+ fi
206
fi
207
169
- latency_command="python3 benchmark_latency.py \
208
+ latency_command=" $latency_envs python3 benchmark_latency.py \
209
--output-json $RESULTS_FOLDER/${test_name}.json \
210
$latency_args"
211
@@ -216,15 +255,24 @@ run_throughput_tests() {
255
# get arguments
256
throughput_params=$(echo "$params" | jq -r '.parameters')
257
throughput_args=$(json2args "$throughput_params")
258
+ throughput_environment_variables=$(echo "$params" | jq -r '.environment_variables')
259
+ throughput_envs=$(json2envs "$throughput_environment_variables")
260
261
# check if there is enough GPU to run the test
262
tp=$(echo "$throughput_params" | jq -r '.tensor_parallel_size')
222
- if [[ $gpu_count -lt $tp ]]; then
223
- echo "Required tensor-parallel-size $tp but only $gpu_count GPU found. Skip testcase $test_name."
224
- continue
263
+ if [ "$ON_CPU" == "1" ];then
264
+ if [[ $numa_count -lt $tp ]]; then
265
+ echo "Required tensor-parallel-size $tp but only $numa_count NUMA nodes found. Skip testcase $test_name."
266
+ continue
267
+ fi
268
+ else
269
+ if [[ $gpu_count -lt $tp ]]; then
270
+ echo "Required tensor-parallel-size $tp but only $gpu_count GPU found. Skip testcase $test_name."
271
+ continue
272
+ fi
273
fi
274
227
- throughput_command="python3 benchmark_throughput.py \
275
+ throughput_command=" $throughput_envs python3 benchmark_throughput.py \
276
--output-json $RESULTS_FOLDER/${test_name}.json \
277
$throughput_args"
278
@@ -272,18 +320,27 @@ run_serving_tests() {
320
321
# get client and server arguments
322
server_params=$(echo "$params" | jq -r '.server_parameters')
323
+ server_envs=$(echo "$params" | jq -r '.server_environment_variables')
324
client_params=$(echo "$params" | jq -r '.client_parameters')
325
server_args=$(json2args "$server_params")
326
+ server_envs=$(json2envs "$server_envs")
327
client_args=$(json2args "$client_params")
328
qps_list=$(echo "$params" | jq -r '.qps_list')
329
qps_list=$(echo "$qps_list" | jq -r '.[] | @sh')
330
echo "Running over qps list $qps_list"
331
282
- # check if there is enough GPU to run the test
332
+ # check if there is enough resources to run the test
333
tp=$(echo "$server_params" | jq -r '.tensor_parallel_size')
284
- if [[ $gpu_count -lt $tp ]]; then
285
- echo "Required tensor-parallel-size $tp but only $gpu_count GPU found. Skip testcase $test_name."
286
- continue
334
+ if [ "$ON_CPU" == "1" ];then
335
+ if [[ $numa_count -lt $tp ]]; then
336
+ echo "Required tensor-parallel-size $tp but only $numa_count NUMA nodes found. Skip testcase $test_name."
337
+ continue
338
+ fi
339
+ else
340
+ if [[ $gpu_count -lt $tp ]]; then
341
+ echo "Required tensor-parallel-size $tp but only $gpu_count GPU found. Skip testcase $test_name."
342
+ continue
343
+ fi
344
fi
345
346
# check if server model and client model is aligned
@@ -294,23 +351,33 @@ run_serving_tests() {
351
continue
352
fi
353
297
- server_command="python3 \
354
+ server_command="$server_envs python3 \
355
-m vllm.entrypoints.openai.api_server \
356
$server_args"
357
358
# run the server
359
echo "Running test case $test_name"
360
echo "Server command: $server_command"
304
- bash -c "$server_command" &
305
- server_pid=$!
306
-
307
- # wait until the server is alive
308
- if wait_for_server; then
309
- echo ""
310
- echo "vllm server is up and running."
361
+ # support remote vllm server
362
+ client_remote_args=""
363
+ if [[ -z "${REMOTE_HOST}" ]]; then
364
+ bash -c "$server_command" &
365
+ server_pid=$!
366
+ # wait until the server is alive
367
+ if wait_for_server; then
368
+ echo ""
369
+ echo "vLLM server is up and running."
370
+ else
371
+ echo ""
372
+ echo "vLLM failed to start within the timeout period."
373
+ fi
374
else
312
- echo ""
313
- echo "vllm failed to start within the timeout period."
375
+ server_command="Using Remote Server $REMOTE_HOST $REMOTE_PORT"
376
+ if [[ ${REMOTE_PORT} ]]; then
377
+ client_remote_args=" --host=$REMOTE_HOST --port=$REMOTE_PORT "
378
+ else
379
+ client_remote_args=" --host=$REMOTE_HOST "
380
+ fi
381
fi
382
383
# iterate over different QPS
@@ -332,7 +399,7 @@ run_serving_tests() {
399
--result-filename ${new_test_name}.json \
400
--request-rate $qps \
401
--metadata "tensor_parallel_size=$tp" \
335
- $client_args"
402
+ $client_args $client_remote_args "
403
404
echo "Running test case $test_name with qps $qps"
405
echo "Client command: $client_command"
@@ -360,7 +427,14 @@ run_serving_tests() {
427
}
428
429
main() {
363
- check_gpus
430
+ local ARCH
431
+ ARCH=''
432
+ if [ "$ON_CPU" == "1" ];then
433
+ check_cpus
434
+ ARCH='-cpu'
435
+ else
436
+ check_gpus
437
+ fi
438
check_hf_token
439
440
# Set to v1 to run v1 benchmark
@@ -386,9 +460,9 @@ main() {
460
QUICK_BENCHMARK_ROOT=../.buildkite/nightly-benchmarks/
461
462
# benchmarking
389
- run_serving_tests $QUICK_BENCHMARK_ROOT/tests/serving-tests.json
390
- run_latency_tests $QUICK_BENCHMARK_ROOT/tests/latency-tests.json
391
- run_throughput_tests $QUICK_BENCHMARK_ROOT/tests/throughput-tests.json
463
+ run_serving_tests $QUICK_BENCHMARK_ROOT/tests/"${SERVING_JSON:-serving-tests$ARCH.json}"
464
+ run_latency_tests $QUICK_BENCHMARK_ROOT/tests/"${LATENCY_JSON:-latency-tests$ARCH.json}"
465
+ run_throughput_tests $QUICK_BENCHMARK_ROOT/tests/"${THROUGHPUT_JSON:-throughput-tests$ARCH.json}"
466
467
# postprocess benchmarking results
468
pip install tabulate pandas
.buildkite/nightly-benchmarks/tests/latency-tests-cpu.json
new
+30
@@ -0,0 +1,30 @@
1
+[
2
+ {
3
+ "test_name": "latency_llama8B_tp1",
4
+ "environment_variables": {
5
+ "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
6
+ "VLLM_CPU_KVCACHE_SPACE": 40
7
+ },
8
+ "parameters": {
9
+ "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
10
+ "tensor_parallel_size": 1,
11
+ "load_format": "dummy",
12
+ "num_iters_warmup": 5,
13
+ "num_iters": 15
14
+ }
15
+ },
16
+ {
17
+ "test_name": "latency_llama8B_tp4",
18
+ "environment_variables": {
19
+ "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
20
+ "VLLM_CPU_KVCACHE_SPACE": 40
21
+ },
22
+ "parameters": {
23
+ "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
24
+ "tensor_parallel_size": 4,
25
+ "load_format": "dummy",
26
+ "num_iters_warmup": 5,
27
+ "num_iters": 15
28
+ }
29
+ }
30
+]
.buildkite/nightly-benchmarks/tests/serving-tests-cpu.json
new
+158
@@ -0,0 +1,158 @@
1
+[
2
+ {
3
+ "test_name": "serving_llama8B_tp1_sharegpt",
4
+ "qps_list": [1, 4, 16, "inf"],
5
+ "server_environment_variables": {
6
+ "VLLM_RPC_TIMEOUT": 100000,
7
+ "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
8
+ "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
9
+ "VLLM_CPU_KVCACHE_SPACE": 40
10
+ },
11
+ "server_parameters": {
12
+ "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
13
+ "tensor_parallel_size": 1,
14
+ "dtype": "bfloat16",
15
+ "distributed_executor_backend": "mp",
16
+ "block_size": 128,
17
+ "trust_remote_code": "",
18
+ "disable_log_stats": "",
19
+ "disable_log_requests": "",
20
+ "enforce_eager": "",
21
+ "load_format": "dummy"
22
+ },
23
+ "client_parameters": {
24
+ "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
25
+ "backend": "vllm",
26
+ "dataset_name": "sharegpt",
27
+ "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
28
+ "max_concurrency": 60,
29
+ "num_prompts": 200
30
+ }
31
+ },
32
+ {
33
+ "test_name": "serving_llama8B_tp2_sharegpt",
34
+ "qps_list": [1, 4, 16, "inf"],
35
+ "server_environment_variables": {
36
+ "VLLM_RPC_TIMEOUT": 100000,
37
+ "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
38
+ "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
39
+ "VLLM_CPU_KVCACHE_SPACE": 40
40
+ },
41
+ "server_parameters": {
42
+ "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
43
+ "tensor_parallel_size": 2,
44
+ "dtype": "bfloat16",
45
+ "distributed_executor_backend": "mp",
46
+ "block_size": 128,
47
+ "trust_remote_code": "",
48
+ "disable_log_stats": "",
49
+ "disable_log_requests": "",
50
+ "enforce_eager": "",
51
+ "load_format": "dummy"
52
+ },
53
+ "client_parameters": {
54
+ "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
55
+ "backend": "vllm",
56
+ "dataset_name": "sharegpt",
57
+ "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
58
+ "max_concurrency": 60,
59
+ "num_prompts": 200
60
+ }
61
+ },
62
+ {
63
+ "test_name": "serving_llama8B_tp4_sharegpt",
64
+ "qps_list": [1, 4, 16, "inf"],
65
+ "server_environment_variables": {
66
+ "VLLM_RPC_TIMEOUT": 100000,
67
+ "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
68
+ "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
69
+ "VLLM_CPU_KVCACHE_SPACE": 40
70
+ },
71
+ "server_parameters": {
72
+ "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
73
+ "tensor_parallel_size": 4,
74
+ "dtype": "bfloat16",
75
+ "distributed_executor_backend": "mp",
76
+ "block_size": 128,
77
+ "trust_remote_code": "",
78
+ "disable_log_stats": "",
79
+ "disable_log_requests": "",
80
+ "enforce_eager": "",
81
+ "load_format": "dummy"
82
+ },
83
+ "client_parameters": {
84
+ "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
85
+ "backend": "vllm",
86
+ "dataset_name": "sharegpt",
87
+ "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
88
+ "max_concurrency": 60,
89
+ "num_prompts": 200
90
+ }
91
+ },
92
+ {
93
+ "test_name": "serving_llama8B_tp4_random_1024_128",
94
+ "qps_list": [1, 4, 16, "inf"],
95
+ "server_environment_variables": {
96
+ "VLLM_RPC_TIMEOUT": 100000,
97
+ "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
98
+ "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
99
+ "VLLM_CPU_KVCACHE_SPACE": 40
100
+ },
101
+ "server_parameters": {
102
+ "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
103
+ "tensor_parallel_size": 4,
104
+ "dtype": "bfloat16",
105
+ "distributed_executor_backend": "mp",
106
+ "block_size": 128,
107
+ "trust_remote_code": "",
108
+ "enable_chunked_prefill": "",
109
+ "disable_log_stats": "",
110
+ "disable_log_requests": "",
111
+ "enforce_eager": "",
112
+ "load_format": "dummy"
113
+ },
114
+ "client_parameters": {
115
+ "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
116
+ "backend": "vllm",
117
+ "dataset_name": "random",
118
+ "random-input-len": 1024,
119
+ "random-output-len": 128,
120
+ "ignore-eos": "",
121
+ "max_concurrency": 100,
122
+ "num_prompts": 100
123
+ }
124
+ },
125
+ {
126
+ "test_name": "serving_llama8B_pp6_random_1024_128",
127
+ "qps_list": [1, 4, 16, "inf"],
128
+ "server_environment_variables": {
129
+ "VLLM_RPC_TIMEOUT": 100000,
130
+ "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
131
+ "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
132
+ "VLLM_CPU_KVCACHE_SPACE": 40
133
+ },
134
+ "server_parameters": {
135
+ "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
136
+ "pipeline_parallel_size": 6,
137
+ "dtype": "bfloat16",
138
+ "distributed_executor_backend": "mp",
139
+ "block_size": 128,
140
+ "trust_remote_code": "",
141
+ "enable_chunked_prefill": "",
142
+ "disable_log_stats": "",
143
+ "disable_log_requests": "",
144
+ "enforce_eager": "",
145
+ "load_format": "dummy"
146
+ },
147
+ "client_parameters": {
148
+ "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
149
+ "backend": "vllm",
150
+ "dataset_name": "random",
151
+ "random-input-len": 1024,
152
+ "random-output-len": 128,
153
+ "ignore-eos": "",
154
+ "max_concurrency": 100,
155
+ "num_prompts": 100
156
+ }
157
+ }
158
+]
.buildkite/nightly-benchmarks/tests/throughput-tests-cpu.json
new
+32
@@ -0,0 +1,32 @@
1
+[
2
+ {
3
+ "test_name": "throughput_llama8B_tp1",
4
+ "environment_variables": {
5
+ "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
6
+ "VLLM_CPU_KVCACHE_SPACE": 40
7
+ },
8
+ "parameters": {
9
+ "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
10
+ "tensor_parallel_size": 1,
11
+ "load_format": "dummy",
12
+ "dataset": "./ShareGPT_V3_unfiltered_cleaned_split.json",
13
+ "num_prompts": 200,
14
+ "backend": "vllm"
15
+ }
16
+ },
17
+ {
18
+ "test_name": "throughput_llama8B_tp4",
19
+ "environment_variables": {
20
+ "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
21
+ "VLLM_CPU_KVCACHE_SPACE": 40
22
+ },
23
+ "parameters": {
24
+ "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
25
+ "tensor_parallel_size": 4,
26
+ "load_format": "dummy",
27
+ "dataset": "./ShareGPT_V3_unfiltered_cleaned_split.json",
28
+ "num_prompts": 200,
29
+ "backend": "vllm"
30
+ }
31
+ }
32
+]
docker/Dockerfile.cpu
+2
-1
@@ -25,7 +25,7 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
25
--mount=type=cache,target=/var/lib/apt,sharing=locked \
26
apt-get update -y \
27
&& apt-get install -y --no-install-recommends ccache git curl wget ca-certificates \
28
- gcc-12 g++-12 libtcmalloc-minimal4 libnuma-dev ffmpeg libsm6 libxext6 libgl1 \
28
+ gcc-12 g++-12 libtcmalloc-minimal4 libnuma-dev ffmpeg libsm6 libxext6 libgl1 jq lsof \
29
&& update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 10 --slave /usr/bin/g++ g++ /usr/bin/g++-12 \
30
&& curl -LsSf https://astral.sh/uv/install.sh | sh
31
@@ -134,6 +134,7 @@ ADD ./tests/ ./tests/
134
ADD ./examples/ ./examples/
135
ADD ./benchmarks/ ./benchmarks/
136
ADD ./vllm/collect_env.py .
137
+ADD ./.buildkite/ ./.buildkite/
138
139
# install development dependencies (for testing)
140
RUN --mount=type=cache,target=/root/.cache/uv \