@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 420ba2dbb6
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 11, 03:45 UTC

Enable aarch64 CPU performance benchmarks (#26494)

Signed-off-by: Ioana Ghiban <ioana.ghiban@arm.com> Signed-off-by: Fadi Arafeh <fadi.arafeh@arm.com> Co-authored-by: Ioana Ghiban <ioana.ghiban@arm.com> Co-authored-by: Fadi Arafeh <fadi.arafeh@arm.com>

Li, Jiang committed Dec 19, 2025 at 20:16 UTC 420ba2dbb6cf9bec13492563731e5f056e4c86f8
6 files changed +213 -18
.buildkite/performance-benchmarks/README.md
+5 -4
@@ -7,7 +7,7 @@ vLLM also maintains a continuous performance benchmark under [perf.vllm.ai](http
7
8 ## Performance benchmark quick overview
9
10 -**Benchmarking Coverage**: latency, throughput and fix-qps serving on B200, A100, H100, Intel® Xeon® Processors and Intel® Gaudi® 3 Accelerators with different models.
10 +**Benchmarking Coverage**: latency, throughput and fix-qps serving on B200, A100, H100, Intel® Xeon® Processors, Intel® Gaudi® 3 Accelerators and Arm® Neoverse™ with different models.
11
12 **Benchmarking Duration**: about 1hr.
13
@@ -23,7 +23,7 @@ bash .buildkite/performance-benchmarks/scripts/run-performance-benchmarks.sh
23
24 Runtime environment variables:
25
26 -- `ON_CPU`: set the value to '1' on Intel® Xeon® Processors. Default value is 0.
26 +- `ON_CPU`: set the value to '1' on Intel® Xeon® and Arm® Neoverse™ Processors. Default value is 0.
27 - `SERVING_JSON`: JSON file to use for the serving tests. Default value is empty string (use default file).
28 - `LATENCY_JSON`: JSON file to use for the latency tests. Default value is empty string (use default file).
29 - `THROUGHPUT_JSON`: JSON file to use for the throughout tests. Default value is empty string (use default file).
@@ -34,8 +34,9 @@ Runtime environment variables:
34
35 See [performance-benchmarks-descriptions.md](performance-benchmarks-descriptions.md) for detailed descriptions, and use `tests/latency-tests.json`, `tests/throughput-tests.json`, `tests/serving-tests.json` to configure the test cases.
36 > NOTE: For Intel® Xeon® Processors, use `tests/latency-tests-cpu.json`, `tests/throughput-tests-cpu.json`, `tests/serving-tests-cpu.json` instead.
37 -For Intel® Gaudi® 3 Accelerators, use `tests/latency-tests-hpu.json`, `tests/throughput-tests-hpu.json`, `tests/serving-tests-hpu.json` instead.
38 ->
37 +> For Intel® Gaudi® 3 Accelerators, use `tests/latency-tests-hpu.json`, `tests/throughput-tests-hpu.json`, `tests/serving-tests-hpu.json` instead.
38 +> For Arm® Neoverse™, use `tests/latency-tests-arm64-cpu.json`, `tests/throughput-tests-arm64-cpu.json`, `tests/serving-tests-arm64-cpu.json` instead.
39 +
40 ### Latency test
41
42 Here is an example of one test inside `latency-tests.json`:
.buildkite/performance-benchmarks/scripts/run-performance-benchmarks.sh
+14 -10
@@ -49,7 +49,11 @@ check_cpus() {
49 echo "Need at least 1 NUMA to run benchmarking."
50 exit 1
51 fi
52 - declare -g gpu_type="cpu"
52 + if [[ "$(uname -m)" == "aarch64" ]] || [[ "$(uname -m)" == "arm64" ]]; then
53 + declare -g gpu_type="arm64-cpu"
54 + else
55 + declare -g gpu_type="cpu"
56 + fi
57 echo "GPU type is $gpu_type"
58 }
59
@@ -207,8 +211,8 @@ run_latency_tests() {
211
212 # check if there is enough GPU to run the test
213 tp=$(echo "$latency_params" | jq -r '.tensor_parallel_size')
210 - if [ "$ON_CPU" == "1" ]; then
211 - pp=$(echo "$latency_params" | jq -r '.pipeline_parallel_size')
214 + if [[ "$ON_CPU" == "1" ]]; then
215 + pp=$(echo "$latency_params" | jq -r '.pipeline_parallel_size // 1')
216 world_size=$(($tp*$pp))
217 if [[ $numa_count -lt $world_size && -z "${REMOTE_HOST}" ]]; then
218 echo "Required world-size $world_size but only $numa_count NUMA nodes found. Skip testcase $test_name."
@@ -276,8 +280,8 @@ run_throughput_tests() {
280
281 # check if there is enough GPU to run the test
282 tp=$(echo "$throughput_params" | jq -r '.tensor_parallel_size')
279 - if [ "$ON_CPU" == "1" ]; then
280 - pp=$(echo "$throughput_params" | jq -r '.pipeline_parallel_size')
283 + if [[ "$ON_CPU" == "1" ]]; then
284 + pp=$(echo "$throughput_params" | jq -r '.pipeline_parallel_size // 1')
285 world_size=$(($tp*$pp))
286 if [[ $numa_count -lt $world_size && -z "${REMOTE_HOST}" ]]; then
287 echo "Required world-size $world_size but only $numa_count NUMA nodes found. Skip testcase $test_name."
@@ -393,8 +397,8 @@ run_serving_tests() {
397
398 # check if there is enough resources to run the test
399 tp=$(echo "$server_params" | jq -r '.tensor_parallel_size')
396 - if [ "$ON_CPU" == "1" ]; then
397 - pp=$(echo "$server_params" | jq -r '.pipeline_parallel_size')
400 + if [[ "$ON_CPU" == "1" ]]; then
401 + pp=$(echo "$server_params" | jq -r '.pipeline_parallel_size // 1')
402 world_size=$(($tp*$pp))
403 if [[ $numa_count -lt $world_size && -z "${REMOTE_HOST}" ]]; then
404 echo "Required world-size $world_size but only $numa_count NUMA nodes found. Skip testcase $test_name."
@@ -496,9 +500,9 @@ run_serving_tests() {
500 main() {
501 local ARCH
502 ARCH=''
499 - if [ "$ON_CPU" == "1" ];then
500 - check_cpus
501 - ARCH='-cpu'
503 + if [[ "$ON_CPU" == "1" ]]; then
504 + check_cpus
505 + ARCH="-$gpu_type"
506 else
507 check_gpus
508 ARCH="$arch_suffix"
.buildkite/performance-benchmarks/tests/latency-tests-arm64-cpu.json new
+26
@@ -0,0 +1,26 @@
1 +[
2 + {
3 + "test_name": "latency_llama8B_tp1",
4 + "environment_variables": {
5 + "VLLM_RPC_TIMEOUT": 100000,
6 + "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
7 + "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
8 + "VLLM_CPU_KVCACHE_SPACE": 40
9 + },
10 + "parameters": {
11 + "model": "meta-llama/Llama-3.1-8B-Instruct",
12 + "tensor_parallel_size": 1,
13 + "load_format": "dummy",
14 + "dtype": "bfloat16",
15 + "distributed_executor_backend": "mp",
16 + "block_size": 128,
17 + "trust_remote_code": "",
18 + "disable_log_stats": "",
19 + "enforce_eager": "",
20 + "max_num_batched_tokens": 2048,
21 + "max_num_seqs": 256,
22 + "num_iters_warmup": 5,
23 + "num_iters": 15
24 + }
25 + }
26 +]
.buildkite/performance-benchmarks/tests/serving-tests-arm64-cpu.json new
+130
@@ -0,0 +1,130 @@
1 +{
2 + "defaults": {
3 + "qps_list": [
4 + "inf"
5 + ],
6 + "max_concurrency_list": [
7 + 12,
8 + 16,
9 + 24,
10 + 32,
11 + 64,
12 + 128,
13 + 200
14 + ],
15 + "server_environment_variables": {
16 + "VLLM_RPC_TIMEOUT": 100000,
17 + "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
18 + "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
19 + "VLLM_CPU_SGL_KERNEL": 1,
20 + "VLLM_CPU_KVCACHE_SPACE": 40
21 + },
22 + "server_parameters": {
23 + "model": "meta-llama/Llama-3.1-8B-Instruct",
24 + "tensor_parallel_size": 1,
25 + "dtype": "bfloat16",
26 + "distributed_executor_backend": "mp",
27 + "block_size": 128,
28 + "trust_remote_code": "",
29 + "disable_log_stats": "",
30 + "enforce_eager": "",
31 + "max_num_batched_tokens": 2048,
32 + "max_num_seqs": 256,
33 + "load_format": "dummy"
34 + },
35 + "client_parameters": {
36 + "model": "meta-llama/Llama-3.1-8B-Instruct",
37 + "backend": "vllm",
38 + "ignore-eos": "",
39 + "num_prompts": 200
40 + }
41 + },
42 + "tests": [
43 + {
44 + "test_name": "serving_llama8B_tp1_sharegpt",
45 + "server_parameters": {
46 + "tensor_parallel_size": 1
47 + },
48 + "client_parameters": {
49 + "dataset_name": "sharegpt",
50 + "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json"
51 + }
52 + },
53 + {
54 + "test_name": "serving_llama8B_tp2_sharegpt",
55 + "server_parameters": {
56 + "tensor_parallel_size": 2
57 + },
58 + "client_parameters": {
59 + "dataset_name": "sharegpt",
60 + "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json"
61 + }
62 + },
63 + {
64 + "test_name": "serving_llama8B_tp1_random_128_128",
65 + "server_parameters": {
66 + "tensor_parallel_size": 1
67 + },
68 + "client_parameters": {
69 + "dataset_name": "random",
70 + "random-input-len": 128,
71 + "random-output-len": 128
72 + }
73 + },
74 + {
75 + "test_name": "serving_llama8B_tp2_random_128_128",
76 + "server_parameters": {
77 + "tensor_parallel_size": 2
78 + },
79 + "client_parameters": {
80 + "dataset_name": "random",
81 + "random-input-len": 128,
82 + "random-output-len": 128
83 + }
84 + },
85 + {
86 + "test_name": "serving_llama8B_tp1_random_128_2048",
87 + "server_parameters": {
88 + "tensor_parallel_size": 1
89 + },
90 + "client_parameters": {
91 + "dataset_name": "random",
92 + "random-input-len": 128,
93 + "random-output-len": 2048
94 + }
95 + },
96 + {
97 + "test_name": "serving_llama8B_tp2_random_128_2048",
98 + "server_parameters": {
99 + "tensor_parallel_size": 2
100 + },
101 + "client_parameters": {
102 + "dataset_name": "random",
103 + "random-input-len": 128,
104 + "random-output-len": 2048
105 + }
106 + },
107 + {
108 + "test_name": "serving_llama8B_tp1_random_2048_128",
109 + "server_parameters": {
110 + "tensor_parallel_size": 1
111 + },
112 + "client_parameters": {
113 + "dataset_name": "random",
114 + "random-input-len": 2048,
115 + "random-output-len": 128
116 + }
117 + },
118 + {
119 + "test_name": "serving_llama8B_tp2_random_2048_128",
120 + "server_parameters": {
121 + "tensor_parallel_size": 2
122 + },
123 + "client_parameters": {
124 + "dataset_name": "random",
125 + "random-input-len": 2048,
126 + "random-output-len": 128
127 + }
128 + }
129 + ]
130 +}
\ No newline at end of file
.buildkite/performance-benchmarks/tests/throughput-tests-arm64-cpu.json new
+27
@@ -0,0 +1,27 @@
1 +[
2 + {
3 + "test_name": "throughput_llama8B_tp1",
4 + "environment_variables": {
5 + "VLLM_RPC_TIMEOUT": 100000,
6 + "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
7 + "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
8 + "VLLM_CPU_KVCACHE_SPACE": 40
9 + },
10 + "parameters": {
11 + "model": "meta-llama/Llama-3.1-8B-Instruct",
12 + "tensor_parallel_size": 1,
13 + "load_format": "dummy",
14 + "dtype": "bfloat16",
15 + "distributed_executor_backend": "mp",
16 + "block_size": 128,
17 + "trust_remote_code": "",
18 + "disable_log_stats": "",
19 + "enforce_eager": "",
20 + "max_num_batched_tokens": 2048,
21 + "max_num_seqs": 256,
22 + "dataset": "./ShareGPT_V3_unfiltered_cleaned_split.json",
23 + "num_prompts": 200,
24 + "backend": "vllm"
25 + }
26 + }
27 +]
docs/benchmarking/dashboard.md
+11 -4
@@ -8,12 +8,19 @@ The results are automatically published to the public [vLLM Performance Dashboar
8 ## Manually Trigger the benchmark
9
10 Use [vllm-ci-test-repo images](https://gallery.ecr.aws/q9t5s3a7/vllm-ci-test-repo) with vLLM benchmark suite.
11 -For CPU environment, please use the image with "-cpu" postfix.
11 +For x86 CPU environment, please use the image with "-cpu" postfix. For AArch64 CPU environment, please use the image with "-arm64-cpu" postfix.
12
13 -Here is an example for docker run command for CPU.
13 +Here is an example for docker run command for CPU. For GPUs skip setting the `ON_CPU` env var.
14
15 ```bash
16 -docker run -it --entrypoint /bin/bash -v /data/huggingface:/root/.cache/huggingface -e HF_TOKEN='' --shm-size=16g --name vllm-cpu-ci public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:1da94e673c257373280026f75ceb4effac80e892-cpu
16 +export VLLM_COMMIT=1da94e673c257373280026f75ceb4effac80e892 # use full commit hash from the main branch
17 +export HF_TOKEN=<valid Hugging Face token>
18 +if [[ "$(uname -m)" == aarch64 || "$(uname -m)" == arm64 ]]; then
19 + IMG_SUFFIX="arm64-cpu"
20 +else
21 + IMG_SUFFIX="cpu"
22 +fi
23 +docker run -it --entrypoint /bin/bash -v /data/huggingface:/root/.cache/huggingface -e HF_TOKEN=$HF_TOKEN -e ON_ARM64_CPU=1 --shm-size=16g --name vllm-cpu-ci public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:${VLLM_COMMIT}-${IMG_SUFFIX}
24 ```
25
26 Then, run below command inside the docker instance.
@@ -26,7 +33,7 @@ When run, benchmark script generates results under **benchmark/results** folder,
33
34 ### Runtime environment variables
35
29 -- `ON_CPU`: set the value to '1' on Intel® Xeon® Processors. Default value is 0.
36 +- `ON_CPU`: set the value to '1' on Intel® Xeon® and Arm® Neoverse™ Processors. Default value is 0.
37 - `SERVING_JSON`: JSON file to use for the serving tests. Default value is empty string (use default file).
38 - `LATENCY_JSON`: JSON file to use for the latency tests. Default value is empty string (use default file).
39 - `THROUGHPUT_JSON`: JSON file to use for the throughout tests. Default value is empty string (use default file).