@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 180 lines 7.72 KB
Rendered Raw
1 # vLLM benchmark suite
2
3 ## Introduction
4
5 This directory contains a benchmarking suite for **developers** to run locally and gain clarity on whether their PR improves/degrades vllm's performance.
6 vLLM also maintains a continuous performance benchmark under [perf.vllm.ai](https://perf.vllm.ai/), hosted under PyTorch CI HUD.
7
8 ## Performance benchmark quick overview
9
10 **Benchmarking Coverage**: latency, throughput and fix-qps serving on B200, A100, H100, Intel® Xeon® Processors, Intel® Gaudi® 3 Accelerators and Arm® Neoverse™ with different models.
11
12 **Benchmarking Duration**: about 1hr.
13
14 **For benchmarking developers**: please try your best to constraint the duration of benchmarking to about 1 hr so that it won't take forever to run.
15
16 ## Trigger the benchmark
17
18 The benchmark needs to be triggered manually:
19
20 ```bash
21 bash .buildkite/performance-benchmarks/scripts/run-performance-benchmarks.sh
22 ```
23
24 Runtime environment variables:
25
26 - `ON_CPU`: set the value to '1' on Intel® Xeon® and Arm® Neoverse™ Processors. Default value is 0.
27 - `SERVING_JSON`: JSON file to use for the serving tests. Default value is empty string (use default file).
28 - `LATENCY_JSON`: JSON file to use for the latency tests. Default value is empty string (use default file).
29 - `THROUGHPUT_JSON`: JSON file to use for the throughout tests. Default value is empty string (use default file).
30 - `REMOTE_HOST`: IP for the remote vLLM service to benchmark. Default value is empty string.
31 - `REMOTE_PORT`: Port for the remote vLLM service to benchmark. Default value is empty string.
32
33 ## Performance benchmark details
34
35 See [performance-benchmarks-descriptions.md](performance-benchmarks-descriptions.md) for detailed descriptions, and use `tests/latency-tests.json`, `tests/throughput-tests.json`, `tests/serving-tests.json` to configure the test cases.
36 > NOTE: For Intel® Xeon® Processors, use `tests/latency-tests-cpu.json`, `tests/throughput-tests-cpu.json`, `tests/serving-tests-cpu.json` instead.
37 > For Intel® Gaudi® 3 Accelerators, use `tests/latency-tests-hpu.json`, `tests/throughput-tests-hpu.json`, `tests/serving-tests-hpu.json` instead.
38 > For Arm® Neoverse™, use `tests/latency-tests-arm64-cpu.json`, `tests/throughput-tests-arm64-cpu.json`, `tests/serving-tests-arm64-cpu.json` instead.
39
40 ### Latency test
41
42 Here is an example of one test inside `latency-tests.json`:
43
44 ```json
45 [
46 {
47 "test_name": "latency_llama8B_tp1",
48 "parameters": {
49 "model": "meta-llama/Meta-Llama-3-8B",
50 "tensor_parallel_size": 1,
51 "load_format": "dummy",
52 "num_iters_warmup": 5,
53 "num_iters": 15
54 }
55 },
56 ]
57 ```
58
59 In this example:
60
61 - The `test_name` attributes is a unique identifier for the test. In `latency-tests.json`, it must start with `latency_`.
62 - The `parameters` attribute control the command line arguments to be used for `vllm bench latency`. Note that please use underline `_` instead of the dash `-` when specifying the command line arguments, and `run-performance-benchmarks.sh` will convert the underline to dash when feeding the arguments to `vllm bench latency`. For example, the corresponding command line arguments for `vllm bench latency` will be `--model meta-llama/Meta-Llama-3-8B --tensor-parallel-size 1 --load-format dummy --num-iters-warmup 5 --num-iters 15`
63
64 Note that the performance numbers are highly sensitive to the value of the parameters. Please make sure the parameters are set correctly.
65
66 WARNING: The benchmarking script will save json results by itself, so please do not configure `--output-json` parameter in the json file.
67
68 ### Throughput test
69
70 The tests are specified in `throughput-tests.json`. The syntax is similar to `latency-tests.json`, except for that the parameters will be fed forward to `vllm bench throughput`.
71
72 The number of this test is also stable -- a slight change on the value of this number might vary the performance numbers by a lot.
73
74 ### Serving test
75
76 We test the throughput by using `vllm bench serve` with request rate = inf to cover the online serving overhead. The corresponding parameters are in `serving-tests.json`, and here is an example:
77
78 ```json
79 [
80 {
81 "test_name": "serving_llama8B_tp1_sharegpt",
82 "qps_list": [1, 4, 16, "inf"],
83 "server_parameters": {
84 "model": "meta-llama/Meta-Llama-3-8B",
85 "tensor_parallel_size": 1,
86 "disable_log_stats": "",
87 "load_format": "dummy"
88 },
89 "client_parameters": {
90 "model": "meta-llama/Meta-Llama-3-8B",
91 "backend": "vllm",
92 "dataset_name": "sharegpt",
93 "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
94 "num_prompts": 200
95 }
96 },
97 ]
98 ```
99
100 Inside this example:
101
102 - The `test_name` attribute is also a unique identifier for the test. It must start with `serving_`.
103 - The `server-parameters` includes the command line arguments for vLLM server.
104 - The `client-parameters` includes the command line arguments for `vllm bench serve`.
105 - The `qps_list` controls the list of qps for test. It will be used to configure the `--request-rate` parameter in `vllm bench serve`
106
107 The number of this test is less stable compared to the delay and latency benchmarks (due to randomized sharegpt dataset sampling inside `benchmark_serving.py`), but a large change on this number (e.g. 5% change) still vary the output greatly.
108
109 WARNING: The benchmarking script will save json results by itself, so please do not configure `--save-results` or other results-saving-related parameters in `serving-tests.json`.
110
111 #### Default Parameters Field
112
113 We can specify default parameters in a JSON field with key `defaults`. Parameters defined in the field are applied globally to all serving tests, and can be overridden in test case fields. Here is an example:
114
115 <details>
116 <summary> An Example of default parameters field </summary>
117
118 ```json
119 {
120 "defaults": {
121 "qps_list": [
122 "inf"
123 ],
124 "server_environment_variables": {
125 "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1
126 },
127 "server_parameters": {
128 "tensor_parallel_size": 1,
129 "dtype": "bfloat16",
130 "block_size": 128,
131 "disable_log_stats": "",
132 "load_format": "dummy"
133 },
134 "client_parameters": {
135 "backend": "vllm",
136 "dataset_name": "random",
137 "random-input-len": 128,
138 "random-output-len": 128,
139 "num_prompts": 200,
140 "ignore-eos": ""
141 }
142 },
143 "tests": [
144 {
145 "test_name": "serving_llama3B_tp2_random_128_128",
146 "server_parameters": {
147 "model": "meta-llama/Llama-3.2-3B-Instruct",
148 "tensor_parallel_size": 2,
149 },
150 "client_parameters": {
151 "model": "meta-llama/Llama-3.2-3B-Instruct",
152 }
153 },
154 {
155 "test_name": "serving_qwen3_tp4_random_128_128",
156 "server_parameters": {
157 "model": "Qwen/Qwen3-14B",
158 "tensor_parallel_size": 4,
159 },
160 "client_parameters": {
161 "model": "Qwen/Qwen3-14B",
162 }
163 },
164 ]
165 }
166 ```
167
168 </details>
169
170 ### Visualizing the results
171
172 The `convert-results-json-to-markdown.py` helps you put the benchmarking results inside a markdown table, by formatting [descriptions.md](performance-benchmarks-descriptions.md) with real benchmarking results.
173 You can find the result presented as a table inside the `buildkite/performance-benchmark` job page.
174 If you do not see the table, please wait till the benchmark finish running.
175 The json version of the table (together with the json version of the benchmark) will be also attached to the markdown file.
176 The raw benchmarking results (in the format of json files) are in the `Artifacts` tab of the benchmarking.
177
178 #### Performance Results Comparison
179
180 Follow the instructions in [performance results comparison](https://docs.vllm.ai/en/latest/benchmarking/dashboard/#performance-results-comparison) to analyze performance results and the sizing guide.