A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 12, 09:45 UTC
[Doc] Reorganize benchmark docs (#29658)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Cyrus Leung committed
Nov 28, 2025 at 17:19 UTC
ccbdf51bd57761a7a7e7a5adf685fcec67c9c1bd
5 files changed
+291
-292
docs/.nav.yml
+5
@@ -52,6 +52,11 @@ nav:
52
- Plugins:
53
- design/*plugin*.md
54
- design/*
55
+ - Benchmarking:
56
+ - benchmarking/README.md
57
+ - benchmarking/cli.md
58
+ - benchmarking/sweeps.md
59
+ - benchmarking/dashboard.md
60
- API Reference:
61
- api/README.md
62
- api/vllm
docs/benchmarking/README.md
new
+7
@@ -0,0 +1,7 @@
1
+# Benchmark Suites
2
+
3
+vLLM provides comprehensive benchmarking tools for performance testing and evaluation:
4
+
5
+- **[Benchmark CLI](./cli.md)**: `vllm bench` CLI tools and specialized benchmark scripts for interactive performance testing.
6
+- **[Parameter Sweeps](./sweeps.md)**: Automate `vllm bench` runs for multiple configurations, useful for [optimization and tuning](../configuration/optimization.md).
7
+- **[Performance Dashboard](./dashboard.md)**: Automated CI that publishes benchmarks on each commit.
docs/benchmarking/cli.md
renamed
+43
-292
@@ -1,22 +1,10 @@
1
----
2
-toc_depth: 4
3
----
1
+# Benchmark CLI
2
5
-# Benchmark Suites
3
+This section guides you through running benchmark tests with the extensive datasets supported on vLLM.
4
7
-vLLM provides comprehensive benchmarking tools for performance testing and evaluation:
5
+It's a living document, updated as new features and datasets become available.
6
9
-- **[Benchmark CLI](#benchmark-cli)**: `vllm bench` CLI tools and specialized benchmark scripts for interactive performance testing
10
-- **[Parameter sweeps](#parameter-sweeps)**: Automate `vllm bench` runs for multiple configurations
11
-- **[Performance benchmarks](#performance-benchmarks)**: Automated CI benchmarks for development
12
-
13
-## Benchmark CLI
14
-
15
-This section guides you through running benchmark tests with the extensive
16
-datasets supported on vLLM. It's a living document, updated as new features and datasets
17
-become available.
18
-
19
-### Dataset Overview
7
+## Dataset Overview
8
9
<style>
10
th {
@@ -59,9 +47,9 @@ Legend:
47
--dataset-path /datasets/VisionArena-Chat/ --hf-name lmarena-ai/VisionArena-Chat
48
```
49
62
-### Examples
50
+## Examples
51
64
-#### 🚀 Online Benchmark
52
+### 🚀 Online Benchmark
53
54
<details class="admonition abstract" markdown="1">
55
<summary>Show more</summary>
@@ -112,7 +100,7 @@ P99 ITL (ms): 8.39
100
==================================================
101
```
102
115
-##### Custom Dataset
103
+#### Custom Dataset
104
105
If the dataset you want to benchmark is not supported yet in vLLM, even then you can benchmark on it using `CustomDataset`. Your data needs to be in `.jsonl` format and needs to have "prompt" field per entry, e.g., data.jsonl
106
@@ -145,7 +133,7 @@ vllm bench serve --port 9001 --save-result --save-detailed \
133
134
You can skip applying chat template if your data already has it by using `--custom-skip-chat-template`.
135
148
-##### VisionArena Benchmark for Vision Language Models
136
+#### VisionArena Benchmark for Vision Language Models
137
138
```bash
139
# need a model with vision capability here
@@ -163,7 +151,7 @@ vllm bench serve \
151
--num-prompts 1000
152
```
153
166
-##### InstructCoder Benchmark with Speculative Decoding
154
+#### InstructCoder Benchmark with Speculative Decoding
155
156
``` bash
157
vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
@@ -180,7 +168,7 @@ vllm bench serve \
168
--num-prompts 2048
169
```
170
183
-##### Spec Bench Benchmark with Speculative Decoding
171
+#### Spec Bench Benchmark with Speculative Decoding
172
173
``` bash
174
vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
@@ -217,7 +205,7 @@ vllm bench serve \
205
--spec-bench-category "summarization"
206
```
207
220
-##### Other HuggingFaceDataset Examples
208
+#### Other HuggingFaceDataset Examples
209
210
```bash
211
vllm serve Qwen/Qwen2-VL-7B-Instruct
@@ -283,7 +271,7 @@ vllm bench serve \
271
--blazedit-max-distance 0.99
272
```
273
286
-##### Running With Sampling Parameters
274
+#### Running With Sampling Parameters
275
276
When using OpenAI-compatible backends such as `vllm`, optional sampling
277
parameters can be specified. Example client command:
@@ -301,7 +289,7 @@ vllm bench serve \
289
--num-prompts 10
290
```
291
304
-##### Running With Ramp-Up Request Rate
292
+#### Running With Ramp-Up Request Rate
293
294
The benchmark tool also supports ramping up the request rate over the
295
duration of the benchmark run. This can be useful for stress testing the
@@ -318,11 +306,11 @@ The following arguments can be used to control the ramp-up:
306
- `--ramp-up-start-rps`: The request rate at the beginning of the benchmark.
307
- `--ramp-up-end-rps`: The request rate at the end of the benchmark.
308
321
-##### Load Pattern Configuration
309
+#### Load Pattern Configuration
310
311
vLLM's benchmark serving script provides sophisticated load pattern simulation capabilities through three key parameters that control request generation and concurrency behavior:
312
325
-###### Load Pattern Control Parameters
313
+##### Load Pattern Control Parameters
314
315
- `--request-rate`: Controls the target request generation rate (requests per second). Set to `inf` for maximum throughput testing or finite values for controlled load simulation.
316
- `--burstiness`: Controls traffic variability using a Gamma distribution (range: > 0). Lower values create bursty traffic, higher values create uniform traffic.
@@ -387,7 +375,7 @@ Using KV cache metrics for load pattern configuration:
375
376
</details>
377
390
-#### 📈 Offline Throughput Benchmark
378
+### 📈 Offline Throughput Benchmark
379
380
<details class="admonition abstract" markdown="1">
381
<summary>Show more</summary>
@@ -408,7 +396,7 @@ Total num prompt tokens: 5014
396
Total num output tokens: 1500
397
```
398
411
-##### VisionArena Benchmark for Vision Language Models
399
+#### VisionArena Benchmark for Vision Language Models
400
401
```bash
402
vllm bench throughput \
@@ -428,7 +416,7 @@ Total num prompt tokens: 14527
416
Total num output tokens: 1280
417
```
418
431
-##### InstructCoder Benchmark with Speculative Decoding
419
+#### InstructCoder Benchmark with Speculative Decoding
420
421
``` bash
422
VLLM_WORKER_MULTIPROC_METHOD=spawn \
@@ -451,7 +439,7 @@ Total num prompt tokens: 261136
439
Total num output tokens: 204800
440
```
441
454
-##### Other HuggingFaceDataset Examples
442
+#### Other HuggingFaceDataset Examples
443
444
`lmms-lab/LLaVA-OneVision-Data`:
445
@@ -509,20 +497,20 @@ vllm bench throughput \
497
498
</details>
499
512
-#### 🛠️ Structured Output Benchmark
500
+### 🛠️ Structured Output Benchmark
501
502
<details class="admonition abstract" markdown="1">
503
<summary>Show more</summary>
504
505
Benchmark the performance of structured output generation (JSON, grammar, regex).
506
519
-##### Server Setup
507
+#### Server Setup
508
509
```bash
510
vllm serve NousResearch/Hermes-3-Llama-3.1-8B
511
```
512
525
-##### JSON Schema Benchmark
513
+#### JSON Schema Benchmark
514
515
```bash
516
python3 benchmarks/benchmark_serving_structured_output.py \
@@ -534,7 +522,7 @@ python3 benchmarks/benchmark_serving_structured_output.py \
522
--num-prompts 1000
523
```
524
537
-##### Grammar-based Generation Benchmark
525
+#### Grammar-based Generation Benchmark
526
527
```bash
528
python3 benchmarks/benchmark_serving_structured_output.py \
@@ -546,7 +534,7 @@ python3 benchmarks/benchmark_serving_structured_output.py \
534
--num-prompts 1000
535
```
536
549
-##### Regex-based Generation Benchmark
537
+#### Regex-based Generation Benchmark
538
539
```bash
540
python3 benchmarks/benchmark_serving_structured_output.py \
@@ -557,7 +545,7 @@ python3 benchmarks/benchmark_serving_structured_output.py \
545
--num-prompts 1000
546
```
547
560
-##### Choice-based Generation Benchmark
548
+#### Choice-based Generation Benchmark
549
550
```bash
551
python3 benchmarks/benchmark_serving_structured_output.py \
@@ -568,7 +556,7 @@ python3 benchmarks/benchmark_serving_structured_output.py \
556
--num-prompts 1000
557
```
558
571
-##### XGrammar Benchmark Dataset
559
+#### XGrammar Benchmark Dataset
560
561
```bash
562
python3 benchmarks/benchmark_serving_structured_output.py \
@@ -581,14 +569,14 @@ python3 benchmarks/benchmark_serving_structured_output.py \
569
570
</details>
571
584
-#### 📚 Long Document QA Benchmark
572
+### 📚 Long Document QA Benchmark
573
574
<details class="admonition abstract" markdown="1">
575
<summary>Show more</summary>
576
577
Benchmark the performance of long document question-answering with prefix caching.
578
591
-##### Basic Long Document QA Test
579
+#### Basic Long Document QA Test
580
581
```bash
582
python3 benchmarks/benchmark_long_document_qa_throughput.py \
@@ -600,7 +588,7 @@ python3 benchmarks/benchmark_long_document_qa_throughput.py \
588
--repeat-count 5
589
```
590
603
-##### Different Repeat Modes
591
+#### Different Repeat Modes
592
593
```bash
594
# Random mode (default) - shuffle prompts randomly
@@ -633,14 +621,14 @@ python3 benchmarks/benchmark_long_document_qa_throughput.py \
621
622
</details>
623
636
-#### 🗂️ Prefix Caching Benchmark
624
+### 🗂️ Prefix Caching Benchmark
625
626
<details class="admonition abstract" markdown="1">
627
<summary>Show more</summary>
628
629
Benchmark the efficiency of automatic prefix caching.
630
643
-##### Fixed Prompt with Prefix Caching
631
+#### Fixed Prompt with Prefix Caching
632
633
```bash
634
python3 benchmarks/benchmark_prefix_caching.py \
@@ -651,7 +639,7 @@ python3 benchmarks/benchmark_prefix_caching.py \
639
--input-length-range 128:256
640
```
641
654
-##### ShareGPT Dataset with Prefix Caching
642
+#### ShareGPT Dataset with Prefix Caching
643
644
```bash
645
# download dataset
@@ -682,14 +670,14 @@ vllm bench serve \
670
671
</details>
672
685
-#### ⚡ Request Prioritization Benchmark
673
+### ⚡ Request Prioritization Benchmark
674
675
<details class="admonition abstract" markdown="1">
676
<summary>Show more</summary>
677
678
Benchmark the performance of request prioritization in vLLM.
679
692
-##### Basic Prioritization Test
680
+#### Basic Prioritization Test
681
682
```bash
683
python3 benchmarks/benchmark_prioritization.py \
@@ -700,7 +688,7 @@ python3 benchmarks/benchmark_prioritization.py \
688
--scheduling-policy priority
689
```
690
703
-##### Multiple Sequences per Prompt
691
+#### Multiple Sequences per Prompt
692
693
```bash
694
python3 benchmarks/benchmark_prioritization.py \
@@ -714,14 +702,14 @@ python3 benchmarks/benchmark_prioritization.py \
702
703
</details>
704
717
-#### 👁️ Multi-Modal Benchmark
705
+### 👁️ Multi-Modal Benchmark
706
707
<details class="admonition abstract" markdown="1">
708
<summary>Show more</summary>
709
710
Benchmark the performance of multi-modal requests in vLLM.
711
724
-##### Images (ShareGPT4V)
712
+#### Images (ShareGPT4V)
713
714
Start vLLM:
715
@@ -747,7 +735,7 @@ vllm bench serve \
735
--endpoint /v1/chat/completions
736
```
737
750
-##### Videos (ShareGPT4Video)
738
+#### Videos (ShareGPT4Video)
739
740
Start vLLM:
741
@@ -773,7 +761,7 @@ vllm bench serve \
761
--endpoint /v1/chat/completions
762
```
763
776
-##### Synthetic Random Images (random-mm)
764
+#### Synthetic Random Images (random-mm)
765
766
Generate synthetic image inputs alongside random text prompts to stress-test vision models without external datasets.
767
@@ -846,14 +834,14 @@ This should be seen as an edge case, and if this behavior can be avoided by sett
834
835
</details>
836
849
-#### Embedding Benchmark
837
+### Embedding Benchmark
838
839
Benchmark the performance of embedding requests in vLLM.
840
841
<details class="admonition abstract" markdown="1">
842
<summary>Show more</summary>
843
856
-##### Text Embeddings
844
+#### Text Embeddings
845
846
Unlike generative models which use Completions API or Chat Completions API,
847
you should set `--backend openai-embeddings` and `--endpoint /v1/embeddings` to use the Embeddings API.
@@ -879,7 +867,7 @@ vllm bench serve \
867
--dataset-path <your data path>/ShareGPT_V3_unfiltered_cleaned_split.json
868
```
869
882
-##### Multi-modal Embeddings
870
+#### Multi-modal Embeddings
871
872
Unlike generative models which use Completions API or Chat Completions API,
873
you should set `--endpoint /v1/embeddings` to use the Embeddings API. The backend to use depends on the model:
@@ -944,7 +932,7 @@ vllm bench serve \
932
933
</details>
934
947
-#### Reranker Benchmark
935
+### Reranker Benchmark
936
937
Benchmark the performance of rerank requests in vLLM.
938
@@ -988,240 +976,3 @@ to account for the extra prompt which is the query. The token accounting to repo
976
throughput numbers correctly is also adjusted.
977
978
</details>
991
-
992
-## Parameter Sweeps
993
-
994
-### Online Benchmark
995
-
996
-[`vllm/benchmarks/sweep/serve.py`](../../vllm/benchmarks/sweep/serve.py) automatically starts `vllm serve` and runs `vllm bench serve` to evaluate vLLM over multiple configurations.
997
-
998
-Follow these steps to run the script:
999
-
1000
-1. Construct the base command to `vllm serve`, and pass it to the `--serve-cmd` option.
1001
-2. Construct the base command to `vllm bench serve`, and pass it to the `--bench-cmd` option.
1002
-3. (Optional) If you would like to vary the settings of `vllm serve`, create a new JSON file and populate it with the parameter combinations you want to test. Pass the file path to `--serve-params`.
1003
-
1004
- - Example: Tuning `--max-num-seqs` and `--max-num-batched-tokens`:
1005
-
1006
- ```json
1007
- [
1008
- {
1009
- "max_num_seqs": 32,
1010
- "max_num_batched_tokens": 1024
1011
- },
1012
- {
1013
- "max_num_seqs": 64,
1014
- "max_num_batched_tokens": 1024
1015
- },
1016
- {
1017
- "max_num_seqs": 64,
1018
- "max_num_batched_tokens": 2048
1019
- },
1020
- {
1021
- "max_num_seqs": 128,
1022
- "max_num_batched_tokens": 2048
1023
- },
1024
- {
1025
- "max_num_seqs": 128,
1026
- "max_num_batched_tokens": 4096
1027
- },
1028
- {
1029
- "max_num_seqs": 256,
1030
- "max_num_batched_tokens": 4096
1031
- }
1032
- ]
1033
- ```
1034
-
1035
-4. (Optional) If you would like to vary the settings of `vllm bench serve`, create a new JSON file and populate it with the parameter combinations you want to test. Pass the file path to `--bench-params`.
1036
-
1037
- - Example: Using different input/output lengths for random dataset:
1038
-
1039
- ```json
1040
- [
1041
- {
1042
- "random_input_len": 128,
1043
- "random_output_len": 32
1044
- },
1045
- {
1046
- "random_input_len": 256,
1047
- "random_output_len": 64
1048
- },
1049
- {
1050
- "random_input_len": 512,
1051
- "random_output_len": 128
1052
- }
1053
- ]
1054
- ```
1055
-
1056
-5. Determine where you want to save the results, and pass that to `--output-dir`.
1057
-
1058
-Example command:
1059
-
1060
-```bash
1061
-vllm bench sweep serve \
1062
- --serve-cmd 'vllm serve meta-llama/Llama-2-7b-chat-hf' \
1063
- --bench-cmd 'vllm bench serve --model meta-llama/Llama-2-7b-chat-hf --backend vllm --endpoint /v1/completions --dataset-name sharegpt --dataset-path benchmarks/ShareGPT_V3_unfiltered_cleaned_split.json' \
1064
- --serve-params benchmarks/serve_hparams.json \
1065
- --bench-params benchmarks/bench_hparams.json \
1066
- -o benchmarks/results
1067
-```
1068
-
1069
-!!! important
1070
- If both `--serve-params` and `--bench-params` are passed, the script will iterate over the Cartesian product between them.
1071
- You can use `--dry-run` to preview the commands to be run.
1072
-
1073
- We only start the server once for each `--serve-params`, and keep it running for multiple `--bench-params`.
1074
- Between each benchmark run, we call the `/reset_prefix_cache` and `/reset_mm_cache` endpoints to get a clean slate for the next run.
1075
- In case you are using a custom `--serve-cmd`, you can override the commands used for resetting the state by setting `--after-bench-cmd`.
1076
-
1077
-!!! note
1078
- By default, each parameter combination is run 3 times to make the results more reliable. You can adjust the number of runs by setting `--num-runs`.
1079
-
1080
-!!! tip
1081
- You can use the `--resume` option to continue the parameter sweep if one of the runs failed.
1082
-
1083
-### SLA Auto-Tuner
1084
-
1085
-[`vllm/benchmarks/sweep/serve_sla.py`](../../vllm/benchmarks/sweep/serve_sla.py) is a wrapper over [`vllm/benchmarks/sweep/serve.py`](../../vllm/benchmarks/sweep/serve.py) that tunes either the request rate or concurrency (choose using `--sla-variable`) in order to satisfy the SLA constraints given by `--sla-params`.
1086
-
1087
-For example, to ensure E2E latency within different target values for 99% of requests:
1088
-
1089
-```json
1090
-[
1091
- {
1092
- "p99_e2el_ms": "<=200"
1093
- },
1094
- {
1095
- "p99_e2el_ms": "<=500"
1096
- },
1097
- {
1098
- "p99_e2el_ms": "<=1000"
1099
- },
1100
- {
1101
- "p99_e2el_ms": "<=2000"
1102
- }
1103
-]
1104
-```
1105
-
1106
-Example command:
1107
-
1108
-```bash
1109
-vllm bench sweep serve_sla \
1110
- --serve-cmd 'vllm serve meta-llama/Llama-2-7b-chat-hf' \
1111
- --bench-cmd 'vllm bench serve --model meta-llama/Llama-2-7b-chat-hf --backend vllm --endpoint /v1/completions --dataset-name sharegpt --dataset-path benchmarks/ShareGPT_V3_unfiltered_cleaned_split.json' \
1112
- --serve-params benchmarks/serve_hparams.json \
1113
- --bench-params benchmarks/bench_hparams.json \
1114
- --sla-params benchmarks/sla_hparams.json \
1115
- --sla-variable max_concurrency \
1116
- -o benchmarks/results
1117
-```
1118
-
1119
-The algorithm for adjusting the SLA variable is as follows:
1120
-
1121
-1. Run the benchmark with infinite QPS, and use the corresponding metrics to determine the initial value of the variable.
1122
- - For example, the initial request rate is set to the concurrency under infinite QPS.
1123
-2. If the SLA is still satisfied, keep doubling the value until the SLA is no longer satisfied. This gives a relatively narrow window that contains the point where the SLA is barely satisfied.
1124
-3. Apply binary search over the window to find the maximum value that still satisfies the SLA.
1125
-
1126
-!!! important
1127
- SLA tuning is applied over each combination of `--serve-params`, `--bench-params`, and `--sla-params`.
1128
-
1129
- For a given combination of `--serve-params` and `--bench-params`, we share the benchmark results across `--sla-params` to avoid rerunning benchmarks with the same SLA variable value.
1130
-
1131
-### Visualizer
1132
-
1133
-[`vllm/benchmarks/sweep/plot.py`](../../vllm/benchmarks/sweep/plot.py) can be used to plot performance curves from parameter sweep results.
1134
-
1135
-Example command:
1136
-
1137
-```bash
1138
-vllm bench sweep plot benchmarks/results/<timestamp> \
1139
- --var-x max_concurrency \
1140
- --row-by random_input_len \
1141
- --col-by random_output_len \
1142
- --curve-by api_server_count,max_num_batched_tokens \
1143
- --filter-by 'max_concurrency<=1024'
1144
-```
1145
-
1146
-!!! tip
1147
- You can use `--dry-run` to preview the figures to be plotted.
1148
-
1149
-### Pareto visualization (tokens/s/user vs tokens/s/GPU)
1150
-
1151
-`vllm bench sweep plot_pareto` helps pick configurations that balance per-user and per-GPU throughput.
1152
-
1153
-Higher concurrency or batch size can raise GPU efficiency (per-GPU), but can add per user latency; lower concurrency improves per-user rate but underutilizes GPUs; The Pareto frontier shows the best achievable pairs across your runs.
1154
-
1155
-- x-axis: tokens/s/user = `output_throughput` ÷ concurrency (`--user-count-var`, default `max_concurrency`, fallback `max_concurrent_requests`).
1156
-- y-axis: tokens/s/GPU = `output_throughput` ÷ GPU count (`--gpu-count-var` if set; else gpu_count is TP×PP*DP).
1157
-- Output: a single figure at `OUTPUT_DIR/pareto/PARETO.png`.
1158
-- Show the configuration used in each data point `--label-by` (default: `max_concurrency,gpu_count`).
1159
-
1160
-Example:
1161
-
1162
-```bash
1163
-vllm bench sweep plot_pareto benchmarks/results/<timestamp> \
1164
- --label-by max_concurrency,tensor_parallel_size,pipeline_parallel_size
1165
-```
1166
-
1167
-## Performance Benchmarks
1168
-
1169
-The performance benchmarks are used for development to confirm whether new changes improve performance under various workloads. They are triggered on every commit with both the `perf-benchmarks` and `ready` labels, and when a PR is merged into vLLM.
1170
-
1171
-### Manually Trigger the benchmark
1172
-
1173
-Use [vllm-ci-test-repo images](https://gallery.ecr.aws/q9t5s3a7/vllm-ci-test-repo) with vLLM benchmark suite.
1174
-For CPU environment, please use the image with "-cpu" postfix.
1175
-
1176
-Here is an example for docker run command for CPU.
1177
-
1178
-```bash
1179
-docker run -it --entrypoint /bin/bash -v /data/huggingface:/root/.cache/huggingface -e HF_TOKEN='' --shm-size=16g --name vllm-cpu-ci public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:1da94e673c257373280026f75ceb4effac80e892-cpu
1180
-```
1181
-
1182
-Then, run below command inside the docker instance.
1183
-
1184
-```bash
1185
-bash .buildkite/performance-benchmarks/scripts/run-performance-benchmarks.sh
1186
-```
1187
-
1188
-When run, benchmark script generates results under **benchmark/results** folder, along with the benchmark_results.md and benchmark_results.json.
1189
-
1190
-#### Runtime environment variables
1191
-
1192
-- `ON_CPU`: set the value to '1' on Intel® Xeon® Processors. Default value is 0.
1193
-- `SERVING_JSON`: JSON file to use for the serving tests. Default value is empty string (use default file).
1194
-- `LATENCY_JSON`: JSON file to use for the latency tests. Default value is empty string (use default file).
1195
-- `THROUGHPUT_JSON`: JSON file to use for the throughout tests. Default value is empty string (use default file).
1196
-- `REMOTE_HOST`: IP for the remote vLLM service to benchmark. Default value is empty string.
1197
-- `REMOTE_PORT`: Port for the remote vLLM service to benchmark. Default value is empty string.
1198
-
1199
-For more results visualization, check the [visualizing the results](https://github.com/intel-ai-tce/vllm/blob/more_cpu_models/.buildkite/nightly-benchmarks/README.md#visualizing-the-results).
1200
-
1201
-The latest performance results are hosted on the public [vLLM Performance Dashboard](https://hud.pytorch.org/benchmark/llms?repoName=vllm-project%2Fvllm).
1202
-
1203
-More information on the performance benchmarks and their parameters can be found in [Benchmark README](https://github.com/intel-ai-tce/vllm/blob/more_cpu_models/.buildkite/nightly-benchmarks/README.md) and [performance benchmark description](../../.buildkite/performance-benchmarks/performance-benchmarks-descriptions.md).
1204
-
1205
-### Continuous Benchmarking
1206
-
1207
-The continuous benchmarking provides automated performance monitoring for vLLM across different models and GPU devices. This helps track vLLM's performance characteristics over time and identify any performance regressions or improvements.
1208
-
1209
-#### How It Works
1210
-
1211
-The continuous benchmarking is triggered via a [GitHub workflow CI](https://github.com/pytorch/pytorch-integration-testing/actions/workflows/vllm-benchmark.yml) in the PyTorch infrastructure repository, which runs automatically every 4 hours. The workflow executes three types of performance tests:
1212
-
1213
-- **Serving tests**: Measure request handling and API performance
1214
-- **Throughput tests**: Evaluate token generation rates
1215
-- **Latency tests**: Assess response time characteristics
1216
-
1217
-#### Benchmark Configuration
1218
-
1219
-The benchmarking currently runs on a predefined set of models configured in the [vllm-benchmarks directory](https://github.com/pytorch/pytorch-integration-testing/tree/main/vllm-benchmarks/benchmarks). To add new models for benchmarking:
1220
-
1221
-1. Navigate to the appropriate GPU directory in the benchmarks configuration
1222
-2. Add your model specifications to the corresponding configuration files
1223
-3. The new models will be included in the next scheduled benchmark run
1224
-
1225
-#### Viewing Results
1226
-
1227
-All continuous benchmarking results are automatically published to the public [vLLM Performance Dashboard](https://hud.pytorch.org/benchmark/llms?repoName=vllm-project%2Fvllm).
docs/benchmarking/dashboard.md
new
+58
@@ -0,0 +1,58 @@
1
+# Performance Dashboard
2
+
3
+The performance dashboard is used to confirm whether new changes improve/degrade performance under various workloads.
4
+It is updated by triggering benchmark runs on every commit with both the `perf-benchmarks` and `ready` labels, and when a PR is merged into vLLM.
5
+
6
+The results are automatically published to the public [vLLM Performance Dashboard](https://hud.pytorch.org/benchmark/llms?repoName=vllm-project%2Fvllm).
7
+
8
+## Manually Trigger the benchmark
9
+
10
+Use [vllm-ci-test-repo images](https://gallery.ecr.aws/q9t5s3a7/vllm-ci-test-repo) with vLLM benchmark suite.
11
+For CPU environment, please use the image with "-cpu" postfix.
12
+
13
+Here is an example for docker run command for CPU.
14
+
15
+```bash
16
+docker run -it --entrypoint /bin/bash -v /data/huggingface:/root/.cache/huggingface -e HF_TOKEN='' --shm-size=16g --name vllm-cpu-ci public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:1da94e673c257373280026f75ceb4effac80e892-cpu
17
+```
18
+
19
+Then, run below command inside the docker instance.
20
+
21
+```bash
22
+bash .buildkite/performance-benchmarks/scripts/run-performance-benchmarks.sh
23
+```
24
+
25
+When run, benchmark script generates results under **benchmark/results** folder, along with the benchmark_results.md and benchmark_results.json.
26
+
27
+### Runtime environment variables
28
+
29
+- `ON_CPU`: set the value to '1' on Intel® Xeon® Processors. Default value is 0.
30
+- `SERVING_JSON`: JSON file to use for the serving tests. Default value is empty string (use default file).
31
+- `LATENCY_JSON`: JSON file to use for the latency tests. Default value is empty string (use default file).
32
+- `THROUGHPUT_JSON`: JSON file to use for the throughout tests. Default value is empty string (use default file).
33
+- `REMOTE_HOST`: IP for the remote vLLM service to benchmark. Default value is empty string.
34
+- `REMOTE_PORT`: Port for the remote vLLM service to benchmark. Default value is empty string.
35
+
36
+For more results visualization, check the [visualizing the results](https://github.com/intel-ai-tce/vllm/blob/more_cpu_models/.buildkite/nightly-benchmarks/README.md#visualizing-the-results).
37
+
38
+More information on the performance benchmarks and their parameters can be found in [Benchmark README](https://github.com/intel-ai-tce/vllm/blob/more_cpu_models/.buildkite/nightly-benchmarks/README.md) and [performance benchmark description](../../.buildkite/performance-benchmarks/performance-benchmarks-descriptions.md).
39
+
40
+## Continuous Benchmarking
41
+
42
+The continuous benchmarking provides automated performance monitoring for vLLM across different models and GPU devices. This helps track vLLM's performance characteristics over time and identify any performance regressions or improvements.
43
+
44
+### How It Works
45
+
46
+The continuous benchmarking is triggered via a [GitHub workflow CI](https://github.com/pytorch/pytorch-integration-testing/actions/workflows/vllm-benchmark.yml) in the PyTorch infrastructure repository, which runs automatically every 4 hours. The workflow executes three types of performance tests:
47
+
48
+- **Serving tests**: Measure request handling and API performance
49
+- **Throughput tests**: Evaluate token generation rates
50
+- **Latency tests**: Assess response time characteristics
51
+
52
+### Benchmark Configuration
53
+
54
+The benchmarking currently runs on a predefined set of models configured in the [vllm-benchmarks directory](https://github.com/pytorch/pytorch-integration-testing/tree/main/vllm-benchmarks/benchmarks). To add new models for benchmarking:
55
+
56
+1. Navigate to the appropriate GPU directory in the benchmarks configuration
57
+2. Add your model specifications to the corresponding configuration files
58
+3. The new models will be included in the next scheduled benchmark run
docs/benchmarking/sweeps.md
new
+178
@@ -0,0 +1,178 @@
1
+# Parameter Sweeps
2
+
3
+## Online Benchmark
4
+
5
+### Basic
6
+
7
+`vllm bench sweep serve` automatically starts `vllm serve` and runs `vllm bench serve` to evaluate vLLM over multiple configurations.
8
+
9
+Follow these steps to run the script:
10
+
11
+1. Construct the base command to `vllm serve`, and pass it to the `--serve-cmd` option.
12
+2. Construct the base command to `vllm bench serve`, and pass it to the `--bench-cmd` option.
13
+3. (Optional) If you would like to vary the settings of `vllm serve`, create a new JSON file and populate it with the parameter combinations you want to test. Pass the file path to `--serve-params`.
14
+
15
+ - Example: Tuning `--max-num-seqs` and `--max-num-batched-tokens`:
16
+
17
+ ```json
18
+ [
19
+ {
20
+ "max_num_seqs": 32,
21
+ "max_num_batched_tokens": 1024
22
+ },
23
+ {
24
+ "max_num_seqs": 64,
25
+ "max_num_batched_tokens": 1024
26
+ },
27
+ {
28
+ "max_num_seqs": 64,
29
+ "max_num_batched_tokens": 2048
30
+ },
31
+ {
32
+ "max_num_seqs": 128,
33
+ "max_num_batched_tokens": 2048
34
+ },
35
+ {
36
+ "max_num_seqs": 128,
37
+ "max_num_batched_tokens": 4096
38
+ },
39
+ {
40
+ "max_num_seqs": 256,
41
+ "max_num_batched_tokens": 4096
42
+ }
43
+ ]
44
+ ```
45
+
46
+4. (Optional) If you would like to vary the settings of `vllm bench serve`, create a new JSON file and populate it with the parameter combinations you want to test. Pass the file path to `--bench-params`.
47
+
48
+ - Example: Using different input/output lengths for random dataset:
49
+
50
+ ```json
51
+ [
52
+ {
53
+ "random_input_len": 128,
54
+ "random_output_len": 32
55
+ },
56
+ {
57
+ "random_input_len": 256,
58
+ "random_output_len": 64
59
+ },
60
+ {
61
+ "random_input_len": 512,
62
+ "random_output_len": 128
63
+ }
64
+ ]
65
+ ```
66
+
67
+5. Determine where you want to save the results, and pass that to `--output-dir`.
68
+
69
+Example command:
70
+
71
+```bash
72
+vllm bench sweep serve \
73
+ --serve-cmd 'vllm serve meta-llama/Llama-2-7b-chat-hf' \
74
+ --bench-cmd 'vllm bench serve --model meta-llama/Llama-2-7b-chat-hf --backend vllm --endpoint /v1/completions --dataset-name sharegpt --dataset-path benchmarks/ShareGPT_V3_unfiltered_cleaned_split.json' \
75
+ --serve-params benchmarks/serve_hparams.json \
76
+ --bench-params benchmarks/bench_hparams.json \
77
+ -o benchmarks/results
78
+```
79
+
80
+!!! important
81
+ If both `--serve-params` and `--bench-params` are passed, the script will iterate over the Cartesian product between them.
82
+ You can use `--dry-run` to preview the commands to be run.
83
+
84
+ We only start the server once for each `--serve-params`, and keep it running for multiple `--bench-params`.
85
+ Between each benchmark run, we call the `/reset_prefix_cache` and `/reset_mm_cache` endpoints to get a clean slate for the next run.
86
+ In case you are using a custom `--serve-cmd`, you can override the commands used for resetting the state by setting `--after-bench-cmd`.
87
+
88
+!!! note
89
+ By default, each parameter combination is run 3 times to make the results more reliable. You can adjust the number of runs by setting `--num-runs`.
90
+
91
+!!! tip
92
+ You can use the `--resume` option to continue the parameter sweep if one of the runs failed.
93
+
94
+### SLA auto-tuner
95
+
96
+`vllm bench sweep serve_sla` is a wrapper over `vllm bench sweep serve` that tunes either the request rate or concurrency (choose using `--sla-variable`) in order to satisfy the SLA constraints given by `--sla-params`.
97
+
98
+For example, to ensure E2E latency within different target values for 99% of requests:
99
+
100
+```json
101
+[
102
+ {
103
+ "p99_e2el_ms": "<=200"
104
+ },
105
+ {
106
+ "p99_e2el_ms": "<=500"
107
+ },
108
+ {
109
+ "p99_e2el_ms": "<=1000"
110
+ },
111
+ {
112
+ "p99_e2el_ms": "<=2000"
113
+ }
114
+]
115
+```
116
+
117
+Example command:
118
+
119
+```bash
120
+vllm bench sweep serve_sla \
121
+ --serve-cmd 'vllm serve meta-llama/Llama-2-7b-chat-hf' \
122
+ --bench-cmd 'vllm bench serve --model meta-llama/Llama-2-7b-chat-hf --backend vllm --endpoint /v1/completions --dataset-name sharegpt --dataset-path benchmarks/ShareGPT_V3_unfiltered_cleaned_split.json' \
123
+ --serve-params benchmarks/serve_hparams.json \
124
+ --bench-params benchmarks/bench_hparams.json \
125
+ --sla-params benchmarks/sla_hparams.json \
126
+ --sla-variable max_concurrency \
127
+ -o benchmarks/results
128
+```
129
+
130
+The algorithm for adjusting the SLA variable is as follows:
131
+
132
+1. Run the benchmark with infinite QPS, and use the corresponding metrics to determine the initial value of the variable.
133
+ - For example, the initial request rate is set to the concurrency under infinite QPS.
134
+2. If the SLA is still satisfied, keep doubling the value until the SLA is no longer satisfied. This gives a relatively narrow window that contains the point where the SLA is barely satisfied.
135
+3. Apply binary search over the window to find the maximum value that still satisfies the SLA.
136
+
137
+!!! important
138
+ SLA tuning is applied over each combination of `--serve-params`, `--bench-params`, and `--sla-params`.
139
+
140
+ For a given combination of `--serve-params` and `--bench-params`, we share the benchmark results across `--sla-params` to avoid rerunning benchmarks with the same SLA variable value.
141
+
142
+## Visualization
143
+
144
+### Basic
145
+
146
+`vllm bench sweep plot` can be used to plot performance curves from parameter sweep results.
147
+
148
+Example command:
149
+
150
+```bash
151
+vllm bench sweep plot benchmarks/results/<timestamp> \
152
+ --var-x max_concurrency \
153
+ --row-by random_input_len \
154
+ --col-by random_output_len \
155
+ --curve-by api_server_count,max_num_batched_tokens \
156
+ --filter-by 'max_concurrency<=1024'
157
+```
158
+
159
+!!! tip
160
+ You can use `--dry-run` to preview the figures to be plotted.
161
+
162
+### Pareto chart
163
+
164
+`vllm bench sweep plot_pareto` helps pick configurations that balance per-user and per-GPU throughput.
165
+
166
+Higher concurrency or batch size can raise GPU efficiency (per-GPU), but can add per user latency; lower concurrency improves per-user rate but underutilizes GPUs; The Pareto frontier shows the best achievable pairs across your runs.
167
+
168
+- x-axis: tokens/s/user = `output_throughput` ÷ concurrency (`--user-count-var`, default `max_concurrency`, fallback `max_concurrent_requests`).
169
+- y-axis: tokens/s/GPU = `output_throughput` ÷ GPU count (`--gpu-count-var` if set; else gpu_count is TP×PP*DP).
170
+- Output: a single figure at `OUTPUT_DIR/pareto/PARETO.png`.
171
+- Show the configuration used in each data point `--label-by` (default: `max_concurrency,gpu_count`).
172
+
173
+Example:
174
+
175
+```bash
176
+vllm bench sweep plot_pareto benchmarks/results/<timestamp> \
177
+ --label-by max_concurrency,tensor_parallel_size,pipeline_parallel_size
178
+```