A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 05:30 UTC
[Docs] move benchmarks README to contributing guides (#24820)
Ye (Charlotte) Qi committed
Sep 16, 2025 at 05:52 UTC
85e0df139236d90acfe6f940ff1c96157e07dae0
2 files changed
+800
-874
benchmarks/README.md
+13
-867
@@ -1,874 +1,20 @@
1
-# Benchmarking vLLM
1
+# Benchmarks
2
3
-This README guides you through running benchmark tests with the extensive
4
-datasets supported on vLLM. It’s a living document, updated as new features and datasets
5
-become available.
3
+This directory used to contain vLLM's benchmark scripts and utilities for performance testing and evaluation.
4
7
-## Dataset Overview
5
+## Contents
6
9
-<table style="width:100%; border-collapse: collapse;">
10
- <thead>
11
- <tr>
12
- <th style="width:15%; text-align: left;">Dataset</th>
13
- <th style="width:10%; text-align: center;">Online</th>
14
- <th style="width:10%; text-align: center;">Offline</th>
15
- <th style="width:65%; text-align: left;">Data Path</th>
16
- </tr>
17
- </thead>
18
- <tbody>
19
- <tr>
20
- <td><strong>ShareGPT</strong></td>
21
- <td style="text-align: center;">✅</td>
22
- <td style="text-align: center;">✅</td>
23
- <td><code>wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json</code></td>
24
- </tr>
25
- <tr>
26
- <td><strong>ShareGPT4V (Image)</strong></td>
27
- <td style="text-align: center;">✅</td>
28
- <td style="text-align: center;">✅</td>
29
- <td>
30
- <code>wget https://huggingface.co/datasets/Lin-Chen/ShareGPT4V/blob/main/sharegpt4v_instruct_gpt4-vision_cap100k.json</code>
31
- <br>
32
- <div>Note that the images need to be downloaded separately. For example, to download COCO's 2017 Train images:</div>
33
- <code>wget http://images.cocodataset.org/zips/train2017.zip</code>
34
- </td>
35
- </tr>
36
- <tr>
37
- <td><strong>ShareGPT4Video (Video)</strong></td>
38
- <td style="text-align: center;">✅</td>
39
- <td style="text-align: center;">✅</td>
40
- <td>
41
- <code>git clone https://huggingface.co/datasets/ShareGPT4Video/ShareGPT4Video</code>
42
- </td>
43
- </tr>
44
- <tr>
45
- <td><strong>BurstGPT</strong></td>
46
- <td style="text-align: center;">✅</td>
47
- <td style="text-align: center;">✅</td>
48
- <td><code>wget https://github.com/HPMLL/BurstGPT/releases/download/v1.1/BurstGPT_without_fails_2.csv</code></td>
49
- </tr>
50
- <tr>
51
- <td><strong>Sonnet (deprecated)</strong></td>
52
- <td style="text-align: center;">✅</td>
53
- <td style="text-align: center;">✅</td>
54
- <td>Local file: <code>benchmarks/sonnet.txt</code></td>
55
- </tr>
56
- <tr>
57
- <td><strong>Random</strong></td>
58
- <td style="text-align: center;">✅</td>
59
- <td style="text-align: center;">✅</td>
60
- <td><code>synthetic</code></td>
61
- </tr>
62
- <tr>
63
- <td><strong>RandomMultiModal (Image/Video)</strong></td>
64
- <td style="text-align: center;">🟡</td>
65
- <td style="text-align: center;">🚧</td>
66
- <td><code>synthetic</code> </td>
67
- </tr>
68
- <tr>
69
- <td><strong>Prefix Repetition</strong></td>
70
- <td style="text-align: center;">✅</td>
71
- <td style="text-align: center;">✅</td>
72
- <td><code>synthetic</code></td>
73
- </tr>
74
- <tr>
75
- <td><strong>HuggingFace-VisionArena</strong></td>
76
- <td style="text-align: center;">✅</td>
77
- <td style="text-align: center;">✅</td>
78
- <td><code>lmarena-ai/VisionArena-Chat</code></td>
79
- </tr>
80
- <tr>
81
- <td><strong>HuggingFace-InstructCoder</strong></td>
82
- <td style="text-align: center;">✅</td>
83
- <td style="text-align: center;">✅</td>
84
- <td><code>likaixin/InstructCoder</code></td>
85
- </tr>
86
- <tr>
87
- <td><strong>HuggingFace-AIMO</strong></td>
88
- <td style="text-align: center;">✅</td>
89
- <td style="text-align: center;">✅</td>
90
- <td><code>AI-MO/aimo-validation-aime</code> , <code>AI-MO/NuminaMath-1.5</code>, <code>AI-MO/NuminaMath-CoT</code></td>
91
- </tr>
92
- <tr>
93
- <td><strong>HuggingFace-Other</strong></td>
94
- <td style="text-align: center;">✅</td>
95
- <td style="text-align: center;">✅</td>
96
- <td><code>lmms-lab/LLaVA-OneVision-Data</code>, <code>Aeala/ShareGPT_Vicuna_unfiltered</code></td>
97
- </tr>
98
- <tr>
99
- <td><strong>HuggingFace-MTBench</strong></td>
100
- <td style="text-align: center;">✅</td>
101
- <td style="text-align: center;">✅</td>
102
- <td><code>philschmid/mt-bench</code></td>
103
- </tr>
104
- <tr>
105
- <td><strong>HuggingFace-Blazedit</strong></td>
106
- <td style="text-align: center;">✅</td>
107
- <td style="text-align: center;">✅</td>
108
- <td><code>vdaita/edit_5k_char</code>, <code>vdaita/edit_10k_char</code></td>
109
- </tr>
110
- <tr>
111
- <td><strong>Spec Bench</strong></td>
112
- <td style="text-align: center;">✅</td>
113
- <td style="text-align: center;">✅</td>
114
- <td><code>wget https://raw.githubusercontent.com/hemingkx/Spec-Bench/refs/heads/main/data/spec_bench/question.jsonl</code></td>
115
- </tr>
116
- <tr>
117
- <td><strong>Custom</strong></td>
118
- <td style="text-align: center;">✅</td>
119
- <td style="text-align: center;">✅</td>
120
- <td>Local file: <code>data.jsonl</code></td>
121
- </tr>
122
- </tbody>
123
-</table>
7
+- **Serving benchmarks**: Scripts for testing online inference performance (latency, throughput)
8
+- **Throughput benchmarks**: Scripts for testing offline batch inference performance
9
+- **Specialized benchmarks**: Tools for testing specific features like structured output, prefix caching, long document QA, request prioritization, and multi-modal inference
10
+- **Dataset utilities**: Framework for loading and sampling from various benchmark datasets (ShareGPT, HuggingFace datasets, synthetic data, etc.)
11
125
-✅: supported
12
+## Usage
13
127
-🟡: Partial support
14
+For detailed usage instructions, examples, and dataset information, see the [Benchmark CLI documentation](https://docs.vllm.ai/en/latest/contributing/benchmarks.html#benchmark-cli).
15
129
-🚧: to be supported
16
+For full CLI reference see:
17
131
-**Note**: HuggingFace dataset's `dataset-name` should be set to `hf`.
132
-For local `dataset-path`, please set `hf-name` to its Hugging Face ID like
133
-
134
-```bash
135
---dataset-path /datasets/VisionArena-Chat/ --hf-name lmarena-ai/VisionArena-Chat
136
-```
137
-
138
-## 🚀 Example - Online Benchmark
139
-
140
-<details>
141
-<summary>Show more</summary>
142
-
143
-<br/>
144
-
145
-First start serving your model
146
-
147
-```bash
148
-vllm serve NousResearch/Hermes-3-Llama-3.1-8B
149
-```
150
-
151
-Then run the benchmarking script
152
-
153
-```bash
154
-# download dataset
155
-# wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
156
-vllm bench serve \
157
- --backend vllm \
158
- --model NousResearch/Hermes-3-Llama-3.1-8B \
159
- --endpoint /v1/completions \
160
- --dataset-name sharegpt \
161
- --dataset-path <your data path>/ShareGPT_V3_unfiltered_cleaned_split.json \
162
- --num-prompts 10
163
-```
164
-
165
-If successful, you will see the following output
166
-
167
-```text
168
-============ Serving Benchmark Result ============
169
-Successful requests: 10
170
-Benchmark duration (s): 5.78
171
-Total input tokens: 1369
172
-Total generated tokens: 2212
173
-Request throughput (req/s): 1.73
174
-Output token throughput (tok/s): 382.89
175
-Total Token throughput (tok/s): 619.85
176
----------------Time to First Token----------------
177
-Mean TTFT (ms): 71.54
178
-Median TTFT (ms): 73.88
179
-P99 TTFT (ms): 79.49
180
------Time per Output Token (excl. 1st token)------
181
-Mean TPOT (ms): 7.91
182
-Median TPOT (ms): 7.96
183
-P99 TPOT (ms): 8.03
184
----------------Inter-token Latency----------------
185
-Mean ITL (ms): 7.74
186
-Median ITL (ms): 7.70
187
-P99 ITL (ms): 8.39
188
-==================================================
189
-```
190
-
191
-### Custom Dataset
192
-
193
-If the dataset you want to benchmark is not supported yet in vLLM, even then you can benchmark on it using `CustomDataset`. Your data needs to be in `.jsonl` format and needs to have "prompt" field per entry, e.g., data.jsonl
194
-
195
-```json
196
-{"prompt": "What is the capital of India?"}
197
-{"prompt": "What is the capital of Iran?"}
198
-{"prompt": "What is the capital of China?"}
199
-```
200
-
201
-```bash
202
-# start server
203
-VLLM_USE_V1=1 vllm serve meta-llama/Llama-3.1-8B-Instruct
204
-```
205
-
206
-```bash
207
-# run benchmarking script
208
-vllm bench serve --port 9001 --save-result --save-detailed \
209
- --backend vllm \
210
- --model meta-llama/Llama-3.1-8B-Instruct \
211
- --endpoint /v1/completions \
212
- --dataset-name custom \
213
- --dataset-path <path-to-your-data-jsonl> \
214
- --custom-skip-chat-template \
215
- --num-prompts 80 \
216
- --max-concurrency 1 \
217
- --temperature=0.3 \
218
- --top-p=0.75 \
219
- --result-dir "./log/"
220
-```
221
-
222
-You can skip applying chat template if your data already has it by using `--custom-skip-chat-template`.
223
-
224
-### VisionArena Benchmark for Vision Language Models
225
-
226
-```bash
227
-# need a model with vision capability here
228
-vllm serve Qwen/Qwen2-VL-7B-Instruct
229
-```
230
-
231
-```bash
232
-vllm bench serve \
233
- --backend openai-chat \
234
- --endpoint-type openai-chat \
235
- --model Qwen/Qwen2-VL-7B-Instruct \
236
- --endpoint /v1/chat/completions \
237
- --dataset-name hf \
238
- --dataset-path lmarena-ai/VisionArena-Chat \
239
- --hf-split train \
240
- --num-prompts 1000
241
-```
242
-
243
-### InstructCoder Benchmark with Speculative Decoding
244
-
245
-``` bash
246
-VLLM_USE_V1=1 vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
247
- --speculative-config $'{"method": "ngram",
248
- "num_speculative_tokens": 5, "prompt_lookup_max": 5,
249
- "prompt_lookup_min": 2}'
250
-```
251
-
252
-``` bash
253
-vllm bench serve \
254
- --model meta-llama/Meta-Llama-3-8B-Instruct \
255
- --dataset-name hf \
256
- --dataset-path likaixin/InstructCoder \
257
- --num-prompts 2048
258
-```
259
-
260
-### Spec Bench Benchmark with Speculative Decoding
261
-
262
-``` bash
263
-VLLM_USE_V1=1 vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
264
- --speculative-config $'{"method": "ngram",
265
- "num_speculative_tokens": 5, "prompt_lookup_max": 5,
266
- "prompt_lookup_min": 2}'
267
-```
268
-
269
-[SpecBench dataset](https://github.com/hemingkx/Spec-Bench)
270
-
271
-Run all categories:
272
-
273
-``` bash
274
-# Download the dataset using:
275
-# wget https://raw.githubusercontent.com/hemingkx/Spec-Bench/refs/heads/main/data/spec_bench/question.jsonl
276
-
277
-vllm bench serve \
278
- --model meta-llama/Meta-Llama-3-8B-Instruct \
279
- --dataset-name spec_bench \
280
- --dataset-path "<YOUR_DOWNLOADED_PATH>/data/spec_bench/question.jsonl" \
281
- --num-prompts -1
282
-```
283
-
284
-Available categories include `[writing, roleplay, reasoning, math, coding, extraction, stem, humanities, translation, summarization, qa, math_reasoning, rag]`.
285
-
286
-Run only a specific category like "summarization":
287
-
288
-``` bash
289
-vllm bench serve \
290
- --model meta-llama/Meta-Llama-3-8B-Instruct \
291
- --dataset-name spec_bench \
292
- --dataset-path "<YOUR_DOWNLOADED_PATH>/data/spec_bench/question.jsonl" \
293
- --num-prompts -1
294
- --spec-bench-category "summarization"
295
-```
296
-
297
-### Other HuggingFaceDataset Examples
298
-
299
-```bash
300
-vllm serve Qwen/Qwen2-VL-7B-Instruct
301
-```
302
-
303
-`lmms-lab/LLaVA-OneVision-Data`:
304
-
305
-```bash
306
-vllm bench serve \
307
- --backend openai-chat \
308
- --endpoint-type openai-chat \
309
- --model Qwen/Qwen2-VL-7B-Instruct \
310
- --endpoint /v1/chat/completions \
311
- --dataset-name hf \
312
- --dataset-path lmms-lab/LLaVA-OneVision-Data \
313
- --hf-split train \
314
- --hf-subset "chart2text(cauldron)" \
315
- --num-prompts 10
316
-```
317
-
318
-`Aeala/ShareGPT_Vicuna_unfiltered`:
319
-
320
-```bash
321
-vllm bench serve \
322
- --backend openai-chat \
323
- --endpoint-type openai-chat \
324
- --model Qwen/Qwen2-VL-7B-Instruct \
325
- --endpoint /v1/chat/completions \
326
- --dataset-name hf \
327
- --dataset-path Aeala/ShareGPT_Vicuna_unfiltered \
328
- --hf-split train \
329
- --num-prompts 10
330
-```
331
-
332
-`AI-MO/aimo-validation-aime`:
333
-
334
-``` bash
335
-vllm bench serve \
336
- --model Qwen/QwQ-32B \
337
- --dataset-name hf \
338
- --dataset-path AI-MO/aimo-validation-aime \
339
- --num-prompts 10 \
340
- --seed 42
341
-```
342
-
343
-`philschmid/mt-bench`:
344
-
345
-``` bash
346
-vllm bench serve \
347
- --model Qwen/QwQ-32B \
348
- --dataset-name hf \
349
- --dataset-path philschmid/mt-bench \
350
- --num-prompts 80
351
-```
352
-
353
-`vdaita/edit_5k_char` or `vdaita/edit_10k_char`:
354
-
355
-``` bash
356
-vllm bench serve \
357
- --model Qwen/QwQ-32B \
358
- --dataset-name hf \
359
- --dataset-path vdaita/edit_5k_char \
360
- --num-prompts 90 \
361
- --blazedit-min-distance 0.01 \
362
- --blazedit-max-distance 0.99
363
-```
364
-
365
-### Running With Sampling Parameters
366
-
367
-When using OpenAI-compatible backends such as `vllm`, optional sampling
368
-parameters can be specified. Example client command:
369
-
370
-```bash
371
-vllm bench serve \
372
- --backend vllm \
373
- --model NousResearch/Hermes-3-Llama-3.1-8B \
374
- --endpoint /v1/completions \
375
- --dataset-name sharegpt \
376
- --dataset-path <your data path>/ShareGPT_V3_unfiltered_cleaned_split.json \
377
- --top-k 10 \
378
- --top-p 0.9 \
379
- --temperature 0.5 \
380
- --num-prompts 10
381
-```
382
-
383
-### Running With Ramp-Up Request Rate
384
-
385
-The benchmark tool also supports ramping up the request rate over the
386
-duration of the benchmark run. This can be useful for stress testing the
387
-server or finding the maximum throughput that it can handle, given some latency budget.
388
-
389
-Two ramp-up strategies are supported:
390
-
391
-- `linear`: Increases the request rate linearly from a start value to an end value.
392
-- `exponential`: Increases the request rate exponentially.
393
-
394
-The following arguments can be used to control the ramp-up:
395
-
396
-- `--ramp-up-strategy`: The ramp-up strategy to use (`linear` or `exponential`).
397
-- `--ramp-up-start-rps`: The request rate at the beginning of the benchmark.
398
-- `--ramp-up-end-rps`: The request rate at the end of the benchmark.
399
-
400
-</details>
401
-
402
-## 📈 Example - Offline Throughput Benchmark
403
-
404
-<details>
405
-<summary>Show more</summary>
406
-
407
-<br/>
408
-
409
-```bash
410
-vllm bench throughput \
411
- --model NousResearch/Hermes-3-Llama-3.1-8B \
412
- --dataset-name sonnet \
413
- --dataset-path vllm/benchmarks/sonnet.txt \
414
- --num-prompts 10
415
-```
416
-
417
-If successful, you will see the following output
418
-
419
-```text
420
-Throughput: 7.15 requests/s, 4656.00 total tokens/s, 1072.15 output tokens/s
421
-Total num prompt tokens: 5014
422
-Total num output tokens: 1500
423
-```
424
-
425
-### VisionArena Benchmark for Vision Language Models
426
-
427
-```bash
428
-vllm bench throughput \
429
- --model Qwen/Qwen2-VL-7B-Instruct \
430
- --backend vllm-chat \
431
- --dataset-name hf \
432
- --dataset-path lmarena-ai/VisionArena-Chat \
433
- --num-prompts 1000 \
434
- --hf-split train
435
-```
436
-
437
-The `num prompt tokens` now includes image token counts
438
-
439
-```text
440
-Throughput: 2.55 requests/s, 4036.92 total tokens/s, 326.90 output tokens/s
441
-Total num prompt tokens: 14527
442
-Total num output tokens: 1280
443
-```
444
-
445
-### InstructCoder Benchmark with Speculative Decoding
446
-
447
-``` bash
448
-VLLM_WORKER_MULTIPROC_METHOD=spawn \
449
-VLLM_USE_V1=1 \
450
-vllm bench throughput \
451
- --dataset-name=hf \
452
- --dataset-path=likaixin/InstructCoder \
453
- --model=meta-llama/Meta-Llama-3-8B-Instruct \
454
- --input-len=1000 \
455
- --output-len=100 \
456
- --num-prompts=2048 \
457
- --async-engine \
458
- --speculative-config $'{"method": "ngram",
459
- "num_speculative_tokens": 5, "prompt_lookup_max": 5,
460
- "prompt_lookup_min": 2}'
461
-```
462
-
463
-```text
464
-Throughput: 104.77 requests/s, 23836.22 total tokens/s, 10477.10 output tokens/s
465
-Total num prompt tokens: 261136
466
-Total num output tokens: 204800
467
-```
468
-
469
-### Other HuggingFaceDataset Examples
470
-
471
-`lmms-lab/LLaVA-OneVision-Data`:
472
-
473
-```bash
474
-vllm bench throughput \
475
- --model Qwen/Qwen2-VL-7B-Instruct \
476
- --backend vllm-chat \
477
- --dataset-name hf \
478
- --dataset-path lmms-lab/LLaVA-OneVision-Data \
479
- --hf-split train \
480
- --hf-subset "chart2text(cauldron)" \
481
- --num-prompts 10
482
-```
483
-
484
-`Aeala/ShareGPT_Vicuna_unfiltered`:
485
-
486
-```bash
487
-vllm bench throughput \
488
- --model Qwen/Qwen2-VL-7B-Instruct \
489
- --backend vllm-chat \
490
- --dataset-name hf \
491
- --dataset-path Aeala/ShareGPT_Vicuna_unfiltered \
492
- --hf-split train \
493
- --num-prompts 10
494
-```
495
-
496
-`AI-MO/aimo-validation-aime`:
497
-
498
-```bash
499
-vllm bench throughput \
500
- --model Qwen/QwQ-32B \
501
- --backend vllm \
502
- --dataset-name hf \
503
- --dataset-path AI-MO/aimo-validation-aime \
504
- --hf-split train \
505
- --num-prompts 10
506
-```
507
-
508
-Benchmark with LoRA adapters:
509
-
510
-``` bash
511
-# download dataset
512
-# wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
513
-vllm bench throughput \
514
- --model meta-llama/Llama-2-7b-hf \
515
- --backend vllm \
516
- --dataset_path <your data path>/ShareGPT_V3_unfiltered_cleaned_split.json \
517
- --dataset_name sharegpt \
518
- --num-prompts 10 \
519
- --max-loras 2 \
520
- --max-lora-rank 8 \
521
- --enable-lora \
522
- --lora-path yard1/llama-2-7b-sql-lora-test
523
- ```
524
-
525
-</details>
526
-
527
-## 🛠️ Example - Structured Output Benchmark
528
-
529
-<details>
530
-<summary>Show more</summary>
531
-
532
-<br/>
533
-
534
-Benchmark the performance of structured output generation (JSON, grammar, regex).
535
-
536
-### Server Setup
537
-
538
-```bash
539
-vllm serve NousResearch/Hermes-3-Llama-3.1-8B
540
-```
541
-
542
-### JSON Schema Benchmark
543
-
544
-```bash
545
-python3 benchmarks/benchmark_serving_structured_output.py \
546
- --backend vllm \
547
- --model NousResearch/Hermes-3-Llama-3.1-8B \
548
- --dataset json \
549
- --structured-output-ratio 1.0 \
550
- --request-rate 10 \
551
- --num-prompts 1000
552
-```
553
-
554
-### Grammar-based Generation Benchmark
555
-
556
-```bash
557
-python3 benchmarks/benchmark_serving_structured_output.py \
558
- --backend vllm \
559
- --model NousResearch/Hermes-3-Llama-3.1-8B \
560
- --dataset grammar \
561
- --structure-type grammar \
562
- --request-rate 10 \
563
- --num-prompts 1000
564
-```
565
-
566
-### Regex-based Generation Benchmark
567
-
568
-```bash
569
-python3 benchmarks/benchmark_serving_structured_output.py \
570
- --backend vllm \
571
- --model NousResearch/Hermes-3-Llama-3.1-8B \
572
- --dataset regex \
573
- --request-rate 10 \
574
- --num-prompts 1000
575
-```
576
-
577
-### Choice-based Generation Benchmark
578
-
579
-```bash
580
-python3 benchmarks/benchmark_serving_structured_output.py \
581
- --backend vllm \
582
- --model NousResearch/Hermes-3-Llama-3.1-8B \
583
- --dataset choice \
584
- --request-rate 10 \
585
- --num-prompts 1000
586
-```
587
-
588
-### XGrammar Benchmark Dataset
589
-
590
-```bash
591
-python3 benchmarks/benchmark_serving_structured_output.py \
592
- --backend vllm \
593
- --model NousResearch/Hermes-3-Llama-3.1-8B \
594
- --dataset xgrammar_bench \
595
- --request-rate 10 \
596
- --num-prompts 1000
597
-```
598
-
599
-</details>
600
-
601
-## 📚 Example - Long Document QA Benchmark
602
-
603
-<details>
604
-<summary>Show more</summary>
605
-
606
-<br/>
607
-
608
-Benchmark the performance of long document question-answering with prefix caching.
609
-
610
-### Basic Long Document QA Test
611
-
612
-```bash
613
-python3 benchmarks/benchmark_long_document_qa_throughput.py \
614
- --model meta-llama/Llama-2-7b-chat-hf \
615
- --enable-prefix-caching \
616
- --num-documents 16 \
617
- --document-length 2000 \
618
- --output-len 50 \
619
- --repeat-count 5
620
-```
621
-
622
-### Different Repeat Modes
623
-
624
-```bash
625
-# Random mode (default) - shuffle prompts randomly
626
-python3 benchmarks/benchmark_long_document_qa_throughput.py \
627
- --model meta-llama/Llama-2-7b-chat-hf \
628
- --enable-prefix-caching \
629
- --num-documents 8 \
630
- --document-length 3000 \
631
- --repeat-count 3 \
632
- --repeat-mode random
633
-
634
-# Tile mode - repeat entire prompt list in sequence
635
-python3 benchmarks/benchmark_long_document_qa_throughput.py \
636
- --model meta-llama/Llama-2-7b-chat-hf \
637
- --enable-prefix-caching \
638
- --num-documents 8 \
639
- --document-length 3000 \
640
- --repeat-count 3 \
641
- --repeat-mode tile
642
-
643
-# Interleave mode - repeat each prompt consecutively
644
-python3 benchmarks/benchmark_long_document_qa_throughput.py \
645
- --model meta-llama/Llama-2-7b-chat-hf \
646
- --enable-prefix-caching \
647
- --num-documents 8 \
648
- --document-length 3000 \
649
- --repeat-count 3 \
650
- --repeat-mode interleave
651
-```
652
-
653
-</details>
654
-
655
-## 🗂️ Example - Prefix Caching Benchmark
656
-
657
-<details>
658
-<summary>Show more</summary>
659
-
660
-<br/>
661
-
662
-Benchmark the efficiency of automatic prefix caching.
663
-
664
-### Fixed Prompt with Prefix Caching
665
-
666
-```bash
667
-python3 benchmarks/benchmark_prefix_caching.py \
668
- --model meta-llama/Llama-2-7b-chat-hf \
669
- --enable-prefix-caching \
670
- --num-prompts 1 \
671
- --repeat-count 100 \
672
- --input-length-range 128:256
673
-```
674
-
675
-### ShareGPT Dataset with Prefix Caching
676
-
677
-```bash
678
-# download dataset
679
-# wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
680
-
681
-python3 benchmarks/benchmark_prefix_caching.py \
682
- --model meta-llama/Llama-2-7b-chat-hf \
683
- --dataset-path /path/ShareGPT_V3_unfiltered_cleaned_split.json \
684
- --enable-prefix-caching \
685
- --num-prompts 20 \
686
- --repeat-count 5 \
687
- --input-length-range 128:256
688
-```
689
-
690
-### Prefix Repetition Dataset
691
-
692
-```bash
693
-vllm bench serve \
694
- --backend openai \
695
- --model meta-llama/Llama-2-7b-chat-hf \
696
- --dataset-name prefix_repetition \
697
- --num-prompts 100 \
698
- --prefix-repetition-prefix-len 512 \
699
- --prefix-repetition-suffix-len 128 \
700
- --prefix-repetition-num-prefixes 5 \
701
- --prefix-repetition-output-len 128
702
-```
703
-
704
-</details>
705
-
706
-## ⚡ Example - Request Prioritization Benchmark
707
-
708
-<details>
709
-<summary>Show more</summary>
710
-
711
-<br/>
712
-
713
-Benchmark the performance of request prioritization in vLLM.
714
-
715
-### Basic Prioritization Test
716
-
717
-```bash
718
-python3 benchmarks/benchmark_prioritization.py \
719
- --model meta-llama/Llama-2-7b-chat-hf \
720
- --input-len 128 \
721
- --output-len 64 \
722
- --num-prompts 100 \
723
- --scheduling-policy priority
724
-```
725
-
726
-### Multiple Sequences per Prompt
727
-
728
-```bash
729
-python3 benchmarks/benchmark_prioritization.py \
730
- --model meta-llama/Llama-2-7b-chat-hf \
731
- --input-len 128 \
732
- --output-len 64 \
733
- --num-prompts 100 \
734
- --scheduling-policy priority \
735
- --n 2
736
-```
737
-
738
-</details>
739
-
740
-## 👁️ Example - Multi-Modal Benchmark
741
-
742
-<details>
743
-<summary>Show more</summary>
744
-
745
-<br/>
746
-
747
-Benchmark the performance of multi-modal requests in vLLM.
748
-
749
-### Images (ShareGPT4V)
750
-
751
-Start vLLM:
752
-
753
-```bash
754
-python -m vllm.entrypoints.openai.api_server \
755
- --model Qwen/Qwen2.5-VL-7B-Instruct \
756
- --dtype bfloat16 \
757
- --limit-mm-per-prompt '{"image": 1}' \
758
- --allowed-local-media-path /path/to/sharegpt4v/images
759
-```
760
-
761
-Send requests with images:
762
-
763
-```bash
764
-vllm bench serve \
765
- --backend openai-chat \
766
- --model Qwen/Qwen2.5-VL-7B-Instruct \
767
- --dataset-name sharegpt \
768
- --dataset-path /path/to/ShareGPT4V/sharegpt4v_instruct_gpt4-vision_cap100k.json \
769
- --num-prompts 100 \
770
- --save-result \
771
- --result-dir ~/vllm_benchmark_results \
772
- --save-detailed \
773
- --endpoint /v1/chat/completion
774
-```
775
-
776
-### Videos (ShareGPT4Video)
777
-
778
-Start vLLM:
779
-
780
-```bash
781
-python -m vllm.entrypoints.openai.api_server \
782
- --model Qwen/Qwen2.5-VL-7B-Instruct \
783
- --dtype bfloat16 \
784
- --limit-mm-per-prompt '{"video": 1}' \
785
- --allowed-local-media-path /path/to/sharegpt4video/videos
786
-```
787
-
788
-Send requests with videos:
789
-
790
-```bash
791
-vllm bench serve \
792
- --backend openai-chat \
793
- --model Qwen/Qwen2.5-VL-7B-Instruct \
794
- --dataset-name sharegpt \
795
- --dataset-path /path/to/ShareGPT4Video/llava_v1_5_mix665k_with_video_chatgpt72k_share4video28k.json \
796
- --num-prompts 100 \
797
- --save-result \
798
- --result-dir ~/vllm_benchmark_results \
799
- --save-detailed \
800
- --endpoint /v1/chat/completion
801
-```
802
-
803
-### Synthetic Random Images (random-mm)
804
-
805
-Generate synthetic image inputs alongside random text prompts to stress-test vision models without external datasets.
806
-
807
-Notes:
808
-
809
-- Works only with online benchmark via the OpenAI backend (`--backend openai-chat`) and endpoint `/v1/chat/completions`.
810
-- Video sampling is not yet implemented.
811
-
812
-Start the server (example):
813
-
814
-```bash
815
-vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
816
- --dtype bfloat16 \
817
- --max-model-len 16384 \
818
- --limit-mm-per-prompt '{"image": 3, "video": 0}' \
819
- --mm-processor-kwargs max_pixels=1003520
820
-```
821
-
822
-Benchmark. It is recommended to use the flag `--ignore-eos` to simulate real responses. You can set the size of the output via the arg `random-output-len`.
823
-
824
-Ex.1: Fixed number of items and a single image resolution, enforcing generation of approx 40 tokens:
825
-
826
-```bash
827
-vllm bench serve \
828
- --backend openai-chat \
829
- --model Qwen/Qwen2.5-VL-3B-Instruct \
830
- --endpoint /v1/chat/completions \
831
- --dataset-name random-mm \
832
- --num-prompts 100 \
833
- --max-concurrency 10 \
834
- --random-prefix-len 25 \
835
- --random-input-len 300 \
836
- --random-output-len 40 \
837
- --random-range-ratio 0.2 \
838
- --random-mm-base-items-per-request 2 \
839
- --random-mm-limit-mm-per-prompt '{"image": 3, "video": 0}' \
840
- --random-mm-bucket-config '{(224, 224, 1): 1.0}' \
841
- --request-rate inf \
842
- --ignore-eos \
843
- --seed 42
844
-```
845
-
846
-The number of items per request can be controlled by passing multiple image buckets:
847
-
848
-```bash
849
- --random-mm-base-items-per-request 2 \
850
- --random-mm-num-mm-items-range-ratio 0.5 \
851
- --random-mm-limit-mm-per-prompt '{"image": 4, "video": 0}' \
852
- --random-mm-bucket-config '{(256, 256, 1): 0.7, (720, 1280, 1): 0.3}' \
853
-```
854
-
855
-Flags specific to `random-mm`:
856
-
857
-- `--random-mm-base-items-per-request`: base number of multimodal items per request.
858
-- `--random-mm-num-mm-items-range-ratio`: vary item count uniformly in the closed integer range [floor(n·(1−r)), ceil(n·(1+r))]. Set r=0 to keep it fixed; r=1 allows 0 items.
859
-- `--random-mm-limit-mm-per-prompt`: per-modality hard caps, e.g. '{"image": 3, "video": 0}'.
860
-- `--random-mm-bucket-config`: dict mapping (H, W, T) → probability. Entries with probability 0 are removed; remaining probabilities are renormalized to sum to 1. Use T=1 for images. Set any T>1 for videos (video sampling not yet supported).
861
-
862
-Behavioral notes:
863
-
864
-- If the requested base item count cannot be satisfied under the provided per-prompt limits, the tool raises an error rather than silently clamping.
865
-
866
-How sampling works:
867
-
868
-- Determine per-request item count k by sampling uniformly from the integer range defined by `--random-mm-base-items-per-request` and `--random-mm-num-mm-items-range-ratio`, then clamp k to at most the sum of per-modality limits.
869
-- For each of the k items, sample a bucket (H, W, T) according to the normalized probabilities in `--random-mm-bucket-config`, while tracking how many items of each modality have been added.
870
-- If a modality (e.g., image) reaches its limit from `--random-mm-limit-mm-per-prompt`, all buckets of that modality are excluded and the remaining bucket probabilities are renormalized before continuing.
871
-This should be seen as an edge case, and if this behavior can be avoided by setting `--random-mm-limit-mm-per-prompt` to a large number. Note that this might result in errors due to engine config `--limit-mm-per-prompt`.
872
-- The resulting request contains synthetic image data in `multi_modal_data` (OpenAI Chat format). When `random-mm` is used with the OpenAI Chat backend, prompts remain text and MM content is attached via `multi_modal_data`.
873
-
874
-</details>
18
+- <https://docs.vllm.ai/en/latest/cli/bench/latency.html>
19
+- <https://docs.vllm.ai/en/latest/cli/bench/serve.html>
20
+- <https://docs.vllm.ai/en/latest/cli/bench/throughput.html>
docs/contributing/benchmarks.md
+787
-7
@@ -1,9 +1,789 @@
1
+---
2
+toc_depth: 4
3
+---
4
+
5
# Benchmark Suites
6
3
-vLLM contains two sets of benchmarks:
7
+vLLM provides comprehensive benchmarking tools for performance testing and evaluation:
8
+
9
+- **[Benchmark CLI]**: `vllm bench` CLI tools and specialized benchmark scripts for interactive performance testing
10
+- **[Performance benchmarks][performance-benchmarks]**: Automated CI benchmarks for development
11
+- **[Nightly benchmarks][nightly-benchmarks]**: Comparative benchmarks against alternatives
12
+
13
+[Benchmark CLI]: #benchmark-cli
14
+
15
+## Benchmark CLI
16
+
17
+This section guides you through running benchmark tests with the extensive
18
+datasets supported on vLLM. It's a living document, updated as new features and datasets
19
+become available.
20
+
21
+### Dataset Overview
22
+
23
+<style>
24
+th {
25
+ min-width: 0 !important;
26
+}
27
+</style>
28
+
29
+| Dataset | Online | Offline | Data Path |
30
+|---------|--------|---------|-----------|
31
+| ShareGPT | ✅ | ✅ | `wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json` |
32
+| ShareGPT4V (Image) | ✅ | ✅ | `wget https://huggingface.co/datasets/Lin-Chen/ShareGPT4V/blob/main/sharegpt4v_instruct_gpt4-vision_cap100k.json`<br>Note that the images need to be downloaded separately. For example, to download COCO's 2017 Train images:<br>`wget http://images.cocodataset.org/zips/train2017.zip` |
33
+| ShareGPT4Video (Video) | ✅ | ✅ | `git clone https://huggingface.co/datasets/ShareGPT4Video/ShareGPT4Video` |
34
+| BurstGPT | ✅ | ✅ | `wget https://github.com/HPMLL/BurstGPT/releases/download/v1.1/BurstGPT_without_fails_2.csv` |
35
+| Sonnet (deprecated) | ✅ | ✅ | Local file: `benchmarks/sonnet.txt` |
36
+| Random | ✅ | ✅ | `synthetic` |
37
+| RandomMultiModal (Image/Video) | 🟡 | 🚧 | `synthetic` |
38
+| Prefix Repetition | ✅ | ✅ | `synthetic` |
39
+| HuggingFace-VisionArena | ✅ | ✅ | `lmarena-ai/VisionArena-Chat` |
40
+| HuggingFace-InstructCoder | ✅ | ✅ | `likaixin/InstructCoder` |
41
+| HuggingFace-AIMO | ✅ | ✅ | `AI-MO/aimo-validation-aime`, `AI-MO/NuminaMath-1.5`, `AI-MO/NuminaMath-CoT` |
42
+| HuggingFace-Other | ✅ | ✅ | `lmms-lab/LLaVA-OneVision-Data`, `Aeala/ShareGPT_Vicuna_unfiltered` |
43
+| HuggingFace-MTBench | ✅ | ✅ | `philschmid/mt-bench` |
44
+| HuggingFace-Blazedit | ✅ | ✅ | `vdaita/edit_5k_char`, `vdaita/edit_10k_char` |
45
+| Spec Bench | ✅ | ✅ | `wget https://raw.githubusercontent.com/hemingkx/Spec-Bench/refs/heads/main/data/spec_bench/question.jsonl` |
46
+| Custom | ✅ | ✅ | Local file: `data.jsonl` |
47
+
48
+Legend:
49
+
50
+- ✅ - supported
51
+- 🟡 - Partial support
52
+- 🚧 - to be supported
53
+
54
+!!! note
55
+ HuggingFace dataset's `dataset-name` should be set to `hf`.
56
+ For local `dataset-path`, please set `hf-name` to its Hugging Face ID like
57
+
58
+ ```bash
59
+ --dataset-path /datasets/VisionArena-Chat/ --hf-name lmarena-ai/VisionArena-Chat
60
+ ```
61
+
62
+### Examples
63
+
64
+#### 🚀 Online Benchmark
65
+
66
+<details class="admonition abstract" markdown="1">
67
+<summary>Show more</summary>
68
+
69
+First start serving your model
70
+
71
+```bash
72
+vllm serve NousResearch/Hermes-3-Llama-3.1-8B
73
+```
74
+
75
+Then run the benchmarking script
76
+
77
+```bash
78
+# download dataset
79
+# wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
80
+vllm bench serve \
81
+ --backend vllm \
82
+ --model NousResearch/Hermes-3-Llama-3.1-8B \
83
+ --endpoint /v1/completions \
84
+ --dataset-name sharegpt \
85
+ --dataset-path <your data path>/ShareGPT_V3_unfiltered_cleaned_split.json \
86
+ --num-prompts 10
87
+```
88
+
89
+If successful, you will see the following output
90
+
91
+```text
92
+============ Serving Benchmark Result ============
93
+Successful requests: 10
94
+Benchmark duration (s): 5.78
95
+Total input tokens: 1369
96
+Total generated tokens: 2212
97
+Request throughput (req/s): 1.73
98
+Output token throughput (tok/s): 382.89
99
+Total Token throughput (tok/s): 619.85
100
+---------------Time to First Token----------------
101
+Mean TTFT (ms): 71.54
102
+Median TTFT (ms): 73.88
103
+P99 TTFT (ms): 79.49
104
+-----Time per Output Token (excl. 1st token)------
105
+Mean TPOT (ms): 7.91
106
+Median TPOT (ms): 7.96
107
+P99 TPOT (ms): 8.03
108
+---------------Inter-token Latency----------------
109
+Mean ITL (ms): 7.74
110
+Median ITL (ms): 7.70
111
+P99 ITL (ms): 8.39
112
+==================================================
113
+```
114
+
115
+##### Custom Dataset
116
+
117
+If the dataset you want to benchmark is not supported yet in vLLM, even then you can benchmark on it using `CustomDataset`. Your data needs to be in `.jsonl` format and needs to have "prompt" field per entry, e.g., data.jsonl
118
+
119
+```json
120
+{"prompt": "What is the capital of India?"}
121
+{"prompt": "What is the capital of Iran?"}
122
+{"prompt": "What is the capital of China?"}
123
+```
124
+
125
+```bash
126
+# start server
127
+VLLM_USE_V1=1 vllm serve meta-llama/Llama-3.1-8B-Instruct
128
+```
129
+
130
+```bash
131
+# run benchmarking script
132
+vllm bench serve --port 9001 --save-result --save-detailed \
133
+ --backend vllm \
134
+ --model meta-llama/Llama-3.1-8B-Instruct \
135
+ --endpoint /v1/completions \
136
+ --dataset-name custom \
137
+ --dataset-path <path-to-your-data-jsonl> \
138
+ --custom-skip-chat-template \
139
+ --num-prompts 80 \
140
+ --max-concurrency 1 \
141
+ --temperature=0.3 \
142
+ --top-p=0.75 \
143
+ --result-dir "./log/"
144
+```
145
+
146
+You can skip applying chat template if your data already has it by using `--custom-skip-chat-template`.
147
+
148
+##### VisionArena Benchmark for Vision Language Models
149
+
150
+```bash
151
+# need a model with vision capability here
152
+vllm serve Qwen/Qwen2-VL-7B-Instruct
153
+```
154
+
155
+```bash
156
+vllm bench serve \
157
+ --backend openai-chat \
158
+ --endpoint-type openai-chat \
159
+ --model Qwen/Qwen2-VL-7B-Instruct \
160
+ --endpoint /v1/chat/completions \
161
+ --dataset-name hf \
162
+ --dataset-path lmarena-ai/VisionArena-Chat \
163
+ --hf-split train \
164
+ --num-prompts 1000
165
+```
166
+
167
+##### InstructCoder Benchmark with Speculative Decoding
168
+
169
+``` bash
170
+VLLM_USE_V1=1 vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
171
+ --speculative-config $'{"method": "ngram",
172
+ "num_speculative_tokens": 5, "prompt_lookup_max": 5,
173
+ "prompt_lookup_min": 2}'
174
+```
175
+
176
+``` bash
177
+vllm bench serve \
178
+ --model meta-llama/Meta-Llama-3-8B-Instruct \
179
+ --dataset-name hf \
180
+ --dataset-path likaixin/InstructCoder \
181
+ --num-prompts 2048
182
+```
183
+
184
+##### Spec Bench Benchmark with Speculative Decoding
185
+
186
+``` bash
187
+VLLM_USE_V1=1 vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
188
+ --speculative-config $'{"method": "ngram",
189
+ "num_speculative_tokens": 5, "prompt_lookup_max": 5,
190
+ "prompt_lookup_min": 2}'
191
+```
192
+
193
+[SpecBench dataset](https://github.com/hemingkx/Spec-Bench)
194
+
195
+Run all categories:
196
+
197
+``` bash
198
+# Download the dataset using:
199
+# wget https://raw.githubusercontent.com/hemingkx/Spec-Bench/refs/heads/main/data/spec_bench/question.jsonl
200
+
201
+vllm bench serve \
202
+ --model meta-llama/Meta-Llama-3-8B-Instruct \
203
+ --dataset-name spec_bench \
204
+ --dataset-path "<YOUR_DOWNLOADED_PATH>/data/spec_bench/question.jsonl" \
205
+ --num-prompts -1
206
+```
207
+
208
+Available categories include `[writing, roleplay, reasoning, math, coding, extraction, stem, humanities, translation, summarization, qa, math_reasoning, rag]`.
209
+
210
+Run only a specific category like "summarization":
211
+
212
+``` bash
213
+vllm bench serve \
214
+ --model meta-llama/Meta-Llama-3-8B-Instruct \
215
+ --dataset-name spec_bench \
216
+ --dataset-path "<YOUR_DOWNLOADED_PATH>/data/spec_bench/question.jsonl" \
217
+ --num-prompts -1
218
+ --spec-bench-category "summarization"
219
+```
220
+
221
+##### Other HuggingFaceDataset Examples
222
+
223
+```bash
224
+vllm serve Qwen/Qwen2-VL-7B-Instruct
225
+```
226
+
227
+`lmms-lab/LLaVA-OneVision-Data`:
228
+
229
+```bash
230
+vllm bench serve \
231
+ --backend openai-chat \
232
+ --endpoint-type openai-chat \
233
+ --model Qwen/Qwen2-VL-7B-Instruct \
234
+ --endpoint /v1/chat/completions \
235
+ --dataset-name hf \
236
+ --dataset-path lmms-lab/LLaVA-OneVision-Data \
237
+ --hf-split train \
238
+ --hf-subset "chart2text(cauldron)" \
239
+ --num-prompts 10
240
+```
241
+
242
+`Aeala/ShareGPT_Vicuna_unfiltered`:
243
+
244
+```bash
245
+vllm bench serve \
246
+ --backend openai-chat \
247
+ --endpoint-type openai-chat \
248
+ --model Qwen/Qwen2-VL-7B-Instruct \
249
+ --endpoint /v1/chat/completions \
250
+ --dataset-name hf \
251
+ --dataset-path Aeala/ShareGPT_Vicuna_unfiltered \
252
+ --hf-split train \
253
+ --num-prompts 10
254
+```
255
+
256
+`AI-MO/aimo-validation-aime`:
257
+
258
+``` bash
259
+vllm bench serve \
260
+ --model Qwen/QwQ-32B \
261
+ --dataset-name hf \
262
+ --dataset-path AI-MO/aimo-validation-aime \
263
+ --num-prompts 10 \
264
+ --seed 42
265
+```
266
+
267
+`philschmid/mt-bench`:
268
+
269
+``` bash
270
+vllm bench serve \
271
+ --model Qwen/QwQ-32B \
272
+ --dataset-name hf \
273
+ --dataset-path philschmid/mt-bench \
274
+ --num-prompts 80
275
+```
276
+
277
+`vdaita/edit_5k_char` or `vdaita/edit_10k_char`:
278
+
279
+``` bash
280
+vllm bench serve \
281
+ --model Qwen/QwQ-32B \
282
+ --dataset-name hf \
283
+ --dataset-path vdaita/edit_5k_char \
284
+ --num-prompts 90 \
285
+ --blazedit-min-distance 0.01 \
286
+ --blazedit-max-distance 0.99
287
+```
288
+
289
+##### Running With Sampling Parameters
290
+
291
+When using OpenAI-compatible backends such as `vllm`, optional sampling
292
+parameters can be specified. Example client command:
293
+
294
+```bash
295
+vllm bench serve \
296
+ --backend vllm \
297
+ --model NousResearch/Hermes-3-Llama-3.1-8B \
298
+ --endpoint /v1/completions \
299
+ --dataset-name sharegpt \
300
+ --dataset-path <your data path>/ShareGPT_V3_unfiltered_cleaned_split.json \
301
+ --top-k 10 \
302
+ --top-p 0.9 \
303
+ --temperature 0.5 \
304
+ --num-prompts 10
305
+```
306
+
307
+##### Running With Ramp-Up Request Rate
308
+
309
+The benchmark tool also supports ramping up the request rate over the
310
+duration of the benchmark run. This can be useful for stress testing the
311
+server or finding the maximum throughput that it can handle, given some latency budget.
312
+
313
+Two ramp-up strategies are supported:
314
+
315
+- `linear`: Increases the request rate linearly from a start value to an end value.
316
+- `exponential`: Increases the request rate exponentially.
317
+
318
+The following arguments can be used to control the ramp-up:
319
+
320
+- `--ramp-up-strategy`: The ramp-up strategy to use (`linear` or `exponential`).
321
+- `--ramp-up-start-rps`: The request rate at the beginning of the benchmark.
322
+- `--ramp-up-end-rps`: The request rate at the end of the benchmark.
323
+
324
+</details>
325
+
326
+#### 📈 Offline Throughput Benchmark
327
+
328
+<details class="admonition abstract" markdown="1">
329
+<summary>Show more</summary>
330
+
331
+```bash
332
+vllm bench throughput \
333
+ --model NousResearch/Hermes-3-Llama-3.1-8B \
334
+ --dataset-name sonnet \
335
+ --dataset-path vllm/benchmarks/sonnet.txt \
336
+ --num-prompts 10
337
+```
338
+
339
+If successful, you will see the following output
340
+
341
+```text
342
+Throughput: 7.15 requests/s, 4656.00 total tokens/s, 1072.15 output tokens/s
343
+Total num prompt tokens: 5014
344
+Total num output tokens: 1500
345
+```
346
+
347
+##### VisionArena Benchmark for Vision Language Models
348
+
349
+```bash
350
+vllm bench throughput \
351
+ --model Qwen/Qwen2-VL-7B-Instruct \
352
+ --backend vllm-chat \
353
+ --dataset-name hf \
354
+ --dataset-path lmarena-ai/VisionArena-Chat \
355
+ --num-prompts 1000 \
356
+ --hf-split train
357
+```
358
+
359
+The `num prompt tokens` now includes image token counts
360
+
361
+```text
362
+Throughput: 2.55 requests/s, 4036.92 total tokens/s, 326.90 output tokens/s
363
+Total num prompt tokens: 14527
364
+Total num output tokens: 1280
365
+```
366
+
367
+##### InstructCoder Benchmark with Speculative Decoding
368
+
369
+``` bash
370
+VLLM_WORKER_MULTIPROC_METHOD=spawn \
371
+VLLM_USE_V1=1 \
372
+vllm bench throughput \
373
+ --dataset-name=hf \
374
+ --dataset-path=likaixin/InstructCoder \
375
+ --model=meta-llama/Meta-Llama-3-8B-Instruct \
376
+ --input-len=1000 \
377
+ --output-len=100 \
378
+ --num-prompts=2048 \
379
+ --async-engine \
380
+ --speculative-config $'{"method": "ngram",
381
+ "num_speculative_tokens": 5, "prompt_lookup_max": 5,
382
+ "prompt_lookup_min": 2}'
383
+```
384
+
385
+```text
386
+Throughput: 104.77 requests/s, 23836.22 total tokens/s, 10477.10 output tokens/s
387
+Total num prompt tokens: 261136
388
+Total num output tokens: 204800
389
+```
390
+
391
+##### Other HuggingFaceDataset Examples
392
+
393
+`lmms-lab/LLaVA-OneVision-Data`:
394
+
395
+```bash
396
+vllm bench throughput \
397
+ --model Qwen/Qwen2-VL-7B-Instruct \
398
+ --backend vllm-chat \
399
+ --dataset-name hf \
400
+ --dataset-path lmms-lab/LLaVA-OneVision-Data \
401
+ --hf-split train \
402
+ --hf-subset "chart2text(cauldron)" \
403
+ --num-prompts 10
404
+```
405
+
406
+`Aeala/ShareGPT_Vicuna_unfiltered`:
407
+
408
+```bash
409
+vllm bench throughput \
410
+ --model Qwen/Qwen2-VL-7B-Instruct \
411
+ --backend vllm-chat \
412
+ --dataset-name hf \
413
+ --dataset-path Aeala/ShareGPT_Vicuna_unfiltered \
414
+ --hf-split train \
415
+ --num-prompts 10
416
+```
417
+
418
+`AI-MO/aimo-validation-aime`:
419
+
420
+```bash
421
+vllm bench throughput \
422
+ --model Qwen/QwQ-32B \
423
+ --backend vllm \
424
+ --dataset-name hf \
425
+ --dataset-path AI-MO/aimo-validation-aime \
426
+ --hf-split train \
427
+ --num-prompts 10
428
+```
429
+
430
+Benchmark with LoRA adapters:
431
+
432
+``` bash
433
+# download dataset
434
+# wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
435
+vllm bench throughput \
436
+ --model meta-llama/Llama-2-7b-hf \
437
+ --backend vllm \
438
+ --dataset_path <your data path>/ShareGPT_V3_unfiltered_cleaned_split.json \
439
+ --dataset_name sharegpt \
440
+ --num-prompts 10 \
441
+ --max-loras 2 \
442
+ --max-lora-rank 8 \
443
+ --enable-lora \
444
+ --lora-path yard1/llama-2-7b-sql-lora-test
445
+```
446
+
447
+</details>
448
+
449
+#### 🛠️ Structured Output Benchmark
450
+
451
+<details class="admonition abstract" markdown="1">
452
+<summary>Show more</summary>
453
+
454
+Benchmark the performance of structured output generation (JSON, grammar, regex).
455
+
456
+##### Server Setup
457
+
458
+```bash
459
+vllm serve NousResearch/Hermes-3-Llama-3.1-8B
460
+```
461
+
462
+##### JSON Schema Benchmark
463
+
464
+```bash
465
+python3 benchmarks/benchmark_serving_structured_output.py \
466
+ --backend vllm \
467
+ --model NousResearch/Hermes-3-Llama-3.1-8B \
468
+ --dataset json \
469
+ --structured-output-ratio 1.0 \
470
+ --request-rate 10 \
471
+ --num-prompts 1000
472
+```
473
+
474
+##### Grammar-based Generation Benchmark
475
+
476
+```bash
477
+python3 benchmarks/benchmark_serving_structured_output.py \
478
+ --backend vllm \
479
+ --model NousResearch/Hermes-3-Llama-3.1-8B \
480
+ --dataset grammar \
481
+ --structure-type grammar \
482
+ --request-rate 10 \
483
+ --num-prompts 1000
484
+```
485
+
486
+##### Regex-based Generation Benchmark
487
+
488
+```bash
489
+python3 benchmarks/benchmark_serving_structured_output.py \
490
+ --backend vllm \
491
+ --model NousResearch/Hermes-3-Llama-3.1-8B \
492
+ --dataset regex \
493
+ --request-rate 10 \
494
+ --num-prompts 1000
495
+```
496
+
497
+##### Choice-based Generation Benchmark
498
+
499
+```bash
500
+python3 benchmarks/benchmark_serving_structured_output.py \
501
+ --backend vllm \
502
+ --model NousResearch/Hermes-3-Llama-3.1-8B \
503
+ --dataset choice \
504
+ --request-rate 10 \
505
+ --num-prompts 1000
506
+```
507
+
508
+##### XGrammar Benchmark Dataset
509
+
510
+```bash
511
+python3 benchmarks/benchmark_serving_structured_output.py \
512
+ --backend vllm \
513
+ --model NousResearch/Hermes-3-Llama-3.1-8B \
514
+ --dataset xgrammar_bench \
515
+ --request-rate 10 \
516
+ --num-prompts 1000
517
+```
518
+
519
+</details>
520
+
521
+#### 📚 Long Document QA Benchmark
522
+
523
+<details class="admonition abstract" markdown="1">
524
+<summary>Show more</summary>
525
+
526
+Benchmark the performance of long document question-answering with prefix caching.
527
+
528
+##### Basic Long Document QA Test
529
+
530
+```bash
531
+python3 benchmarks/benchmark_long_document_qa_throughput.py \
532
+ --model meta-llama/Llama-2-7b-chat-hf \
533
+ --enable-prefix-caching \
534
+ --num-documents 16 \
535
+ --document-length 2000 \
536
+ --output-len 50 \
537
+ --repeat-count 5
538
+```
539
+
540
+##### Different Repeat Modes
541
+
542
+```bash
543
+# Random mode (default) - shuffle prompts randomly
544
+python3 benchmarks/benchmark_long_document_qa_throughput.py \
545
+ --model meta-llama/Llama-2-7b-chat-hf \
546
+ --enable-prefix-caching \
547
+ --num-documents 8 \
548
+ --document-length 3000 \
549
+ --repeat-count 3 \
550
+ --repeat-mode random
551
+
552
+# Tile mode - repeat entire prompt list in sequence
553
+python3 benchmarks/benchmark_long_document_qa_throughput.py \
554
+ --model meta-llama/Llama-2-7b-chat-hf \
555
+ --enable-prefix-caching \
556
+ --num-documents 8 \
557
+ --document-length 3000 \
558
+ --repeat-count 3 \
559
+ --repeat-mode tile
560
+
561
+# Interleave mode - repeat each prompt consecutively
562
+python3 benchmarks/benchmark_long_document_qa_throughput.py \
563
+ --model meta-llama/Llama-2-7b-chat-hf \
564
+ --enable-prefix-caching \
565
+ --num-documents 8 \
566
+ --document-length 3000 \
567
+ --repeat-count 3 \
568
+ --repeat-mode interleave
569
+```
570
+
571
+</details>
572
+
573
+#### 🗂️ Prefix Caching Benchmark
574
+
575
+<details class="admonition abstract" markdown="1">
576
+<summary>Show more</summary>
577
+
578
+Benchmark the efficiency of automatic prefix caching.
579
+
580
+##### Fixed Prompt with Prefix Caching
581
+
582
+```bash
583
+python3 benchmarks/benchmark_prefix_caching.py \
584
+ --model meta-llama/Llama-2-7b-chat-hf \
585
+ --enable-prefix-caching \
586
+ --num-prompts 1 \
587
+ --repeat-count 100 \
588
+ --input-length-range 128:256
589
+```
590
+
591
+##### ShareGPT Dataset with Prefix Caching
592
+
593
+```bash
594
+# download dataset
595
+# wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
596
+
597
+python3 benchmarks/benchmark_prefix_caching.py \
598
+ --model meta-llama/Llama-2-7b-chat-hf \
599
+ --dataset-path /path/ShareGPT_V3_unfiltered_cleaned_split.json \
600
+ --enable-prefix-caching \
601
+ --num-prompts 20 \
602
+ --repeat-count 5 \
603
+ --input-length-range 128:256
604
+```
605
+
606
+##### Prefix Repetition Dataset
607
+
608
+```bash
609
+vllm bench serve \
610
+ --backend openai \
611
+ --model meta-llama/Llama-2-7b-chat-hf \
612
+ --dataset-name prefix_repetition \
613
+ --num-prompts 100 \
614
+ --prefix-repetition-prefix-len 512 \
615
+ --prefix-repetition-suffix-len 128 \
616
+ --prefix-repetition-num-prefixes 5 \
617
+ --prefix-repetition-output-len 128
618
+```
619
+
620
+</details>
621
+
622
+#### ⚡ Request Prioritization Benchmark
623
+
624
+<details class="admonition abstract" markdown="1">
625
+<summary>Show more</summary>
626
+
627
+Benchmark the performance of request prioritization in vLLM.
628
+
629
+##### Basic Prioritization Test
630
+
631
+```bash
632
+python3 benchmarks/benchmark_prioritization.py \
633
+ --model meta-llama/Llama-2-7b-chat-hf \
634
+ --input-len 128 \
635
+ --output-len 64 \
636
+ --num-prompts 100 \
637
+ --scheduling-policy priority
638
+```
639
+
640
+##### Multiple Sequences per Prompt
641
+
642
+```bash
643
+python3 benchmarks/benchmark_prioritization.py \
644
+ --model meta-llama/Llama-2-7b-chat-hf \
645
+ --input-len 128 \
646
+ --output-len 64 \
647
+ --num-prompts 100 \
648
+ --scheduling-policy priority \
649
+ --n 2
650
+```
651
+
652
+</details>
653
+
654
+#### 👁️ Multi-Modal Benchmark
655
+
656
+<details class="admonition abstract" markdown="1">
657
+<summary>Show more</summary>
658
+
659
+Benchmark the performance of multi-modal requests in vLLM.
660
+
661
+##### Images (ShareGPT4V)
662
+
663
+Start vLLM:
664
+
665
+```bash
666
+python -m vllm.entrypoints.openai.api_server \
667
+ --model Qwen/Qwen2.5-VL-7B-Instruct \
668
+ --dtype bfloat16 \
669
+ --limit-mm-per-prompt '{"image": 1}' \
670
+ --allowed-local-media-path /path/to/sharegpt4v/images
671
+```
672
+
673
+Send requests with images:
674
+
675
+```bash
676
+vllm bench serve \
677
+ --backend openai-chat \
678
+ --model Qwen/Qwen2.5-VL-7B-Instruct \
679
+ --dataset-name sharegpt \
680
+ --dataset-path /path/to/ShareGPT4V/sharegpt4v_instruct_gpt4-vision_cap100k.json \
681
+ --num-prompts 100 \
682
+ --save-result \
683
+ --result-dir ~/vllm_benchmark_results \
684
+ --save-detailed \
685
+ --endpoint /v1/chat/completion
686
+```
687
+
688
+##### Videos (ShareGPT4Video)
689
+
690
+Start vLLM:
691
+
692
+```bash
693
+python -m vllm.entrypoints.openai.api_server \
694
+ --model Qwen/Qwen2.5-VL-7B-Instruct \
695
+ --dtype bfloat16 \
696
+ --limit-mm-per-prompt '{"video": 1}' \
697
+ --allowed-local-media-path /path/to/sharegpt4video/videos
698
+```
699
+
700
+Send requests with videos:
701
+
702
+```bash
703
+vllm bench serve \
704
+ --backend openai-chat \
705
+ --model Qwen/Qwen2.5-VL-7B-Instruct \
706
+ --dataset-name sharegpt \
707
+ --dataset-path /path/to/ShareGPT4Video/llava_v1_5_mix665k_with_video_chatgpt72k_share4video28k.json \
708
+ --num-prompts 100 \
709
+ --save-result \
710
+ --result-dir ~/vllm_benchmark_results \
711
+ --save-detailed \
712
+ --endpoint /v1/chat/completion
713
+```
714
+
715
+##### Synthetic Random Images (random-mm)
716
+
717
+Generate synthetic image inputs alongside random text prompts to stress-test vision models without external datasets.
718
+
719
+Notes:
720
+
721
+- Works only with online benchmark via the OpenAI backend (`--backend openai-chat`) and endpoint `/v1/chat/completions`.
722
+- Video sampling is not yet implemented.
723
+
724
+Start the server (example):
725
+
726
+```bash
727
+vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
728
+ --dtype bfloat16 \
729
+ --max-model-len 16384 \
730
+ --limit-mm-per-prompt '{"image": 3, "video": 0}' \
731
+ --mm-processor-kwargs max_pixels=1003520
732
+```
733
+
734
+Benchmark. It is recommended to use the flag `--ignore-eos` to simulate real responses. You can set the size of the output via the arg `random-output-len`.
735
+
736
+Ex.1: Fixed number of items and a single image resolution, enforcing generation of approx 40 tokens:
737
+
738
+```bash
739
+vllm bench serve \
740
+ --backend openai-chat \
741
+ --model Qwen/Qwen2.5-VL-3B-Instruct \
742
+ --endpoint /v1/chat/completions \
743
+ --dataset-name random-mm \
744
+ --num-prompts 100 \
745
+ --max-concurrency 10 \
746
+ --random-prefix-len 25 \
747
+ --random-input-len 300 \
748
+ --random-output-len 40 \
749
+ --random-range-ratio 0.2 \
750
+ --random-mm-base-items-per-request 2 \
751
+ --random-mm-limit-mm-per-prompt '{"image": 3, "video": 0}' \
752
+ --random-mm-bucket-config '{(224, 224, 1): 1.0}' \
753
+ --request-rate inf \
754
+ --ignore-eos \
755
+ --seed 42
756
+```
757
+
758
+The number of items per request can be controlled by passing multiple image buckets:
759
+
760
+```bash
761
+ --random-mm-base-items-per-request 2 \
762
+ --random-mm-num-mm-items-range-ratio 0.5 \
763
+ --random-mm-limit-mm-per-prompt '{"image": 4, "video": 0}' \
764
+ --random-mm-bucket-config '{(256, 256, 1): 0.7, (720, 1280, 1): 0.3}' \
765
+```
766
+
767
+Flags specific to `random-mm`:
768
+
769
+- `--random-mm-base-items-per-request`: base number of multimodal items per request.
770
+- `--random-mm-num-mm-items-range-ratio`: vary item count uniformly in the closed integer range [floor(n·(1−r)), ceil(n·(1+r))]. Set r=0 to keep it fixed; r=1 allows 0 items.
771
+- `--random-mm-limit-mm-per-prompt`: per-modality hard caps, e.g. '{"image": 3, "video": 0}'.
772
+- `--random-mm-bucket-config`: dict mapping (H, W, T) → probability. Entries with probability 0 are removed; remaining probabilities are renormalized to sum to 1. Use T=1 for images. Set any T>1 for videos (video sampling not yet supported).
773
+
774
+Behavioral notes:
775
+
776
+- If the requested base item count cannot be satisfied under the provided per-prompt limits, the tool raises an error rather than silently clamping.
777
+
778
+How sampling works:
779
+
780
+- Determine per-request item count k by sampling uniformly from the integer range defined by `--random-mm-base-items-per-request` and `--random-mm-num-mm-items-range-ratio`, then clamp k to at most the sum of per-modality limits.
781
+- For each of the k items, sample a bucket (H, W, T) according to the normalized probabilities in `--random-mm-bucket-config`, while tracking how many items of each modality have been added.
782
+- If a modality (e.g., image) reaches its limit from `--random-mm-limit-mm-per-prompt`, all buckets of that modality are excluded and the remaining bucket probabilities are renormalized before continuing.
783
+This should be seen as an edge case, and if this behavior can be avoided by setting `--random-mm-limit-mm-per-prompt` to a large number. Note that this might result in errors due to engine config `--limit-mm-per-prompt`.
784
+- The resulting request contains synthetic image data in `multi_modal_data` (OpenAI Chat format). When `random-mm` is used with the OpenAI Chat backend, prompts remain text and MM content is attached via `multi_modal_data`.
785
5
-- [Performance benchmarks][performance-benchmarks]
6
-- [Nightly benchmarks][nightly-benchmarks]
786
+</details>
787
788
[](){ #performance-benchmarks }
789
@@ -13,22 +793,22 @@ The performance benchmarks are used for development to confirm whether new chang
793
794
### Manually Trigger the benchmark
795
16
-Use [vllm-ci-test-repo images](https://gallery.ecr.aws/q9t5s3a7/vllm-ci-test-repo) with vLLM benchmark suite.
796
+Use [vllm-ci-test-repo images](https://gallery.ecr.aws/q9t5s3a7/vllm-ci-test-repo) with vLLM benchmark suite.
797
For CPU environment, please use the image with "-cpu" postfix.
798
19
-Here is an example for docker run command for CPU.
799
+Here is an example for docker run command for CPU.
800
801
```bash
802
docker run -it --entrypoint /bin/bash -v /data/huggingface:/root/.cache/huggingface -e HF_TOKEN='' --shm-size=16g --name vllm-cpu-ci public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:1da94e673c257373280026f75ceb4effac80e892-cpu
803
```
804
25
-Then, run below command inside the docker instance.
805
+Then, run below command inside the docker instance.
806
807
```bash
808
bash .buildkite/nightly-benchmarks/scripts/run-performance-benchmarks.sh
809
```
810
31
-When run, benchmark script generates results under **benchmark/results** folder, along with the benchmark_results.md and benchmark_results.json.
811
+When run, benchmark script generates results under **benchmark/results** folder, along with the benchmark_results.md and benchmark_results.json.
812
813
#### Runtime environment variables
814