A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 04:45 UTC
[Misc] Use collapsible blocks for benchmark examples. (#20017)
Signed-off-by: reidliu41 <reid201711@gmail.com> Co-authored-by: reidliu41 <reid201711@gmail.com>
Reid committed
Jun 26, 2025 at 18:35 UTC
167aca45cbbfd8c56d700dfc9a6a5a3482a5bd74
1 file changed
+60
-34
benchmarks/README.md
+60
-34
@@ -4,7 +4,7 @@ This README guides you through running benchmark tests with the extensive
4
datasets supported on vLLM. It’s a living document, updated as new features and datasets
5
become available.
6
7
-## Dataset Overview
7
+**Dataset Overview**
8
9
<table style="width:100%; border-collapse: collapse;">
10
<thead>
@@ -82,7 +82,10 @@ become available.
82
**Note**: HuggingFace dataset's `dataset-name` should be set to `hf`
83
84
---
85
-## Example - Online Benchmark
85
+<details>
86
+<summary><b>🚀 Example - Online Benchmark</b></summary>
87
+
88
+<br/>
89
90
First start serving your model
91
@@ -130,7 +133,8 @@ P99 ITL (ms): 8.39
133
==================================================
134
```
135
133
-### Custom Dataset
136
+**Custom Dataset**
137
+
138
If the dataset you want to benchmark is not supported yet in vLLM, even then you can benchmark on it using `CustomDataset`. Your data needs to be in `.jsonl` format and needs to have "prompt" field per entry, e.g., data.jsonl
139
140
```
@@ -162,7 +166,7 @@ python3 benchmarks/benchmark_serving.py --port 9001 --save-result --save-detaile
166
167
You can skip applying chat template if your data already has it by using `--custom-skip-chat-template`.
168
165
-### VisionArena Benchmark for Vision Language Models
169
+**VisionArena Benchmark for Vision Language Models**
170
171
```bash
172
# need a model with vision capability here
@@ -180,7 +184,7 @@ python3 vllm/benchmarks/benchmark_serving.py \
184
--num-prompts 1000
185
```
186
183
-### InstructCoder Benchmark with Speculative Decoding
187
+**InstructCoder Benchmark with Speculative Decoding**
188
189
``` bash
190
VLLM_USE_V1=1 vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
@@ -197,7 +201,7 @@ python3 benchmarks/benchmark_serving.py \
201
--num-prompts 2048
202
```
203
200
-### Other HuggingFaceDataset Examples
204
+**Other HuggingFaceDataset Examples**
205
206
```bash
207
vllm serve Qwen/Qwen2-VL-7B-Instruct --disable-log-requests
@@ -251,7 +255,7 @@ python3 vllm/benchmarks/benchmark_serving.py \
255
--num-prompts 80
256
```
257
254
-### Running With Sampling Parameters
258
+**Running With Sampling Parameters**
259
260
When using OpenAI-compatible backends such as `vllm`, optional sampling
261
parameters can be specified. Example client command:
@@ -269,7 +273,7 @@ python3 vllm/benchmarks/benchmark_serving.py \
273
--num-prompts 10
274
```
275
272
-### Running With Ramp-Up Request Rate
276
+**Running With Ramp-Up Request Rate**
277
278
The benchmark tool also supports ramping up the request rate over the
279
duration of the benchmark run. This can be useful for stress testing the
@@ -284,8 +288,12 @@ The following arguments can be used to control the ramp-up:
288
- `--ramp-up-start-rps`: The request rate at the beginning of the benchmark.
289
- `--ramp-up-end-rps`: The request rate at the end of the benchmark.
290
287
----
288
-## Example - Offline Throughput Benchmark
291
+</details>
292
+
293
+<details>
294
+<summary><b>📈 Example - Offline Throughput Benchmark</b></summary>
295
+
296
+<br/>
297
298
```bash
299
python3 vllm/benchmarks/benchmark_throughput.py \
@@ -303,7 +311,7 @@ Total num prompt tokens: 5014
311
Total num output tokens: 1500
312
```
313
306
-### VisionArena Benchmark for Vision Language Models
314
+**VisionArena Benchmark for Vision Language Models**
315
316
``` bash
317
python3 vllm/benchmarks/benchmark_throughput.py \
@@ -323,7 +331,7 @@ Total num prompt tokens: 14527
331
Total num output tokens: 1280
332
```
333
326
-### InstructCoder Benchmark with Speculative Decoding
334
+**InstructCoder Benchmark with Speculative Decoding**
335
336
``` bash
337
VLLM_WORKER_MULTIPROC_METHOD=spawn \
@@ -347,7 +355,7 @@ Total num prompt tokens: 261136
355
Total num output tokens: 204800
356
```
357
350
-### Other HuggingFaceDataset Examples
358
+**Other HuggingFaceDataset Examples**
359
360
**`lmms-lab/LLaVA-OneVision-Data`**
361
@@ -386,7 +394,7 @@ python3 benchmarks/benchmark_throughput.py \
394
--num-prompts 10
395
```
396
389
-### Benchmark with LoRA Adapters
397
+**Benchmark with LoRA Adapters**
398
399
``` bash
400
# download dataset
@@ -403,18 +411,22 @@ python3 vllm/benchmarks/benchmark_throughput.py \
411
--lora-path yard1/llama-2-7b-sql-lora-test
412
```
413
406
----
407
-## Example - Structured Output Benchmark
414
+</details>
415
+
416
+<details>
417
+<summary><b>🛠️ Example - Structured Output Benchmark</b></summary>
418
+
419
+<br/>
420
421
Benchmark the performance of structured output generation (JSON, grammar, regex).
422
411
-### Server Setup
423
+**Server Setup**
424
425
```bash
426
vllm serve NousResearch/Hermes-3-Llama-3.1-8B --disable-log-requests
427
```
428
417
-### JSON Schema Benchmark
429
+**JSON Schema Benchmark**
430
431
```bash
432
python3 benchmarks/benchmark_serving_structured_output.py \
@@ -426,7 +438,7 @@ python3 benchmarks/benchmark_serving_structured_output.py \
438
--num-prompts 1000
439
```
440
429
-### Grammar-based Generation Benchmark
441
+**Grammar-based Generation Benchmark**
442
443
```bash
444
python3 benchmarks/benchmark_serving_structured_output.py \
@@ -438,7 +450,7 @@ python3 benchmarks/benchmark_serving_structured_output.py \
450
--num-prompts 1000
451
```
452
441
-### Regex-based Generation Benchmark
453
+**Regex-based Generation Benchmark**
454
455
```bash
456
python3 benchmarks/benchmark_serving_structured_output.py \
@@ -449,7 +461,7 @@ python3 benchmarks/benchmark_serving_structured_output.py \
461
--num-prompts 1000
462
```
463
452
-### Choice-based Generation Benchmark
464
+**Choice-based Generation Benchmark**
465
466
```bash
467
python3 benchmarks/benchmark_serving_structured_output.py \
@@ -460,7 +472,7 @@ python3 benchmarks/benchmark_serving_structured_output.py \
472
--num-prompts 1000
473
```
474
463
-### XGrammar Benchmark Dataset
475
+**XGrammar Benchmark Dataset**
476
477
```bash
478
python3 benchmarks/benchmark_serving_structured_output.py \
@@ -471,12 +483,16 @@ python3 benchmarks/benchmark_serving_structured_output.py \
483
--num-prompts 1000
484
```
485
474
----
475
-## Example - Long Document QA Throughput Benchmark
486
+</details>
487
+
488
+<details>
489
+<summary><b>📚 Example - Long Document QA Benchmark</b></summary>
490
+
491
+<br/>
492
493
Benchmark the performance of long document question-answering with prefix caching.
494
479
-### Basic Long Document QA Test
495
+**Basic Long Document QA Test**
496
497
```bash
498
python3 benchmarks/benchmark_long_document_qa_throughput.py \
@@ -488,7 +504,7 @@ python3 benchmarks/benchmark_long_document_qa_throughput.py \
504
--repeat-count 5
505
```
506
491
-### Different Repeat Modes
507
+**Different Repeat Modes**
508
509
```bash
510
# Random mode (default) - shuffle prompts randomly
@@ -519,12 +535,16 @@ python3 benchmarks/benchmark_long_document_qa_throughput.py \
535
--repeat-mode interleave
536
```
537
522
----
523
-## Example - Prefix Caching Benchmark
538
+</details>
539
+
540
+<details>
541
+<summary><b>🗂️ Example - Prefix Caching Benchmark</b></summary>
542
+
543
+<br/>
544
545
Benchmark the efficiency of automatic prefix caching.
546
527
-### Fixed Prompt with Prefix Caching
547
+**Fixed Prompt with Prefix Caching**
548
549
```bash
550
python3 benchmarks/benchmark_prefix_caching.py \
@@ -535,7 +555,7 @@ python3 benchmarks/benchmark_prefix_caching.py \
555
--input-length-range 128:256
556
```
557
538
-### ShareGPT Dataset with Prefix Caching
558
+**ShareGPT Dataset with Prefix Caching**
559
560
```bash
561
# download dataset
@@ -550,12 +570,16 @@ python3 benchmarks/benchmark_prefix_caching.py \
570
--input-length-range 128:256
571
```
572
553
----
554
-## Example - Request Prioritization Benchmark
573
+</details>
574
+
575
+<details>
576
+<summary><b>⚡ Example - Request Prioritization Benchmark</b></summary>
577
+
578
+<br/>
579
580
Benchmark the performance of request prioritization in vLLM.
581
558
-### Basic Prioritization Test
582
+**Basic Prioritization Test**
583
584
```bash
585
python3 benchmarks/benchmark_prioritization.py \
@@ -566,7 +590,7 @@ python3 benchmarks/benchmark_prioritization.py \
590
--scheduling-policy priority
591
```
592
569
-### Multiple Sequences per Prompt
593
+**Multiple Sequences per Prompt**
594
595
```bash
596
python3 benchmarks/benchmark_prioritization.py \
@@ -577,3 +601,5 @@ python3 benchmarks/benchmark_prioritization.py \
601
--scheduling-policy priority \
602
--n 2
603
```
604
+
605
+</details>