A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 7, 05:15 UTC
[Docs] Update docs to include mm processor + encoder benchmarks (#34083)
Signed-off-by: Reagan <reaganjlee@gmail.com>
Reagan Lee committed
Mar 5, 2026 at 01:38 UTC
ac773bbe8095b4493c258abbf35c2a2d10d2faab
2 files changed
+135
-3
docs/benchmarking/cli.md
+89
-3
@@ -25,7 +25,7 @@ th {
25
| BurstGPT | ✅ | ✅ | `wget https://github.com/HPMLL/BurstGPT/releases/download/v1.1/BurstGPT_without_fails_2.csv` |
26
| Sonnet (deprecated) | ✅ | ✅ | Local file: `benchmarks/sonnet.txt` |
27
| Random | ✅ | ✅ | `synthetic` |
28
-| RandomMultiModal (Image/Video) | 🟡 | 🚧 | `synthetic` |
28
+| RandomMultiModal (Image/Video) | ✅ | ✅ | `synthetic` |
29
| RandomForReranking | ✅ | ✅ | `synthetic` |
30
| Prefix Repetition | ✅ | ✅ | `synthetic` |
31
| HuggingFace-VisionArena | ✅ | ✅ | `lmarena-ai/VisionArena-Chat` |
@@ -545,6 +545,24 @@ vllm bench throughput \
545
--lora-path yard1/llama-2-7b-sql-lora-test
546
```
547
548
+#### Synthetic Random Multimodal (random-mm)
549
+
550
+Generate synthetic multimodal inputs for offline throughput testing without external datasets.
551
+Use `--backend vllm-chat` so that image tokens are counted correctly.
552
+
553
+```bash
554
+vllm bench throughput \
555
+ --model Qwen/Qwen2-VL-7B-Instruct \
556
+ --backend vllm-chat \
557
+ --dataset-name random-mm \
558
+ --num-prompts 100 \
559
+ --random-input-len 300 \
560
+ --random-output-len 40 \
561
+ --random-mm-base-items-per-request 2 \
562
+ --random-mm-limit-mm-per-prompt '{"image": 3, "video": 0}' \
563
+ --random-mm-bucket-config '{(256, 256, 1): 0.7, (720, 1280, 1): 0.3}'
564
+```
565
+
566
</details>
567
568
### 🛠️ Structured Output Benchmark
@@ -846,8 +864,8 @@ Generate synthetic image inputs alongside random text prompts to stress-test vis
864
865
Notes:
866
849
-- Works only with online benchmark via the OpenAI backend (`--backend openai-chat`) and endpoint `/v1/chat/completions`.
850
-- Video sampling is not yet implemented.
867
+- For online benchmarks, use `--backend openai-chat` with endpoint `/v1/chat/completions`.
868
+- For offline benchmarks, use `--backend vllm-chat` (see [Offline Throughput Benchmark](#-offline-throughput-benchmark) for an example).
869
870
Start the server (example):
871
@@ -913,6 +931,74 @@ This should be seen as an edge case, and if this behavior can be avoided by sett
931
932
</details>
933
934
+### 🔬 Multimodal Processor Benchmark
935
+
936
+Benchmark per-stage latency of the multimodal (MM) input processor pipeline, including the encoder forward pass. This is useful for profiling preprocessing bottlenecks in vision-language models.
937
+
938
+<details class="admonition abstract" markdown="1">
939
+<summary>Show more</summary>
940
+
941
+The benchmark measures the following stages for each request:
942
+
943
+| Stage | Description |
944
+|-------|-------------|
945
+| `get_mm_hashes_secs` | Time spent hashing multimodal inputs |
946
+| `get_cache_missing_items_secs` | Time spent looking up the processor cache |
947
+| `apply_hf_processor_secs` | Time spent in the HuggingFace processor |
948
+| `merge_mm_kwargs_secs` | Time spent merging multimodal kwargs |
949
+| `apply_prompt_updates_secs` | Time spent updating prompt tokens |
950
+| `preprocessor_total_secs` | Total preprocessing time |
951
+| `encoder_forward_secs` | Time spent in the encoder model forward pass |
952
+| `num_encoder_calls` | Number of encoder invocations per request |
953
+
954
+The benchmark also reports end-to-end latency (TTFT + decode time) per
955
+request. Use `--metric-percentiles` to select which percentiles to report
956
+(default: p99) and `--output-json` to save results.
957
+
958
+#### Basic Example with Synthetic Data (random-mm)
959
+
960
+```bash
961
+vllm bench mm-processor \
962
+ --model Qwen/Qwen2-VL-7B-Instruct \
963
+ --dataset-name random-mm \
964
+ --num-prompts 50 \
965
+ --random-input-len 300 \
966
+ --random-output-len 40 \
967
+ --random-mm-base-items-per-request 2 \
968
+ --random-mm-limit-mm-per-prompt '{"image": 3, "video": 0}' \
969
+ --random-mm-bucket-config '{(256, 256, 1): 0.7, (720, 1280, 1): 0.3}'
970
+```
971
+
972
+#### Using a HuggingFace Dataset
973
+
974
+```bash
975
+vllm bench mm-processor \
976
+ --model Qwen/Qwen2-VL-7B-Instruct \
977
+ --dataset-name hf \
978
+ --dataset-path lmarena-ai/VisionArena-Chat \
979
+ --hf-split train \
980
+ --num-prompts 100
981
+```
982
+
983
+#### Warmup, Custom Percentiles, and JSON Output
984
+
985
+```bash
986
+vllm bench mm-processor \
987
+ --model Qwen/Qwen2-VL-7B-Instruct \
988
+ --dataset-name random-mm \
989
+ --num-prompts 200 \
990
+ --num-warmups 5 \
991
+ --random-input-len 300 \
992
+ --random-output-len 40 \
993
+ --random-mm-base-items-per-request 1 \
994
+ --metric-percentiles 50,90,95,99 \
995
+ --output-json results.json
996
+```
997
+
998
+See [`vllm bench mm-processor`](../cli/bench/mm_processor.md) for the full argument reference.
999
+
1000
+</details>
1001
+
1002
### Embedding Benchmark
1003
1004
Benchmark the performance of embedding requests in vLLM.
docs/cli/bench/mm_processor.md
+46
@@ -1,5 +1,51 @@
1
# vllm bench mm-processor
2
3
+## Overview
4
+
5
+`vllm bench mm-processor` profiles the multimodal input processor pipeline of
6
+vision-language models. It measures per-stage latency from the HuggingFace
7
+processor through to the encoder forward pass, helping you identify
8
+preprocessing bottlenecks and understand how different image resolutions or
9
+item counts affect end-to-end request time.
10
+
11
+The benchmark supports two data sources: synthetic random multimodal inputs
12
+(`random-mm`) and HuggingFace datasets (`hf`). Warmup requests are run before
13
+measurement to ensure stable results.
14
+
15
+## Quick Start
16
+
17
+```bash
18
+vllm bench mm-processor \
19
+ --model Qwen/Qwen2-VL-7B-Instruct \
20
+ --dataset-name random-mm \
21
+ --num-prompts 50 \
22
+ --random-input-len 300 \
23
+ --random-output-len 40 \
24
+ --random-mm-base-items-per-request 2 \
25
+ --random-mm-limit-mm-per-prompt '{"image": 3, "video": 0}' \
26
+ --random-mm-bucket-config '{(256, 256, 1): 0.7, (720, 1280, 1): 0.3}'
27
+```
28
+
29
+## Measured Stages
30
+
31
+| Stage | Description |
32
+|-------|-------------|
33
+| `get_mm_hashes_secs` | Time spent hashing multimodal inputs |
34
+| `get_cache_missing_items_secs` | Time spent looking up the processor cache |
35
+| `apply_hf_processor_secs` | Time spent in the HuggingFace processor |
36
+| `merge_mm_kwargs_secs` | Time spent merging multimodal kwargs |
37
+| `apply_prompt_updates_secs` | Time spent updating prompt tokens |
38
+| `preprocessor_total_secs` | Total preprocessing time |
39
+| `encoder_forward_secs` | Time spent in the encoder model forward pass |
40
+| `num_encoder_calls` | Number of encoder invocations per request |
41
+
42
+The benchmark also reports end-to-end latency (TTFT + decode time) per
43
+request. Use `--metric-percentiles` to select which percentiles to report
44
+(default: p99) and `--output-json` to save results.
45
+
46
+For more examples (HF datasets, warmup, JSON output), see
47
+[Benchmarking CLI — Multimodal Processor Benchmark](../../benchmarking/cli.md#multimodal-processor-benchmark).
48
+
49
## JSON CLI Arguments
50
51
--8<-- "docs/cli/json_tip.inc.md"