A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 6, 00:45 UTC
[Docs] Add Encoder (ViT) CUDA Graphs section to CUDA Graphs design doc (#37914)
Signed-off-by: Baorun Mu <bmu@nvidia.com>
Baorun (Lauren) Mu committed
Mar 24, 2026 at 22:53 UTC
9d0351c91d3115215f84f3bd4b9f366d3fbd13b3
2 files changed
+170
docs/design/cuda_graphs.md
+1
@@ -12,6 +12,7 @@ In this document we will discuss the:
12
* [CUDA Graphs modes](#cudagraphmodes)
13
* [Detailed design](#detailed-design)
14
* [Example usage of the different CUDA Graphs modes](#usage-guide)
15
+* [Vision Encoder (ViT) CUDA Graphs](cuda_graphs_multimodal.md)
16
17
!!! note
18
In this document, we refer to pure decode (`max_query_len=1`) or speculative decode (`max_query_len =1+num_spec_tokens`) as **uniform decode** batches, and the opposite would be **non-uniform** batches (i.e., prefill or mixed prefill-decode batches).
docs/design/cuda_graphs_multimodal.md
new
+169
@@ -0,0 +1,169 @@
1
+# Vision Encoder (ViT) CUDA Graphs
2
+
3
+The [CUDA Graphs](cuda_graphs.md) infrastructure in vLLM primarily targets the **decoder** (language model) forward pass. vLLM also supports capturing the **encoder** (vision transformer) forward pass as CUDA Graphs, independently from the decoder. This is based on <https://github.com/vllm-project/vllm/pull/35963>.
4
+
5
+!!! note
6
+ Encoder CUDA Graphs are orthogonal to decoder CUDA Graphs — both can be enabled simultaneously. Encoder graphs capture the vision encoder execution (e.g., ViT in Qwen3-VL), while decoder graphs capture the language model execution as described in the [CUDA Graphs design document](cuda_graphs.md).
7
+
8
+## Motivation
9
+
10
+Vision encoder inference incurs CUDA kernel launch overhead on the host side. The overhead is more significant when the batch size is small or image size is small.
11
+
12
+Encoder CUDA Graphs eliminate this overhead by pre-capturing the full encoder forward pass at multiple token budget levels during model initialization, then replaying the appropriate graph at runtime.
13
+
14
+## Design
15
+
16
+The encoder CUDA Graph system uses a **budget-based capture/replay** strategy, managed by [EncoderCudaGraphManager][vllm.v1.worker.gpu.mm.encoder_cudagraph.EncoderCudaGraphManager]. The system contains the following core components:
17
+
18
+* [EncoderCudaGraphManager][vllm.v1.worker.gpu.mm.encoder_cudagraph.EncoderCudaGraphManager]: orchestrates capture, replay, greedy packing, and data-parallel execution for encoder CUDA Graphs.
19
+* [SupportsEncoderCudaGraph][vllm.model_executor.models.interfaces.SupportsEncoderCudaGraph]: a runtime-checkable protocol that models implement to opt-in to encoder CUDA Graphs.
20
+* [BudgetGraphMetadata][vllm.v1.worker.gpu.mm.encoder_cudagraph.BudgetGraphMetadata]: holds the captured CUDA Graph and its associated I/O buffers for a single token budget level.
21
+
22
+### Budget-based graph capture
23
+
24
+Multiple CUDA Graphs are pre-captured at different **token budget** levels (e.g., `[2048, 4096, 8192, 13824]`). Each budget defines a fixed token capacity, and all budgets share the same maximum batch size (number of images). The `BudgetGraphMetadata` for each level stores the graph along with pre-allocated input, metadata, and output buffers:
25
+
26
+```python
27
+@dataclass
28
+class BudgetGraphMetadata:
29
+ token_budget: int
30
+ max_batch_size: int
31
+ graph: torch.cuda.CUDAGraph
32
+ input_buffer: torch.Tensor # e.g. pixel_values
33
+ metadata_buffers: dict[str, torch.Tensor] # e.g. embeddings, seq metadata
34
+ output_buffer: torch.Tensor # encoder hidden states
35
+```
36
+
37
+Budgets are auto-generated as power-of-2 levels from a model-provided range via `get_encoder_cudagraph_budget_range()`, with the maximum budget always included even if it does not fall on a power-of-2 boundary. Budgets can also be explicitly specified by the user via `encoder_cudagraph_token_budgets` in `CompilationConfig`.
38
+
39
+### Greedy bin-packing at runtime
40
+
41
+When a batch of images arrives, the manager sorts images by output token count (smallest first) and greedily packs as many images as possible into each sub-batch while staying within the **largest** token budget and the maximum batch size. Once a sub-batch is finalized (the next image would overflow either constraint), the manager finds the **smallest** budget that fits the sub-batch's total tokens and replays the corresponding CUDA Graph. This repeats until the batch is exhausted. Images that exceed all budgets fall back to eager execution.
42
+
43
+For each graph replay:
44
+
45
+1. Zero the pre-allocated `input_buffer`, then copy input tensors (e.g., `pixel_values`) into it.
46
+2. Zero `metadata_buffers`, then slice-copy precomputed values (e.g., rotary embeddings, sequence metadata).
47
+3. Replay the CUDA Graph.
48
+4. Clone outputs from `output_buffer` (cloning is necessary since the buffer is reused across replays).
49
+
50
+### Data-parallel support
51
+
52
+When `mm_encoder_tp_mode="data"`, the manager distributes images across TP ranks using load-balanced assignment via `get_load_balance_assignment`, executes locally on each rank, then gathers results back in the original order via `tensor_model_parallel_all_gather`.
53
+
54
+## Model integration via `SupportsEncoderCudaGraph`
55
+
56
+Models opt-in to encoder CUDA Graphs by implementing the [SupportsEncoderCudaGraph][vllm.model_executor.models.interfaces.SupportsEncoderCudaGraph] protocol. This protocol encapsulates all model-specific logic so that the manager remains model-agnostic. The protocol defines the following methods:
57
+
58
+* `get_encoder_cudagraph_config()` — returns static configuration (supported modalities, input key, buffer keys, output hidden size).
59
+* `get_encoder_cudagraph_budget_range(vllm_config)` — returns `(min_budget, max_budget)` for auto-inference of token budgets.
60
+* `get_encoder_cudagraph_num_items(mm_kwargs)` — returns the number of items (e.g. images) in the batch.
61
+* `get_encoder_cudagraph_per_item_output_tokens(mm_kwargs)` — returns per-item output token counts, used for greedy packing.
62
+* `get_encoder_cudagraph_per_item_input_sizes(mm_kwargs)` — returns per-item input sizes (e.g. patch counts), used for DP load balancing.
63
+* `select_encoder_cudagraph_items(mm_kwargs, indices)` — extracts a sub-batch of items by index, used during greedy packing and DP sharding.
64
+* `prepare_encoder_cudagraph_capture_inputs(...)` — creates dummy inputs for graph capture.
65
+* `prepare_encoder_cudagraph_replay_buffers(...)` — computes new buffer values from actual batch inputs before replay.
66
+* `encoder_cudagraph_forward(...)` — forward pass using precomputed buffers (called during capture and replay).
67
+* `encoder_eager_forward(...)` — fallback eager forward when no graph fits.
68
+
69
+Currently supported: **Qwen3-VL** (see `vllm/model_executor/models/qwen3_vl.py`).
70
+
71
+!!! note
72
+ The `SupportsEncoderCudaGraph` protocol is designed to be model-agnostic. New vision encoder models can opt-in by implementing the protocol methods without modifying the manager.
73
+
74
+!!! note
75
+ Encoder CUDA Graphs have currently been tested with `--mm-encoder-attn-backend=FLASH_ATTN` and `--mm-encoder-attn-backend=FLASHINFER` on Blackwell GPUs.
76
+
77
+## Configuration
78
+
79
+Three fields in `CompilationConfig` control encoder CUDA Graphs:
80
+
81
+* `cudagraph_mm_encoder` (`bool`, default `False`) — enable CUDA Graph capture for multimodal encoder. When enabled, captures the full encoder forward as a CUDA Graph for each token budget level.
82
+* `encoder_cudagraph_token_budgets` (`list[int]`, default `[]`) — token budget levels for capture. If empty (default), auto-inferred from model architecture as power-of-2 levels. User-provided values override auto-inference.
83
+* `encoder_cudagraph_max_images_per_batch` (`int`, default `0`) — maximum number of images per batch during capture. If 0 (default), auto-inferred as `max_budget // min_budget`.
84
+
85
+## Usage guide
86
+
87
+Enable encoder CUDA Graphs via `compilation_config`:
88
+
89
+```bash
90
+vllm serve Qwen/Qwen3-VL-32B \
91
+ --compilation-config '{"cudagraph_mm_encoder": true}'
92
+```
93
+
94
+With explicit budgets:
95
+
96
+```bash
97
+vllm serve Qwen/Qwen3-VL-32B \
98
+ --compilation-config '{"cudagraph_mm_encoder": true, "encoder_cudagraph_token_budgets": [2048, 4096, 8192, 13824], "encoder_cudagraph_max_images_per_batch": 8}'
99
+```
100
+
101
+Python example:
102
+
103
+```python
104
+import vllm
105
+
106
+compilation_config = {
107
+ "cudagraph_mm_encoder": True,
108
+ # Optional: override auto-inferred budgets
109
+ # "encoder_cudagraph_token_budgets": [2048, 4096, 8192, 13824],
110
+ # "encoder_cudagraph_max_images_per_batch": 8,
111
+}
112
+
113
+model = vllm.LLM(
114
+ model="Qwen/Qwen3-VL-32B",
115
+ compilation_config=compilation_config,
116
+)
117
+```
118
+
119
+The manager tracks hit/miss statistics and logs them periodically. A "hit" means an image was processed via CUDA Graph replay; a "miss" means eager fallback (image exceeded all budgets).
120
+
121
+## About the Performance
122
+
123
+The following benchmarks were run on Blackwell GPUs (GB200) using `vllm bench mm-processor`. See [#35963](https://github.com/vllm-project/vllm/pull/35963) for full details.
124
+
125
+### Single GPU (1x GB200)
126
+
127
+Model: `Qwen/Qwen3-VL-30B-A3B-Instruct`, dataset: `lmarena-ai/VisionArena-Chat` (3000 prompts, 300 warmup), `max_model_len=32768`.
128
+
129
+| Backend | Mean latency improvement | P99 latency improvement |
130
+| :------ | :----------------------- | :---------------------- |
131
+| FLASH_ATTN | +11.8% (5.13→4.52ms) | +31.6% (9.16→6.26ms) |
132
+| FLASHINFER | +19.6% (5.42→4.36ms) | +40.3% (10.87→6.49ms) |
133
+
134
+To reproduce:
135
+
136
+```bash
137
+vllm bench mm-processor \
138
+ --model Qwen/Qwen3-VL-30B-A3B-Instruct \
139
+ --dataset-name hf --dataset-path lmarena-ai/VisionArena-Chat \
140
+ --num-prompts 3000 --num-warmups 300 \
141
+ --max-model-len 32768 --seed 42 \
142
+ --mm-encoder-attn-backend FLASH_ATTN \
143
+ --compilation-config '{"cudagraph_mm_encoder": true, "encoder_cudagraph_token_budgets": [512, 1024, 1536, 2048, 2560, 3072, 3584, 4096, 4864], "encoder_cudagraph_max_images_per_batch": 8}'
144
+```
145
+
146
+### Multi-GPU (4x GB200, TP=4, DP=4)
147
+
148
+Model: `Qwen/Qwen3-VL-32B-Instruct`, dataset: `random-mm` (1000 prompts, 200 warmup, 20 images/request at 336x336), `max_model_len=8192`.
149
+
150
+| Backend | Mean latency improvement | P99 latency improvement |
151
+| :------ | :----------------------- | :---------------------- |
152
+| FLASH_ATTN | +18.4% (28.39→23.16ms) | +14.0% (238.78→205.28ms) |
153
+| FLASHINFER | +44.4% (23.24→12.91ms) | +84.9% (172.41→26.05ms) |
154
+
155
+To reproduce:
156
+
157
+```bash
158
+vllm bench mm-processor \
159
+ --model Qwen/Qwen3-VL-32B-Instruct \
160
+ --dataset-name random-mm \
161
+ --random-mm-base-items-per-request 20 \
162
+ --random-mm-num-mm-items-range-ratio 0.0 \
163
+ --random-mm-bucket-config '{"(336,336,1)": 1.0}' \
164
+ --num-prompts 1000 --num-warmups 200 \
165
+ --max-model-len 8192 --seed 42 \
166
+ --mm-encoder-attn-backend FLASHINFER \
167
+ --tensor-parallel-size 4 --mm-encoder-tp-mode data \
168
+ --compilation-config '{"cudagraph_mm_encoder": true, "encoder_cudagraph_token_budgets": [512, 1024, 1536, 2048, 2560, 3072, 3584, 4096, 4864], "encoder_cudagraph_max_images_per_batch": 8}'
169
+```