A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 4, 22:30 UTC
[Doc] Update ViT CUDA graph doc for mixed (image+video) inputs (#40355)
Signed-off-by: shen-shanshan <467638484@qq.com>
Shanshan Shen committed
Apr 21, 2026 at 10:31 UTC
8097591286909deb2da2e7abf3eeae0560f5be66
1 file changed
+2
-5
docs/design/cuda_graphs_multimodal.md
+2
-5
@@ -52,14 +52,14 @@ For each graph replay:
52
53
When `mm_encoder_tp_mode="data"`, the manager distributes images across TP ranks using load-balanced assignment via `get_load_balance_assignment`, executes locally on each rank, then gathers results back in the original order via `tensor_model_parallel_all_gather`.
54
55
-### Video inference support (experimental)
55
+### Video inference support
56
57
Following <https://github.com/vllm-project/vllm/pull/35963> (ViT full CUDA graph support for image inference), <https://github.com/vllm-project/vllm/pull/38061> extends the encoder CUDA graph framework to support video inference for Qwen3-VL. Previously, the CUDA graph capture/replay path only handled image inputs (`pixel_values` + `image_grid_thw`). Video inputs use different keys (`pixel_values_videos` + `video_grid_thw`) and require larger `cu_seqlens` buffers because each video item contributes multiple frames (`T` attention sequences). This PR generalizes the protocol and manager to handle both modalities through a single shared graph manager.
58
59
!!! note
60
Video CUDA graphs are automatically disabled when EVS (Efficient Video Sampling) pruning is enabled, since EVS makes the token count data-dependent and incompatible with CUDA graph capture.
61
62
- Currently, we only support image-only or video-only inputs when enabling CUDA graph, mixed inputs (image + video) are not supported yet (we will work on it in the near future). Thus, it's recommended to turn off the image modality by `--limit-mm-per-prompt '{"image": 0}'` for video-only inputs.
62
+ Mixed inputs (image+video) per prompt are also supported now.
63
64
## Model integration via `SupportsEncoderCudaGraph`
65
@@ -142,7 +142,6 @@ Enable encoder CUDA Graphs via `compilation_config`:
142
143
```bash
144
vllm serve Qwen/Qwen3-VL-32B \
145
- --limit-mm-per-prompt '{"image": 0}' \
145
--compilation-config '{"cudagraph_mm_encoder": true}'
146
```
147
@@ -150,7 +149,6 @@ With explicit budgets:
149
150
```bash
151
vllm serve Qwen/Qwen3-VL-32B \
153
- --limit-mm-per-prompt '{"image": 0}' \
152
--compilation-config '{"cudagraph_mm_encoder": true, "encoder_cudagraph_token_budgets": [2048, 4096, 8192, 13824], "encoder_cudagraph_max_vision_items_per_batch": 8, "encoder_cudagraph_max_frames_per_batch": 64}'
153
```
154
@@ -169,7 +167,6 @@ compilation_config = {
167
168
model = vllm.LLM(
169
model="Qwen/Qwen3-VL-32B",
172
- limit_mm_per_prompt='{"image": 0}',
170
compilation_config=compilation_config,
171
)
172
```