@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 128 lines 4.77 KB
Rendered Raw
1 # FP8 ViT Encoder Attention
2
3 For visual understanding workloads with large images (e.g. QHD, 4K) and relatively
4 short text prompts/generation, the ViT encoder attention can become a significant
5 bottleneck, especially when the text model is quantized (e.g. NVFP4). vLLM
6 supports optional FP8 quantization for ViT encoder attention via FlashInfer
7 cuDNN on NVIDIA GPUs and AITER on AMD GPUs. Q/K/V are quantized on-the-fly to
8 FP8 before the attention call.
9
10 !!! note
11 - Currently supports Qwen3-VL family models only (`qwen3_vl`, `qwen3_vl_moe`,
12 `qwen3_5`, `qwen3_5_moe`, and other models using Qwen3 ViT).
13 - Dynamic scaling is not compatible with ViT full CUDA graphs.
14 - Performance gains are mostly visible at QHD/4K resolutions or multi-image
15 requests. Smaller images may see no speedup due to quantization overhead
16 (3 quantization kernel launches + un-padding).
17 - FP8 tensor-core speedup is more pronounced on GB300 than GB200.
18 - On ROCm, packed variable-length image and video batches are supported by
19 AITER's native varlen FP8 attention.
20
21 ## Requirements
22
23 - NVIDIA: FlashInfer cuDNN backend with cuDNN >= 9.17.1.
24 - AMD: AITER with `flash_attn_varlen_fp8_pertensor_func` support on gfx942
25 (MI300 series) or gfx950 (MI350 series).
26
27 ## Usage
28
29 Enable FP8 ViT attention by passing `--mm-encoder-attn-dtype fp8` and selecting
30 the backend for the current platform:
31
32 ```bash
33 vllm serve $MODEL \
34 --mm-encoder-attn-backend FLASHINFER \
35 --mm-encoder-attn-dtype fp8
36
37 # AMD ROCm
38 vllm serve $MODEL \
39 --mm-encoder-attn-backend ROCM_AITER_FA \
40 --mm-encoder-attn-dtype fp8
41 ```
42
43 By default (no scale file), **dynamic scaling** is used: a 16-entry circular
44 buffer of observed Q/K/V amax values drives per-forward scale updates. This
45 matches BF16 accuracy without any calibration but adds a small per-forward
46 overhead.
47
48 ## Calibrate-Once, Reuse Workflow (Recommended)
49
50 For production, calibrate static scales on a representative dataset once and
51 reuse them to avoid the dynamic overhead:
52
53 ```bash
54 # Step 1: calibrate and save scales (runs dynamic scaling for 16 passes,
55 # then dumps the learned scales to JSON).
56 vllm bench mm-processor \
57 --model $MODEL --mm-encoder-attn-backend $MM_ATTN_BACKEND \
58 --mm-encoder-attn-dtype fp8 \
59 --mm-encoder-fp8-scale-save-path /path/to/scales.json \
60 --dataset-name hf --dataset-path lmarena-ai/VisionArena-Chat \
61 --num-prompts 100
62
63 # Step 2: serve with static scales (no dynamic overhead).
64 vllm serve $MODEL \
65 --mm-encoder-attn-backend $MM_ATTN_BACKEND \
66 --mm-encoder-attn-dtype fp8 \
67 --mm-encoder-fp8-scale-path /path/to/scales.json
68 ```
69
70 Saved scales are multiplied by `--mm-encoder-fp8-scale-save-margin` (default
71 `1.5`) to leave headroom against activation outliers not present in the
72 calibration set. The default has been validated to generalize across datasets
73 (e.g. VisionArena-Chat calibration maintains BF16 accuracy on ChartQA).
74
75 ## Scale File Format
76
77 ```json
78 {
79 "visual.blocks.0.attn.attn": {"q": 224.0, "k": 198.0, "v": 210.0},
80 "visual.blocks.1.attn.attn": {"q": 218.0, "k": 195.0, "v": 207.0}
81 }
82 ```
83
84 Keys `q_scale` / `k_scale` / `v_scale` are accepted as aliases.
85
86 ## Performance
87
88 **Core cuDNN attention kernel** (PyTorch profiler, `cudnn_generated_fort_native_sdpa_sm100_flash_fprop`, head_dim=128, seq_len=8192):
89
90 | Hardware | BF16 | FP8 | Speedup |
91 | -------- | ---- | ---- | ------- |
92 | GB200 | 350 us | 312 us | **1.12x** |
93 | GB300 | 300 us | 211 us | **1.42x** |
94
95 **End-to-end encoder forward time** (Qwen3-VL-30B-A3B-Instruct on GB200, 3 images/request):
96
97 | Resolution | BF16 median | FP8 median | Speedup |
98 | ---------- | ----------- | ---------- | ------- |
99 | HD (720x1280) | 31.77 ms | 36.39 ms | 0.87x |
100 | FullHD (1080x1920) | 57.99 ms | 58.73 ms | ~same |
101 | QHD (1440x2560) | 131.83 ms | 122.30 ms | **1.08x** |
102 | 4K (2160x3840) | 543.44 ms | 460.31 ms | **1.18x** |
103
104 Crossover is around FullHD with 3 images/request. At QHD and above, FP8 wins.
105
106 **Complete AITER attention call on MI300X** (BF16 input, 16 heads,
107 head_dim=72; FP8 includes Q/K/V quantization):
108
109 | Sequence length | AITER BF16 | AITER FP8 | Speedup |
110 | --------------- | ---------- | --------- | ------- |
111 | 2304 | 0.467 ms | 0.337 ms | **1.38x** |
112 | 4096 | 0.812 ms | 0.764 ms | **1.06x** |
113 | 8192 | 2.555 ms | 2.364 ms | **1.08x** |
114 | 16384 | 9.769 ms | 8.655 ms | **1.13x** |
115
116 ## Accuracy
117
118 ChartQA, Qwen3-VL-8B-Instruct, 500 samples. FP8 static uses scales calibrated
119 on VisionArena-Chat (with default 1.5x margin):
120
121 | Metric | BF16 | FP8 dynamic | FP8 static |
122 | ------ | ---- | ----------- | ---------- |
123 | relaxed_accuracy | 0.780 | 0.776 | 0.780 |
124 | anywhere_accuracy | 0.806 | 0.816 | 0.814 |
125 | exact_match | 0.584 | 0.582 | 0.578 |
126
127 All three configurations match within statistical noise, confirming that
128 static scales calibrated on one dataset generalize to another.