A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 7, 02:45 UTC
[docs][torch.compile] Add fusions.md — kernel/operator fusion reference page (#35538)
Signed-off-by: ProExpertProg <luka.govedic@gmail.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ProExpertProg <11367180+ProExpertProg@users.noreply.github.com> Co-authored-by: ProExpertProg <luka.govedic@gmail.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Copilot committed
Mar 6, 2026 at 23:55 UTC
ce8546a12b613085e5d1d0e110f2c970774a1a84
5 files changed
+428
-39
docs/configuration/optimization.md
+11
@@ -5,6 +5,17 @@ This guide covers optimization strategies and performance tuning for vLLM V1.
5
!!! tip
6
Running out of memory? Consult [this guide](./conserving_memory.md) on how to conserve memory.
7
8
+## Optimization Levels
9
+
10
+vLLM provides 4 optimization levels (`-O0`, `-O1`, `-O2`, `-O3`) that allow users to trade off startup time for performance:
11
+
12
+- `-O0`: No optimizations. Fastest startup time, but lowest performance.
13
+- `-O1`: Fast optimization. Simple compilation and fast fusions, and PIECEWISE cudagraphs.
14
+- `-O2`: Default optimization. Additional compilation ranges, additional fusions, FULL_AND_PIECEWISE cudagraphs.
15
+- `-O3`: Aggressive optimization. Currently equal to `-O2`, but may include additional time-consuming or experimental optimizations in the future.
16
+
17
+For more information, see the [optimization level documentation](../design/optimization_levels.md).
18
+
19
## Preemption
20
21
Due to the autoregressive nature of transformer architecture, there are times when KV cache space is insufficient to handle all batched requests.
docs/design/fusions.md
new
+339
@@ -0,0 +1,339 @@
1
+# Fusion torch.compile passes
2
+
3
+vLLM applies a set of kernel/operator fusions at compile time (via custom [`torch.compile`](torch_compile.md) Inductor passes)
4
+to separate optimizations from model definitions and avoid breaking layer abstractions in model code.
5
+These fusions are controlled by fields in [`PassConfig`][vllm.config.compilation.PassConfig] and are automatically enabled
6
+at appropriate [optimization levels](optimization_levels.md).
7
+
8
+## Quick Reference
9
+
10
+The table below maps each fusion to its controlling flag/config knob, the
11
+operations it fuses, what level enables it by default, and an indicative speedup.
12
+The Fullgraph column indicates whether the fusion requires the entire model graph to be
13
+visible (either via Inductor partition or `splitting_ops=[]`),
14
+and the last column indicates whether the fusion activates for all `num_tokens`
15
+or just on the low or high end.
16
+
17
+!!! info
18
+ Speedup depends heavily on the exact model, batch size, and hardware.
19
+ If tuning performance by hand, always benchmark your exact use-case with and without the fusion to verify the impact.
20
+
21
+| Fusion | `PassConfig` flag | Fused operations | Default at | E2E Speedup | Fullgraph | `num_tokens` |
22
+|--------------------------------------------------------------------------------|------------------------------|------------------------------------------------|--------------------------------|--------------------|-----------|--------------|
23
+| [AllReduce + RMSNorm](#allreduce--rmsnorm-fuse_allreduce_rms) | `fuse_allreduce_rms` | All-reduce → RMSNorm (+residual_add) (→ quant) | O2 (Hopper/Blackwell + TP > 1) | 5-20% | No | Low |
24
+| [Attention + Quant](#attention--quantization-fuse_attn_quant) | `fuse_attn_quant` | Attention output → FP8/NVFP4 quant | Off by default | 3-7% | Yes | Always |
25
+| [RoPE + KV-Cache Update](#rope--kv-cache-update-fuse_rope_kvcache) | `fuse_rope_kvcache` | Rotary embedding → KV cache write | O1 (ROCm/AITER only) | TBD | No | Low |
26
+| [QK Norm + RoPE](#qk-norm--rope-enable_qk_norm_rope_fusion) | `enable_qk_norm_rope_fusion` | Q/K RMSNorm → rotary embedding | Off by default | 2-3% | No | Low |
27
+| [Sequence Parallelism](#sequence-parallelism-enable_sp) | `enable_sp` | AllReduce → ReduceScatter + AllGather | Off by default | Prereq for AsyncTP | Yes | High |
28
+| [AsyncTP GEMM + collective](#asynctp-gemm--collective-overlap-fuse_gemm_comms) | `fuse_gemm_comms` | GEMM → reduce-scatter / all-gather → GEMM | Off by default | 7-10% | Yes | High |
29
+| [RMSNorm + Quant](#rmsnorm--quantization-fuse_norm_quant) | `fuse_norm_quant` | RMSNorm (+residual add) → FP8/FP4 quant | O1 (conditional) | 1-4% | No | Always |
30
+| [SiLU+Mul + Quant](#silumul--quantization-fuse_act_quant) | `fuse_act_quant` | SiLU+Mul activation → FP8/FP4 quant | O1 (conditional) | 1-4% | No | Always |
31
+| [RMSNorm + Padding](#rmsnorm--padding-fuse_act_padding) | `fuse_act_padding` | Residual add + RMSNorm → padding | O1 (ROCm/AITER only) | TBD | No | Always |
32
+
33
+## Support Matrix
34
+
35
+The table below lists the quantization schemes supported by each fusion on each platform.
36
+**—** means the fusion is not available on that platform. The latest and in-progress work is available in the tracking issue:
37
+[#36066](https://github.com/vllm-project/vllm/issues/36066)
38
+
39
+| Fusion | SM100 (Blackwell) | SM90 (Hopper) | SM89 (Ada) | SM80 (Ampere) | ROCm |
40
+|------------------------------|------------------------------------------|------------------------------------------|------------------------------------------|---------------|------------------------------------------|
41
+| `fuse_allreduce_rms` | FP16/BF16, FP8 static, NVFP4 | FP16/BF16, FP8 static | — | — | — |
42
+| `fuse_attn_quant`\* | FP8 static\*, NVFP4\* | FP8 static\* | FP8 static\* | — | FP8 static\* |
43
+| `fuse_rope_kvcache` | — | — | — | — | FP16/BF16 |
44
+| `enable_qk_norm_rope_fusion` | FP16/BF16 | FP16/BF16 | FP16/BF16† | FP16/BF16† | — |
45
+| `enable_sp` | FP16/BF16, FP8 static† | FP16/BF16, FP8 static | FP16/BF16† | FP16/BF16† | — |
46
+| `fuse_gemm_comms` | FP16/BF16, FP8 static† | FP16/BF16, FP8 static | FP16/BF16† | FP16/BF16† | — |
47
+| `fuse_norm_quant` | FP8 static, FP8 per-token, FP8 per-group | FP8 static, FP8 per-token, FP8 per-group | FP8 static, FP8 per-token, FP8 per-group | — | FP8 static, FP8 per-token, FP8 per-group |
48
+| `fuse_act_quant` | FP8 static, NVFP4 | FP8 static | FP8 static | — | FP8 per-group |
49
+| `fuse_act_padding` | — | — | — | — | FP16/BF16 |
50
+
51
+\* `fuse_attn_quant` support depends on the attention backend in use; not all backends support
52
+fused quantization output. See the [`fuse_attn_quant` section](#attention--quantization-fuse_attn_quant)
53
+for per-backend details.
54
+
55
+† `enable_sp` and `fuse_gemm_comms` are only autoconfigured for SM90 today;
56
+other architectures support requires setting `PassConfig.sp_min_token_num` explicitly.
57
+SM100 support also requires setting `VLLM_DISABLED_KERNELS=FlashInferFP8ScaledMMLinearKernel`.
58
+
59
+## Enabling / Disabling Fusions
60
+
61
+Fusions are exposed through `PassConfig`, which is nested inside `CompilationConfig`:
62
+
63
+```python
64
+from vllm import LLM
65
+from vllm.config import CompilationConfig, PassConfig
66
+
67
+llm = LLM(
68
+ model="...",
69
+ optimization_level=2, # Default optimization level
70
+ compilation_config=CompilationConfig(
71
+ pass_config=PassConfig(
72
+ fuse_norm_quant=True,
73
+ fuse_act_quant=True,
74
+ fuse_allreduce_rms=False, # disable a specific fusion
75
+ )
76
+ ),
77
+)
78
+```
79
+
80
+Fusions can also be enabled using command-line flags with any `vllm ...` command:
81
+
82
+```bash
83
+# Enable O2 defaults, but turn off allreduce fusion
84
+vllm serve meta-llama/Llama-3.1-8B-Instruct -O2 -cc.pass_config.fuse_allreduce_rms=False
85
+
86
+# The above is equivalent to the more verbose:
87
+vllm serve meta-llama/Llama-3.1-8B-Instruct -O2 --compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}'
88
+
89
+# Same syntax in other commands, e.g. vllm bench:
90
+vllm bench latency --model=meta-llama/Llama-3.1-8B-Instruct -O2 -cc.pass_config.fuse_allreduce_rms=False
91
+```
92
+
93
+Fields set explicitly by the user always take precedence over optimization-level defaults.
94
+
95
+## Fusion Details
96
+
97
+### AllReduce + RMSNorm (`fuse_allreduce_rms`)
98
+
99
+!!! warning
100
+ TP+DP and TP+PP combinations are currently broken
101
+ ([#34458](https://github.com/vllm-project/vllm/issues/34458) and
102
+ [#35426](https://github.com/vllm-project/vllm/issues/35426)).
103
+ Only supported on NVIDIA Hopper (SM90) and Blackwell (SM100) with FlashInfer installed.
104
+
105
+**What it fuses.** Fuses the tensor-parallel all-reduce collective with the subsequent residual add,
106
+RMSNorm, and optionally a quantization step into a single FlashInfer / TRT-LLM communication kernel.
107
+This fusion is only profitable for small `num_tokens`,
108
+so the fusion is only performed in the lower compiled range.
109
+
110
+Patterns covered:
111
+
112
+- `AllReduce → RMSNorm(+residual_add)`: CUDA sm90+ with FlashInfer
113
+- `AllReduce → RMSNorm(+residual_add) → FP8 static quant`: CUDA sm90+ with FlashInfer
114
+- `AllReduce → RMSNorm(+residual_add) → NVFP4 dynamic quant`: CUDA sm100+ with FlashInfer
115
+
116
+The maximum tensor size below which the fused kernel is used is hardware-dependent (64 MB for TP=2
117
+on SM90/SM100) and configurable via `PassConfig.fi_allreduce_fusion_max_size_mb`.
118
+
119
+**Code locations.**
120
+
121
+- Pass: [`vllm/compilation/passes/fusion/allreduce_rms_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/allreduce_rms_fusion.py)
122
+- FlashInfer all-reduce: [`vllm/distributed/device_communicators/flashinfer_all_reduce.py`](https://github.com/vllm-project/vllm/blob/main/vllm/distributed/device_communicators/flashinfer_all_reduce.py)
123
+- Benchmark: [`benchmarks/kernels/benchmark_fused_collective.py`](https://github.com/vllm-project/vllm/blob/main/benchmarks/kernels/benchmark_fused_collective.py)
124
+
125
+### Attention + Quantization (`fuse_attn_quant`)
126
+
127
+!!! info
128
+ `fuse_attn_quant` is currently not enabled at any optimization level by default and must be set
129
+ explicitly. It requires the full model graph to be visible (Inductor partition or `splitting_ops=[]`).
130
+
131
+**What it fuses.** Fuses the attention output quantization directly after the attention computation,
132
+eliminating a full-precision memory round-trip of the attention output. Patterns covered:
133
+
134
+`Attention → FP8 static quant`:
135
+
136
+- `TRITON_ATTN`: CUDA, ROCm
137
+- `FLASHINFER`: CUDA sm100+ with FlashInfer installed
138
+- `ROCM_ATTN`: ROCm
139
+- `ROCM_AITER_UNIFIED_ATTN`: ROCm with AITER
140
+
141
+`Attention → NVFP4 dynamic quant`:
142
+
143
+- `FLASHINFER`: CUDA sm100+ with FlashInfer installed
144
+
145
+Other attention backends do not support fused output quantization yet.
146
+
147
+**Code locations.**
148
+
149
+- Pass: [`vllm/compilation/passes/fusion/attn_quant_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/attn_quant_fusion.py)
150
+- Attention backends: [`vllm/v1/attention/backends/`](https://github.com/vllm-project/vllm/blob/main/vllm/v1/attention/backends/)
151
+
152
+### RoPE + KV-Cache Update (`fuse_rope_kvcache`)
153
+
154
+!!! info
155
+ ROCm/AITER-only. Not available on NVIDIA CUDA or CPU. The fusion is only enabled for
156
+ `num_tokens ≤ 256` by default due to AITER fused kernel performance issues.
157
+ This threshold is configurable via `PassConfig.rope_kvcache_fusion_max_token_num`.
158
+
159
+**What it fuses.** Fuses the rotary positional embedding kernel with the KV-cache scatter/write into
160
+a single kernel, avoiding separate reads and writes of the key and value tensors.
161
+
162
+Requires: AMD ROCm with AITER enabled, the `rotary_embedding` custom op active (automatic),
163
+and the `kv_cache` update op visible in the graph: either by using Inductor graph partition
164
+or removed from `splitting_ops`.
165
+If these conditions are set, the fusion is enabled automatically for optimization level O1 and above.
166
+
167
+**Code locations.**
168
+
169
+- Pass: [`vllm/compilation/passes/fusion/rope_kvcache_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/rope_kvcache_fusion.py)
170
+
171
+### Sequence Parallelism (`enable_sp`)
172
+
173
+**What it fuses.** Replaces all-reduce collectives with reduce-scatter + local RMSNorm + all-gather,
174
+splitting the sequence dimension across TP ranks. This restructures the graph so the subsequent AsyncTP
175
+pass can fuse the reduce-scatter / all-gather with the surrounding GEMMs.
176
+
177
+Sequence Parallelism itself does not directly improve performance; it is a prerequisite for the
178
+AsyncTP pass (`fuse_gemm_comms`). SP is only applied above a minimum token threshold that is
179
+autoconfigured based on device capability and model `hidden_size`. Currently only active on
180
+H100/SM90 for models with `hidden_size >= 8192`. The threshold is configurable via
181
+`PassConfig.sp_min_token_num`.
182
+
183
+The general transformation:
184
+
185
+```text
186
+Input → AllReduce → RMSNorm → Output
187
+becomes:
188
+Input → ReduceScatter → local RMSNorm → AllGather → Output
189
+```
190
+
191
+Patterns covered:
192
+
193
+- First block: `AllReduce → RMSNorm` → `ReduceScatter → RMSNorm → AllGather`
194
+- Middle blocks: `AllReduce → fused_add_RMSNorm` → `ReduceScatter → fused_add_RMSNorm → AllGather`
195
+- Both with optional `→ FP8 static quant` suffix
196
+
197
+Requires: `use_inductor_graph_partition=True` **or** piecewise compilation with static sizes
198
+divisible by `tensor_parallel_size`.
199
+
200
+Supported hardware: Only tested on NVIDIA CUDA, possibly works on ROCm. FP8 all-gather requires sm90+.
201
+
202
+**Code locations.**
203
+
204
+- Pass: [`vllm/compilation/passes/fusion/sequence_parallelism.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/sequence_parallelism.py)
205
+
206
+### AsyncTP GEMM + Collective Overlap (`fuse_gemm_comms`)
207
+
208
+!!! info
209
+ Requires `enable_sp=True` (enabled automatically). This pass is a no-op if Sequence Parallelism has not been applied.
210
+
211
+**What it fuses.** After Sequence Parallelism transforms the graph, fuses GEMM kernels with the
212
+surrounding reduce-scatter (output projection) and all-gather (input projection) using
213
+`torch.ops.symm_mem` symmetric-memory primitives, overlapping communication and computation.
214
+This overlap is only profitable for large `num_tokens`, so the fusion (and preceding SP)
215
+is only performed in the higher compiled range above `PassConfig.sp_min_token_num`.
216
+
217
+Patterns covered:
218
+
219
+- `GEMM → reduce-scatter` → `fused_matmul_reduce_scatter`
220
+- `all-gather → GEMM` → `all_gather_matmul`
221
+- FP8 scaled variants of both patterns
222
+
223
+Supported hardware: NVIDIA CUDA with symmetric-memory (`torch.distributed._symmetric_memory`) support.
224
+
225
+On B200, pattern-matching fp8 FlashInfer scaled MM is not supported, so it must be disabled
226
+([#27893](https://github.com/vllm-project/vllm/issues/27893))
227
+
228
+```shell
229
+VLLM_DISABLED_KERNELS=FlashInferFP8ScaledMMLinearKernel ...
230
+```
231
+
232
+**Code locations.**
233
+
234
+- Pass: [`vllm/compilation/passes/fusion/collective_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/collective_fusion.py)
235
+- Sequence parallelism pass: [`vllm/compilation/passes/fusion/sequence_parallelism.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/sequence_parallelism.py)
236
+
237
+### QK Norm + RoPE (`enable_qk_norm_rope_fusion`)
238
+
239
+!!! info
240
+ Only applicable to models that apply per-head RMSNorm to Q and K before rotary positional
241
+ embedding (e.g. Qwen). Not enabled by default at any optimization level due to perf issues on H100:
242
+ [#34391](https://github.com/vllm-project/vllm/issues/34391)
243
+
244
+**What it fuses.** Fuses the sequence: split QKV → reshape → Q/K RMSNorm → reshape → rotary
245
+embedding into a single `fused_qk_norm_rope` CUDA kernel.
246
+
247
+```text
248
+# Unfused:
249
+q, k, v = split(qkv)
250
+q_norm = rms_norm(q.view(heads))
251
+k_norm = rms_norm(k.view(kv_heads))
252
+q_rope, k_rope = rotary_embedding(q_norm, k_norm, ...)
253
+
254
+# Fused:
255
+fused_qk_norm_rope(qkv, ...)
256
+```
257
+
258
+Supported hardware: CUDA (sm80+) only, tested only on sm90 and sm100.
259
+
260
+**Code locations.**
261
+
262
+- Pass: [`vllm/compilation/passes/fusion/qk_norm_rope_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/qk_norm_rope_fusion.py)
263
+- CUDA kernel: [`csrc/ops.h`](https://github.com/vllm-project/vllm/blob/main/csrc/ops.h) (`fused_qk_norm_rope`)
264
+
265
+### RMSNorm + Quantization (`fuse_norm_quant`)
266
+
267
+!!! warning
268
+ On NVIDIA, Inductor actually generates a faster fused kernel than our custom CUDA kernel.
269
+ Hence, this fusion is only enabled when either `rms_norm` or `quant_fp8` is using a custom kernel.
270
+
271
+**What it fuses.** Combines the custom `rms_norm` / `fused_add_rms_norm`
272
+operations with subsequent quantization into a single fused kernel,
273
+eliminating an intermediate read/write of the full-precision activation tensor.
274
+Two variants are fused:
275
+
276
+- *Plain RMSNorm + quant*: `rms_norm(x) → quant_fp8(y)`
277
+- *Fused-add RMSNorm + quant*: `fused_add_rms_norm(x, residual) → quant_fp8(y)` — also updates the residual in-place.
278
+
279
+Note that AITER fusions are currently in a separate pass in `vllm.compilation.passes.fusion.rocm_aiter_fusion`.
280
+
281
+Supported quantization scheme/hardware combinations:
282
+
283
+- FP8 static per-tensor: CUDA & HIP kernel
284
+- FP8 dynamic per-token: CUDA & HIP kernel, AITER
285
+- FP8 dynamic per-token-group (128/64): CUDA & HIP kernel, AITER
286
+
287
+**Code locations.**
288
+
289
+- Pass: [`vllm/compilation/passes/fusion/rms_quant_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/rms_quant_fusion.py)
290
+- ROCm AITER pass: [`vllm/compilation/passes/fusion/rocm_aiter_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/rocm_aiter_fusion.py)
291
+- CUDA/HIP kernels: [`csrc/layernorm_quant_kernels.cu`](https://github.com/vllm-project/vllm/blob/main/csrc/layernorm_quant_kernels.cu)
292
+
293
+### SiLU+Mul + Quantization (`fuse_act_quant`)
294
+
295
+!!! warning
296
+ Same as `fuse_norm_quant`: on NVIDIA, Inductor generates a faster fused kernel than our custom ops.
297
+ This fusion is only enabled when either `silu_and_mul` or `quant_fp8` are using a custom kernel,
298
+ or for NVFP4-quantized models (where FP4 quant is always a custom op).
299
+
300
+**What it fuses.** Fuses the `silu_and_mul` gate-up projection activation with subsequent quantization into a single kernel,
301
+avoiding materialization of the full-precision post-activation tensor.
302
+
303
+Note that AITER fusions are in a separate pass in `vllm.compilation.passes.fusion.rocm_aiter_fusion`.
304
+
305
+Supported quantization scheme/hardware combinations:
306
+
307
+- FP8 static per-tensor: CUDA & HIP kernel
308
+- NVFP4 dynamic: CUDA sm100+ only with FlashInfer
309
+- FP8 per-token-group (128): ROCm AITER only
310
+
311
+**Code locations.**
312
+
313
+- Pass: [`vllm/compilation/passes/fusion/act_quant_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/act_quant_fusion.py)
314
+- ROCm AITER pass: [`vllm/compilation/passes/fusion/rocm_aiter_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/rocm_aiter_fusion.py)
315
+- CUDA/HIP kernels: [`csrc/quantization/`](https://github.com/vllm-project/vllm/blob/main/csrc/quantization/)
316
+
317
+### RMSNorm + Padding (`fuse_act_padding`)
318
+
319
+!!! info
320
+ ROCm/AITER-only. Targeted at GPT-OSS models.
321
+
322
+**What it fuses.** Fuses a residual add + RMSNorm with a subsequent padding operation that pads
323
+the hidden dimension to a multiple required by downstream AITER Triton GEMM kernels.
324
+
325
+Requires: AMD ROCm with AITER RMSNorm enabled. Enabled by default in optimization level O1 and above
326
+when the hidden size is 2880 and AITER Triton GEMMs *not* enabled.
327
+
328
+**Code locations.**
329
+
330
+- Pass: [`vllm/compilation/passes/fusion/rocm_aiter_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/rocm_aiter_fusion.py) (`RocmAiterTritonAddRMSNormPadFusionPass`)
331
+
332
+## See Also
333
+
334
+- [Optimization Levels](optimization_levels.md) — high-level presets that set
335
+ fusion defaults.
336
+- [torch.compile in vLLM](torch_compile.md) — how the Inductor pass pipeline
337
+ works.
338
+- [Attention Backends](attention_backends.md) — attention-specific kernel
339
+ selection.
docs/design/optimization_levels.md
+54
-37
@@ -1,64 +1,81 @@
1
-<!-- markdownlint-disable -->
2
-
1
# Optimization Levels
2
3
## Overview
4
7
-vLLM now supports optimization levels (`-O0`, `-O1`, `-O2`, `-O3`). Optimization levels provide an intuitive mechanism for users to trade startup time for performance. Higher levels have better performance but worse startup time. These optimization levels have associated defaults to help users get desired out-of-the-box performance. Importantly, defaults set by optimization levels are purely defaults; explicit user settings will not be overwritten.
5
+vLLM provides 4 optimization levels (`-O0`, `-O1`, `-O2`, `-O3`) that allow users to trade off startup time for performance:
6
+
7
+- `-O0`: No optimization. Fastest startup time, but lowest performance.
8
+- `-O1`: Fast optimization. Simple compilation and fast fusions, and PIECEWISE cudagraphs.
9
+- `-O2`: Default optimization. Additional compilation ranges, additional fusions, FULL_AND_PIECEWISE cudagraphs.
10
+- `-O3`: Aggressive optimization. Currently equal to `-O2`, but may include additional time-consuming or experimental optimizations in the future.
11
+
12
+All optimization level defaults can be achieved by manually setting the underlying flags.
13
+User-set flags take precedence over optimization level defaults.
14
15
## Level Summaries and Usage Examples
16
+
17
```bash
18
# CLI usage
12
-python -m vllm.entrypoints.api_server --model RedHatAI/Llama-3.2-1B-FP8 -O0
19
+python -m vllm.entrypoints.api_server --model RedHatAI/Llama-3.2-1B-FP8 -O1
20
21
# Python API usage
22
from vllm.entrypoints.llm import LLM
23
24
llm = LLM(
25
model="RedHatAI/Llama-3.2-1B-FP8",
19
- optimization_level=0
26
+ optimization_level=2 # equivalent to -O2
27
)
28
```
29
23
-#### `-O1`: Quick Optimizations
24
-- **Startup**: Moderate startup time
25
-- **Performance**: Inductor compilation, CUDAGraphMode.PIECEWISE
26
-- **Use case**: Balance for most development scenarios
30
+### `-O0`: No Optimization
31
28
-```bash
29
-# CLI usage
30
-python -m vllm.entrypoints.api_server --model RedHatAI/Llama-3.2-1B-FP8 -O1
32
+Startup as fast as possible - no autotuning, no compilation, and no cudagraphs.
33
+This level is good for initial phases of development and debugging.
34
32
-# Python API usage
33
-from vllm.entrypoints.llm import LLM
35
+Settings:
36
35
-llm = LLM(
36
- model="RedHatAI/Llama-3.2-1B-FP8",
37
- optimization_level=1
38
-)
39
-```
37
+- `-cc.cudagraph_mode=NONE`
38
+- `-cc.mode=NONE` (also resulting in `-cc.custom_ops=["none"]`)
39
+- `-cc.pass_config.fuse_...=False` (all fusions disabled)
40
+- `--kernel-config.enable_flashinfer_autotune=False`
41
41
-#### `-O2`: Full Optimizations (Default)
42
-- **Startup**: Longer startup time
43
-- **Performance**: `-O1` + CUDAGraphMode.FULL_AND_PIECEWISE
44
-- **Use case**: Production workloads where performance is important. This is the default use case. It is also very similar to the previous default. The primary difference is that noop & fusion flags are enabled.
42
+### `-O1`: Fast Optimization
43
46
-```bash
47
-# CLI usage (default, so optional)
48
-python -m vllm.entrypoints.api_server --model RedHatAI/Llama-3.2-1B-FP8 -O2
44
+Prioritize fast startup, but still enable basic optimizations like compilation and cudagraphs.
45
+This level is a good balance for most development scenarios where you want faster startup but
46
+still make sure your code does not break cudagraphs or compilation.
47
50
-# Python API usage
51
-from vllm.entrypoints.llm import LLM
48
+Settings:
49
53
-llm = LLM(
54
- model="RedHatAI/Llama-3.2-1B-FP8",
55
- optimization_level=2 # This is the default
56
-)
57
-```
50
+- `-cc.cudagraph_mode=PIECEWISE`
51
+- `-cc.mode=VLLM_COMPILE`
52
+- `--kernel-config.enable_flashinfer_autotune=True`
53
+
54
+Fusions:
55
+
56
+- `-cc.pass_config.fuse_norm_quant=True`*
57
+- `-cc.pass_config.fuse_act_quant=True`*
58
+- `-cc.pass_config.fuse_act_padding=True`†
59
+- `-cc.pass_config.fuse_rope_kvcache=True`† (will be moved to O2)
60
+
61
+\* These fusions are only enabled when either op is using a custom kernel, otherwise Inductor fusion is better.</br>
62
+† These fusions are ROCm-only and require AITER.
63
+
64
+### `-O2`: Full Optimization (Default)
65
+
66
+Prioritize performance at the expense of additional startup time.
67
+This level is recommended for production workloads and is hence the default.
68
+Fusions in this level _may_ take longer due to additional compile ranges.
69
+
70
+Settings (on top of `-O1`):
71
+
72
+- `-cc.cudagraph_mode=FULL_AND_PIECEWISE`
73
+- `-cc.pass_config.fuse_allreduce_rms=True`
74
+
75
+### `-O3`: Aggressive Optimization
76
59
-#### `-O3`: Full Optimization
60
-Still in development. Added infrastructure to prevent changing API in future
61
-release. Currently behaves the same O2.
77
+This level is currently the same as `-O2`, but may include additional optimizations
78
+in the future that are more time-consuming or experimental.
79
80
## Troubleshooting
81
@@ -66,4 +83,4 @@ release. Currently behaves the same O2.
83
84
1. **Startup Time Too Long**: Use `-O0` or `-O1` for faster startup
85
2. **Compilation Errors**: Use `debug_dump_path` for additional debugging information
69
-3. **Performance Issues**: Ensure using `-O2` for production
\ No newline at end of file
86
+3. **Performance Issues**: Ensure using `-O2` for production
vllm/config/compilation.py
+21
-2
@@ -4,7 +4,7 @@
4
import enum
5
from collections import Counter
6
from collections.abc import Callable
7
-from dataclasses import field
7
+from dataclasses import field, fields
8
from pathlib import Path
9
from typing import TYPE_CHECKING, Any, ClassVar, Literal
10
@@ -269,6 +269,24 @@ class PassConfig:
269
)
270
self.fuse_rope_kvcache = False
271
272
+ def log_enabled_passes(self) -> None:
273
+ """
274
+ Log the enabled custom fusion passes.
275
+ This is called at the end of VLLMConfig post_init,
276
+ after all defaults are finalized.
277
+ TODO also log the compile ranges for which this is enabled.
278
+ """
279
+ enabled_fusions = [
280
+ f.name[len("fuse_") :]
281
+ for f in fields(self)
282
+ if getattr(self, f.name) and f.name.startswith("fuse_")
283
+ ]
284
+
285
+ if enabled_fusions:
286
+ logger.info_once(
287
+ "Enabled custom fusions: %s", ", ".join(enabled_fusions), scope="global"
288
+ )
289
+
290
291
class DynamicShapesType(str, enum.Enum):
292
"""Types of dynamic shapes handling in torch.compile().
@@ -341,7 +359,8 @@ class CompilationConfig:
359
VLLMConfig's post_init does further initialization. If used outside of the
360
VLLMConfig, some fields will be left in an improper state.
361
344
- It has three parts:
362
+ It contains PassConfig, which controls the custom fusion/transformation passes.
363
+ The rest has three parts:
364
365
- Top-level Compilation control:
366
- [`mode`][vllm.config.CompilationConfig.mode]
vllm/config/vllm.py
+3
@@ -1272,6 +1272,9 @@ class VllmConfig:
1272
# Handle the KV connector configs
1273
self._post_init_kv_transfer_config()
1274
1275
+ # Log the custom passes that are enabled
1276
+ self.compilation_config.pass_config.log_enabled_passes()
1277
+
1278
def update_sizes_for_sequence_parallelism(self, possible_sizes: list) -> list:
1279
# remove the sizes that not multiple of tp_size when
1280
# enable sequence parallelism