| 1 | # Optimization Levels |
| 2 | |
| 3 | ## Overview |
| 4 | |
| 5 | vLLM provides 4 optimization levels (`-O0`, `-O1`, `-O2`, `-O3`) that allow users to trade off startup time for performance: |
| 6 | |
| 7 | - `-O0`: No optimization. Fastest startup time, but lowest performance. |
| 8 | - `-O1`: Fast optimization. Simple compilation and fast fusions, and PIECEWISE cudagraphs. |
| 9 | - `-O2`: Default optimization. Additional compilation ranges, additional fusions, FULL_AND_PIECEWISE cudagraphs. |
| 10 | - `-O3`: Aggressive optimization. Currently equal to `-O2`, but may include additional time-consuming or experimental optimizations in the future. |
| 11 | |
| 12 | All optimization level defaults can be achieved by manually setting the underlying flags. |
| 13 | User-set flags take precedence over optimization level defaults. |
| 14 | |
| 15 | ## Level Summaries and Usage Examples |
| 16 | |
| 17 | ```bash |
| 18 | # CLI usage |
| 19 | vllm serve RedHatAI/Llama-3.2-1B-FP8 -O1 |
| 20 | |
| 21 | # Python API usage |
| 22 | from vllm.entrypoints.llm import LLM |
| 23 | |
| 24 | llm = LLM( |
| 25 | model="RedHatAI/Llama-3.2-1B-FP8", |
| 26 | optimization_level=2 # equivalent to -O2 |
| 27 | ) |
| 28 | ``` |
| 29 | |
| 30 | ### `-O0`: No Optimization |
| 31 | |
| 32 | Startup as fast as possible - no autotuning, no compilation, and no cudagraphs. |
| 33 | This level is good for initial phases of development and debugging. |
| 34 | |
| 35 | Settings: |
| 36 | |
| 37 | - `-cc.cudagraph_mode=NONE` |
| 38 | - `-cc.mode=NONE` (also resulting in `-cc.custom_ops=["none"]`) |
| 39 | - `-cc.pass_config.fuse_...=False` (all fusions disabled) |
| 40 | - `--kernel-config.enable_flashinfer_autotune=False` |
| 41 | |
| 42 | ### `-O1`: Fast Optimization |
| 43 | |
| 44 | Prioritize fast startup, but still enable basic optimizations like compilation and cudagraphs. |
| 45 | This level is a good balance for most development scenarios where you want faster startup but |
| 46 | still make sure your code does not break cudagraphs or compilation. |
| 47 | |
| 48 | Settings: |
| 49 | |
| 50 | - `-cc.cudagraph_mode=PIECEWISE` |
| 51 | - `-cc.mode=VLLM_COMPILE` |
| 52 | - `--kernel-config.enable_flashinfer_autotune=True` |
| 53 | |
| 54 | Fusions: |
| 55 | |
| 56 | - `-cc.pass_config.fuse_norm_quant=True`* |
| 57 | - `-cc.pass_config.fuse_act_quant=True`* |
| 58 | - `-cc.pass_config.fuse_act_padding=True`† |
| 59 | - `-cc.pass_config.fuse_mla_dual_rms_norm=True`† |
| 60 | |
| 61 | \* These fusions are only enabled when either op is using a custom kernel, otherwise Inductor fusion is better.</br> |
| 62 | † These fusions are ROCm-only and require AITER. |
| 63 | |
| 64 | ### `-O2`: Full Optimization (Default) |
| 65 | |
| 66 | Prioritize performance at the expense of additional startup time. |
| 67 | This level is recommended for production workloads and is hence the default. |
| 68 | Fusions in this level _may_ take longer due to additional compile ranges. |
| 69 | |
| 70 | Settings (on top of `-O1`): |
| 71 | |
| 72 | - `-cc.cudagraph_mode=FULL_AND_PIECEWISE` |
| 73 | - `-cc.pass_config.fuse_allreduce_rms=True` |
| 74 | - `-cc.pass_config.fuse_rope_kvcache=True`† |
| 75 | |
| 76 | † These fusions are ROCm-only and require AITER. |
| 77 | |
| 78 | ### `-O3`: Aggressive Optimization |
| 79 | |
| 80 | This level is currently the same as `-O2`, but may include additional optimizations |
| 81 | in the future that are more time-consuming or experimental. |
| 82 | |
| 83 | ## Troubleshooting |
| 84 | |
| 85 | ### Common Issues |
| 86 | |
| 87 | 1. **Startup Time Too Long**: Use `-O0` or `-O1` for faster startup |
| 88 | 2. **Compilation Errors**: Use `debug_dump_path` for additional debugging information |
| 89 | 3. **Performance Issues**: Ensure using `-O2` for production |