@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 107 lines 4.96 KB
Rendered Raw
1 # torch.compile with Multimodal Encoders
2
3 `torch.compile` can now be applied to multimodal encoders and miscellaneous nn modules in vLLM, including vision-language models like LLaMA 4, Qwen-VL,
4 and similar encoder-based architectures.
5
6 This document covers the basics of how the `torch.compile` integration works for multimodal encoders in vLLM, as well as how to apply the decorator
7 to new models to improve performance.
8
9 !!! note
10 For general information about `torch.compile` integration in vLLM, see the [torch.compile design document](./torch_compile.md).
11
12 ## Overview
13
14 We have recently enabled the `@support_torch_compile` decorator to work for multiple nn module components within a model type; this enables
15 turning compile on for multimodal encoders, bringing performance improvements to additional components of the stack.
16
17 When applied to the vision block of [`Qwen2_5_vl`](https://github.com/vllm-project/vllm/pull/23207) we observe ~4.5% e2e perf improvements with
18 some increase in compilation time
19
20 This feature is off by default, but can be enabled by setting `compile_mm_encoder: true` in the compilation config when models have the
21 `@support_torch_compile` decorator.
22
23 ## How Compilation Works for Multimodal Components
24
25 ### APIs for Enablement
26
27 To compile a multimodal component such as an encoder, we follow the same mechanism as the LLM text backbone, with a few additional scaffoldings:
28
29 1. The `@support_torch_compile` decorator should include `enable_if=should_torch_compile_mm_encoder`. This will gate the compilation behind our
30 `compile_mm_encoder` configuration
31
32 2. The `@support_torch_compile` decorator should include `is_encoder=True` for encoder components. This is needed for compile range integration
33 (see Compile Range Integration). The decorator automatically uses the class name as the cache directory prefix, avoiding collisions between
34 independently compiled sub-modules (e.g. vision encoder components vs the text backbone).
35
36 ### CompilationConfig
37
38 With the exception of `compile_mm_encoder: true`, the multimodal encoder will inherit from the same compilation config as the text LLM. We may extend
39 this for more configuration in the future.
40
41 ## Applying torch.compile to a New Multimodal Model/Component
42
43 To apply `support_torch_compile` to a new general nn.Module, we advise following the same steps in [`debug_vllm_compile`](./debug_vllm_compile.md); this includes:
44
45 1. Applying `support_torch_compile` on initially small modules (such as basic MLP layers), then raising to more general modules until one reaches a good performance
46 tradeoff
47
48 2. Leveraging [`tlparse`](https://github.com/meta-pytorch/tlparse) to identify and eliminate the source of recompiles and graph breaks
49
50 3. Using `dynamic_arg_dims` and proper `dynamic_shapes_config` to handle dynamism.
51
52 ### Common pitfalls
53
54 ## VllmBackend Feature Support
55
56 ### Compile ranges
57
58 The torch.compile integration will try to rely on max_batch_size to infer compilation ranges for dynamic shapes; however, for modules used in the encoder, this
59 shape can be difficult to infer due to the unspecified range of shapes the encoder may see as input. Therefore, we rely on `is_encoder=True` in the
60 `@support_torch_compile` decorator to alert torch.compile to the fact that this range cannot be inferred, and we default to the range (1, MAX_INT).
61
62 !!! note
63 We may seek to tighten this range for better performance in the future
64
65 ### Cudagraphs
66
67 We have not yet explored compilation for multimodal encoders with CUDAGraph integration; behavior is currently unspecified.
68
69 ## Troubleshooting
70
71 ### Graph Breaks in Vision Encoders
72
73 Some vision encoder operations may cause graph breaks. To identify them:
74
75 ```bash
76 TORCH_LOGS="+dynamo" vllm serve <MODEL>
77 ```
78
79 Common causes of graph breaks in multimodal models:
80
81 - **Dynamic image sizes**: Use `dynamic_shapes_config` to handle variable resolutions
82 - **Untraceable operations**: Some operations (such as to_list) may not be supported by Dynamo
83 - **Conditional processing**: Data-dependent branching based on image properties
84
85 ### Compilation Errors
86
87 If compilation fails for a multimodal model:
88
89 1. **Disable and test**: First verify the model works without compilation:
90 ```bash
91 vllm serve <model> --compilation-config='{"mode":0,"compile_mm_encoder":"false"}'
92 ```
93
94 2. **Check logs**: Enable debug logging to see compilation details:
95 ```bash
96 VLLM_LOGGING_LEVEL=DEBUG vllm serve <model> --compilation-config='{"compile_mm_encoder":"true"}'
97 ```
98
99 3. **Report issues**: If you find a bug, [open an issue on GitHub](https://github.com/vllm-project/vllm/issues/new/choose)
100
101 ## See Also
102
103 - [torch.compile Integration](./torch_compile.md) - Core design document
104 - [Debugging torch.compile](./debug_vllm_compile.md) - Detailed debugging guide
105 - [Multimodal Inputs](../features/multimodal_inputs.md) - How to pass multimodal data
106 - [Disaggregated Encoder](../features/disagg_encoder.md) - Scaling vision encoders
107 - [Supported Multimodal Models](../models/supported_models.md#list-of-multimodal-language-models) - Model compatibility