@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 361 lines 14.6 KB
Rendered Raw
1 # How to debug the vLLM-torch.compile integration
2
3 TL;DR:
4
5 - use tlparse to acquire torch.compile logs. Include these logs in bug reports and/or support asks.
6 - The vLLM-torch.compile integration is multiple pieces. vLLM exposes flags to turn off each piece:
7
8 | Online Flag | Offline Flag | Result |
9 |--------------------------------|--------------------------------------------------------------------------------|------------------------------------------------------|
10 | --enforce-eager | enforce_eager=True | Turn off torch.compile and CUDAGraphs |
11 | -cc.mode=0 | compilation_config=CompilationConfig(mode=CompilationMode.NONE) | Turn off torch.compile only |
12 | -cc.mode=1 | compilation_config=CompilationConfig(mode=CompilationMode.STOCK_TORCH_COMPILE) | Turn off vLLM-compile modifications to torch.compile |
13 | -cc.cudagraph_mode=NONE | compilation_config=CompilationConfig(cudagraph_mode=CUDAGraphMode.NONE) | Turn off CUDAGraphs only |
14 | -cc.backend=eager | compilation_config=CompilationConfig(backend='eager') | Turn off TorchInductor |
15 | -cc.ir_enable_torch_wrap=False | compilation_config=CompilationConfig(ir_enable_torch_wrap=False) | Turn off vLLM IR wrapping |
16
17 ## vLLM-torch.compile overview
18
19 To improve performance, vLLM leverages torch.compile and CUDAGraphs to speed things up.
20 torch.compile generates optimized kernels for PyTorch code while CUDAGraphs eliminates overhead.
21 Most notably, vLLM-compile is NOT torch.compile, it is a custom compiler built using internal PyTorch Compile APIs.
22
23 ![vLLM-compile diagram](../assets/design/debug_vllm_compile/design_diagram.png)
24
25 - Given a model, we do a full graph capture via TorchDynamo that is dynamic on the batch size (number of tokens)
26 - vLLM then optionally splits and/or specializes this graph and then uses TorchInductor to compile each graph into a compiled artifact.
27 This step may use vLLM custom Inductor passes to further optimize the graph. This includes vLLM IR lowering to remove dispatch overhead.
28 - The compiled artifact is saved to vLLM's compile cache so that it can be loaded in the future.
29 - vLLM applies CUDAGraphs to reduce CPU overheads.
30
31 Things can go wrong in each of the four steps. When something does go wrong, please try to isolate the subsystem
32 that went wrong -- this will allow you to turn off the minimal number of things to keep reliability
33 goals while minimizing impact to performance and also helps us (vLLM) when you open a bug report.
34
35 For more details on the design, please see the following resources:
36
37 - [Introduction to vLLM-torch.compile blogpost](https://blog.vllm.ai/2025/08/20/torch-compile.html)
38 - [vLLM-torch.compile integration design](./torch_compile.md)
39 - [vLLM IR design](./vllm_ir.md)
40 - [vLLM Office Hours #26](https://www.youtube.com/live/xLyxc7hxCJc?si=Xulo9pe53C6ywf0V&t=561)
41 - [Talk at PyTorch Conference 2025](https://youtu.be/1wV1ESbGrVQ?si=s1GqymUfwiwOrDTg&t=725)
42
43 ## Use tlparse
44
45 Use [tlparse](https://github.com/meta-pytorch/tlparse) to view torch.compile
46 logs. These logs show all stages of the compilation process, including the fused
47 kernels that torch.compile produces.
48
49 Install tlparse:
50
51 ```sh
52 pip install tlparse
53 ```
54
55 To enable the torch.compile logs, you can set the envvar `TORCH_TRACE=<dir>`.
56 During tracing, a file per rank will be created inside of that directory, with
57 each file containing the artifacts during compilation. If you can, we recommend
58 sending these log files along with bug reports -- they are very helpful.
59
60 Usage (offline inference)
61
62 ```sh
63 TORCH_TRACE=~/trace_dir python my_script.py
64 tlparse ~/trace_dir/<rank_0_log_file>
65 ```
66
67 Usage (serving)
68
69 ```sh
70 TORCH_TRACE=~/trace_dir vllm serve
71 # ctrl-c out of the server
72 tlparse ~/trace_dir/<rank_0_log_file>
73 ```
74
75 Given one of the log files, the `tlparse` command outputs some HTML files
76 (perhaps into e.g. `./tl_out/index.html`).
77 Open it to see the logs. It'll look something like the following:
78
79 ![tlparse example](../assets/design/debug_vllm_compile/tlparse_inductor.png)
80
81 ## Turn off vLLM-torch.compile integration
82
83 Pass `--enforce-eager` to turn off the vLLM-torch.compile integration and run entirely
84 in eager mode. This includes turning off CUDAGraphs.
85
86 ```sh
87 # Online
88 vllm serve --enforce-eager
89 ```
90
91 ```py
92 # Offline
93 LLM(model, enforce_eager=True)
94 ```
95
96 To turn off just torch.compile, pass `mode = NONE` to the compilation config.
97 (`-cc` is short for `--compilation_config`):
98
99 ```sh
100 # Online
101 vllm serve -cc.mode=0
102 ```
103
104 ```py
105 # Offline
106 from vllm.config.compilation import CompilationConfig, CompilationMode
107 LLM(model, compilation_config=CompilationConfig(mode=CompilationMode.NONE))
108 ```
109
110 To turn off just CUDAGraphs, pass `cudagraph_mode = NONE`:
111
112 ```sh
113 # Online
114 vllm serve -cc.cudagraph_mode=NONE
115 ```
116
117 ```py
118 # Offline
119 from vllm.config.compilation import CompilationConfig, CUDAGraphMode
120 LLM(model, compilation_config=CompilationConfig(cudagraph_mode=CUDAGraphMode.NONE))
121 ```
122
123 vLLM IR makes heavy use of the compilation pipeline, from functionalization, custom fusions, and lowering.
124 To turn that off and capture eager-mode dispatching behavior of vLLM IR, run with `ir_enable_torch_wrap=False`.
125 IR torch wrap is only enabled by default when using `mode=VLLM_COMPILE` and `backend="inductor"` (default).
126
127 ```sh
128 # Online
129 vllm serve -cc.ir_enable_torch_wrap=False
130 ```
131
132 ```py
133 # Offline
134 from vllm.config.compilation import CompilationConfig
135 LLM(model, compilation_config=CompilationConfig(ir_enable_torch_wrap=False))
136 ```
137
138 ## Debugging TorchDynamo
139
140 vLLM requires model code be capturable into a full graph via TorchDynamo (torch.compile's frontend).
141 TorchDynamo does not support all of Python. It will error (in fullgraph mode) if it cannot support
142 a feature (this is sometimes known as a graph break).
143
144 If you encounter a graph break, please [open an issue to pytorch/pytorch](https://github.com/pytorch/pytorch) so the PyTorch devs can prioritize.
145 Then, try your best to rewrite the code to avoid the graph break.
146 For more information, see this [Dynamo guide](https://docs.pytorch.org/docs/stable/compile/programming_model.dynamo_core_concepts.html).
147
148 ## Debugging Dynamic Shape full graph capture
149
150 vLLM requires that the model's forward pass be capturable into a full graph that is dynamic
151 on the batch size (i.e. the number of tokens). It (by default) compiles this one graph into
152 one artifact and uses this artifact for all batch sizes.
153
154 If your code cannot be captured with Dynamic Shapes, you may see silent incorrectness,
155 loud errors, or CUDA illegal memory accesses. For example, the following is not
156 capturable into a single graph:
157
158 ```py
159 if data.size[0] % 128 == 0:
160 foo(...)
161 else:
162 bar(...)
163 ```
164
165 This problem is easy to diagnose. Use tlparse and click on `compilation_metrics`:
166 it will tell you symbolic constraints on the batch size. If there is any constraint
167 that restricts the batch sizes, then we've got a problem.
168
169 ![Bad tlparse example](../assets/design/debug_vllm_compile/dynamic_shapes.png)
170
171 To avoid this, please either:
172
173 1. avoid branching on the number of tokens
174 2. wrap the branching logic into a custom operator. TorchDynamo does not
175 trace into custom operators.
176
177 ## Debugging constraint violations and dynamic shapes guards issues
178
179 Dynamic-shape guards are a specific category of Dynamo guards. They are constraints that `torch.compile`
180 attaches to dynamic dimensions (e.g., `seq_len`) to ensure the compiled artifact remains valid.
181 These guards typically appear when framework code, custom passes, or user code branches based on
182 dynamic shape values.
183
184 **Example:**
185
186 ```python
187 if x > 10:
188 # path A
189 else:
190 # path B
191 ```
192
193 This creates a guard `x > 10` or `x <= 10` depending on which path was traced.
194
195 **vLLM's Assumption:**
196 vLLM assumes that all guards added by torch.compile are safe to drop and will not
197 constrain the compiled graph to specific input shapes. When this assumption is violated,
198 it can cause issues that users need to debug.
199 Some side effects that indicates this assumption is violated are runtime errors
200 or `ConstraintViolationErrors`.
201
202 A `ConstraintViolationErrors` will be thrown if a dynamic shape gets constrained to
203 a single value. If you encounter a constraint violation error or suspect that a dynamic
204 shapes guard is being added incorrectly, you can use stricter dynamic shape modes to
205 help debug the issue:
206
207 ```sh
208 # Online - using unbacked mode
209 vllm serve meta-llama/Llama-3.2-1B -cc.dynamic_shapes_config.type=unbacked
210
211 # Online - using backed_size_oblivious mode
212 vllm serve meta-llama/Llama-3.2-1B -cc.dynamic_shapes_config.type=backed_size_oblivious
213 ```
214
215 ```py
216 # Offline - using unbacked mode
217 from vllm.config.compilation import CompilationConfig, DynamicShapesConfig, DynamicShapesType
218 LLM(model, compilation_config=CompilationConfig(
219 dynamic_shapes_config=DynamicShapesConfig(type=DynamicShapesType.UNBACKED)
220 ))
221
222 # Offline - using backed_size_oblivious mode
223 from vllm.config.compilation import CompilationConfig, DynamicShapesConfig, DynamicShapesType
224 LLM(model, compilation_config=CompilationConfig(
225 dynamic_shapes_config=DynamicShapesConfig(type=DynamicShapesType.BACKED_SIZE_OBLIVIOUS)
226 ))
227 ```
228
229 These modes are stricter and reduce or eliminate the need of dynamic shapes guarding, which can help isolate issues:
230
231 - `unbacked`: Uses unbacked symints which don't allow guards, making it easier to identify where guards are being incorrectly added
232 - `backed_size_oblivious`: Uses a mode that is stricter about guarding.
233
234 For more details on dynamic shapes modes, see [Dynamic shapes and vLLM guard dropping](torch_compile.md#dynamic-shapes-and-vllm-guard-dropping).
235
236 ### Printing guards
237
238 To see all guards that are being added during compilation, you can use `TORCH_LOGS=+dynamic`:
239
240 ```sh
241 TORCH_LOGS=+dynamic vllm serve meta-llama/Llama-3.2-1B
242 ```
243
244 Look for `[guard added]` in the logs to see where guards are being added. This can help you identify which operations are
245 causing guards to be added incorrectly.
246
247 ## Debugging TorchInductor
248
249 TorchInductor takes a captured graph and then compiles it down to some Python code
250 that may call 1+ triton kernels. On rare (but unfortunate) occasions, it may
251 produce an incorrect triton kernel. This may manifest as silent incorrectness,
252 CUDA illegal memory accesses, or loud errors.
253
254 ### Inductor runtime assertions
255
256 By default (on torch < 2.12), vLLM disables Inductor's runtime assertions
257 (`assert_size_stride`, `assert_alignment`) to avoid ~2ms overhead per forward
258 pass on large models. Setting `VLLM_LOGGING_LEVEL=DEBUG` automatically
259 re-enables them so debugging sessions get full shape/stride validation:
260
261 ```sh
262 VLLM_LOGGING_LEVEL=DEBUG vllm serve <model>
263 ```
264
265 You can also override them explicitly via `--compilation-config`:
266
267 ```sh
268 vllm serve <model> -cc.inductor_compile_config='{"size_asserts": true, "alignment_asserts": true, "scalar_asserts": true}'
269 ```
270
271 On torch >= 2.12, PyTorch uses an efficient assert-once strategy and these
272 flags are no longer suppressed by vLLM.
273
274 To debug if TorchInductor is at fault, you can disable it by passing `backend='eager'`
275 to the compilation config:
276
277 ```sh
278 # online
279 vllm serve -cc.backend=eager
280 ```
281
282 ```py
283 # offline
284 LLM(compilation_config=CompilationConfig(backend='eager'))
285 ```
286
287 If Inductor is at fault, [file a bug to PyTorch](https://github.com/pytorch/pytorch).
288 If you're feeling adventurous, you can debug the triton kernels in the Inductor output code
289 (that you can locate via using tlparse).
290
291 ![tlparse example](../assets/design/debug_vllm_compile/tlparse_inductor.png)
292
293 You can also use `TORCH_LOGS=output_code <command>` to print the Inductor output code.
294
295 ### Editable TorchInductor code
296
297 You can edit the TorchInductor code that gets run by setting `VLLM_COMPILE_CACHE_SAVE_FORMAT=unpacked`
298 or passing `-cc.compile_cache_save_format=unpacked`. The default is `binary`, which means it is not editable.
299
300 This is a useful technique: you can put breakpoints (e.g. `torch.distributed.breakpoint()`)
301 and print statements in the output code.
302
303 ## Debugging vLLM-compile cache
304
305 vLLM built its own cache for torch.compile artifacts. The idea is that the artifacts
306 can be compiled once and then reused after they have been compiled. This
307 is a layer on top of [torch.compile's compiler cache](https://docs.pytorch.org/tutorials/recipes/torch_compile_caching_tutorial.html).
308
309 While torch.compile's compiler cache is rock-stable, vLLM's compiler cache is unfortunately
310 not always correct. You can disable it via setting `VLLM_DISABLE_COMPILE_CACHE=1`.
311
312 You can also manually remove this cache.
313
314 - Remove vLLM's compile cache with `rm -rf ~/.cache/vllm` (look at logs to see if the location changed)
315 - Remove torch.compile's built-in caches with `rm -rf /tmp/torchinductor_$(whoami)`
316
317 vLLM's cache is a mapping from cache key to a compiled artifact. vLLM computes
318 the cache key via combining multiple factors (e.g. config flags and model name).
319 If vLLM's compile cache is wrong, this usually means that a factor is missing.
320 Please see [this example](https://github.com/vllm-project/vllm/blob/18b39828d90413d05d770dfd2e2f48304f4ca0eb/vllm/config/model.py#L310)
321 of how vLLM computes part of the cache key.
322
323 vLLM's compilation cache requires that the code being compiled ends up being serializable.
324 If this is not the case, then it will error out on save. Usually the fixes are to either:
325
326 - rewrite the non-serializable pieces (perhaps difficult because it's difficult to
327 tell right now what is serializable and what isn't)
328 - file a bug report
329 - ignore the error by setting `VLLM_DISABLE_COMPILE_CACHE=1` (note that this will
330 make warm server starts a lot slower).
331
332 ## Debugging CUDAGraphs
333
334 CUDAGraphs is a feature that allows one to:
335
336 - Capture a callable that launches 1+ CUDA kernels into a CUDAGraph
337 - Replay the CUDAGraph
338
339 The captured CUDAGraph contains all of the memory used during the capture process.
340 The replay of the CUDAGraph reads and writes to exactly the same regions of memory.
341
342 This leads to some restrictions:
343
344 1. In order to use CUDAGraphs on new data, you'll need to copy the data into a buffer
345 that the CUDAGraph is reading from
346 2. CUDAGraphs only capture CUDA kernels, they don't capture work done on CPU.
347
348 vLLM uses the raw CUDAGraphs API, which is unsafe when used incorrectly.
349
350 To turn off just CUDAGraphs, pass `cudagraph_mode = NONE`:
351
352 ```sh
353 # Online
354 vllm serve -cc.cudagraph_mode=NONE
355 ```
356
357 ```py
358 # Offline
359 from vllm.config.compilation import CompilationConfig, CUDAGraphMode
360 LLM(model, compilation_config=CompilationConfig(cudagraph_mode=CUDAGraphMode.NONE))
361 ```