A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 9, 01:15 UTC
[torch.compile] Document the workaround to standalone_compile failing (#33571)
Signed-off-by: Richard Zou <zou3519@gmail.com>
Richard Zou committed
Feb 2, 2026 at 23:16 UTC
fd9c83d0e05e6a4214be7dabf3b2bd64a9696ed8
2 files changed
+28
docs/design/debug_vllm_compile.md
+9
@@ -282,6 +282,15 @@ If vLLM's compile cache is wrong, this usually means that a factor is missing.
282
Please see [this example](https://github.com/vllm-project/vllm/blob/18b39828d90413d05d770dfd2e2f48304f4ca0eb/vllm/config/model.py#L310)
283
of how vLLM computes part of the cache key.
284
285
+vLLM's compilation cache requires that the code being compiled ends up being serializable.
286
+If this is not the case, then it will error out on save. Usually the fixes are to either:
287
+
288
+- rewrite the non-serializable pieces (perhaps difficult because it's difficult to
289
+ tell right now what is serializable and what isn't)
290
+- file a bug report
291
+- ignore the error by setting `VLLM_DISABLE_COMPILE_CACHE=1` (note that this will
292
+ make warm server starts a lot slower).
293
+
294
## Debugging CUDAGraphs
295
296
CUDAGraphs is a feature that allows one to:
vllm/compilation/compiler_interface.py
+19
@@ -273,7 +273,26 @@ class InductorStandaloneAdaptor(CompilerInterface):
273
assert key is not None
274
path = os.path.join(self.cache_dir, key)
275
276
+ def is_saveable_2_10(compiled_artifact):
277
+ # can just use compiled_artifact.is_saveable in 2.11
278
+ if compiled_artifact._artifacts is None:
279
+ return False
280
+ _, cache_info = compiled_artifact._artifacts
281
+ return len(cache_info.aot_autograd_artifacts) == 1
282
+
283
if is_compile_cache_enabled(compiler_config):
284
+ if not is_saveable_2_10(compiled_graph):
285
+ raise RuntimeError(
286
+ "The compiled artifact is not serializable. This usually means "
287
+ "that the model code has something that is not serializable "
288
+ "by torch.compile in it. You can fix this by either "
289
+ "figuring out what is not serializable and rewriting it, "
290
+ "filing a bug report, "
291
+ "or suppressing this error by "
292
+ "disabling vLLM's compilation cache via "
293
+ "VLLM_DISABLE_COMPILE_CACHE=1 "
294
+ "(this will greatly increase vLLM server warm start times)."
295
+ )
296
compiled_graph.save(path=path, format=self.save_format)
297
compilation_counter.num_compiled_artifacts_saved += 1
298
return compiled_graph, (key, path)