A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 02:00 UTC
[doc] explain common errors around torch.compile (#12340)
Signed-off-by: youkaichao <youkaichao@gmail.com>
youkaichao committed
Jan 23, 2025 at 14:56 UTC
511627445e819169ded3aaead54062cddfd37a5d
1 file changed
+21
docs/source/getting_started/troubleshooting.md
+21
@@ -197,6 +197,27 @@ if __name__ == '__main__':
197
llm = vllm.LLM(...)
198
```
199
200
+## `torch.compile` Error
201
+
202
+vLLM heavily depends on `torch.compile` to optimize the model for better performance, which introduces the dependency on the `torch.compile` functionality and the `triton` library. By default, we use `torch.compile` to [optimize some functions](https://github.com/vllm-project/vllm/pull/10406) in the model. Before running vLLM, you can check if `torch.compile` is working as expected by running the following script:
203
+
204
+```python
205
+import torch
206
+
207
+@torch.compile
208
+def f(x):
209
+ # a simple function to test torch.compile
210
+ x = x + 1
211
+ x = x * 2
212
+ x = x.sin()
213
+ return x
214
+
215
+x = torch.randn(4, 4).cuda()
216
+print(f(x))
217
+```
218
+
219
+If it raises errors from `torch/_inductor` directory, usually it means you have a custom `triton` library that is not compatible with the version of PyTorch you are using. See [this issue](https://github.com/vllm-project/vllm/issues/12219) for example.
220
+
221
## Known Issues
222
223
- In `v0.5.2`, `v0.5.3`, and `v0.5.3.post1`, there is a bug caused by [zmq](https://github.com/zeromq/pyzmq/issues/2000) , which can occasionally cause vLLM to hang depending on the machine configuration. The solution is to upgrade to the latest version of `vllm` to include the [fix](gh-pr:6759).