@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 511627445e
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 02:00 UTC

[doc] explain common errors around torch.compile (#12340)

Signed-off-by: youkaichao <youkaichao@gmail.com>

youkaichao committed Jan 23, 2025 at 14:56 UTC 511627445e819169ded3aaead54062cddfd37a5d
1 file changed +21
docs/source/getting_started/troubleshooting.md
+21
@@ -197,6 +197,27 @@ if __name__ == '__main__':
197 llm = vllm.LLM(...)
198 ```
199
200 +## `torch.compile` Error
201 +
202 +vLLM heavily depends on `torch.compile` to optimize the model for better performance, which introduces the dependency on the `torch.compile` functionality and the `triton` library. By default, we use `torch.compile` to [optimize some functions](https://github.com/vllm-project/vllm/pull/10406) in the model. Before running vLLM, you can check if `torch.compile` is working as expected by running the following script:
203 +
204 +```python
205 +import torch
206 +
207 +@torch.compile
208 +def f(x):
209 + # a simple function to test torch.compile
210 + x = x + 1
211 + x = x * 2
212 + x = x.sin()
213 + return x
214 +
215 +x = torch.randn(4, 4).cuda()
216 +print(f(x))
217 +```
218 +
219 +If it raises errors from `torch/_inductor` directory, usually it means you have a custom `triton` library that is not compatible with the version of PyTorch you are using. See [this issue](https://github.com/vllm-project/vllm/issues/12219) for example.
220 +
221 ## Known Issues
222
223 - In `v0.5.2`, `v0.5.3`, and `v0.5.3.post1`, there is a bug caused by [zmq](https://github.com/zeromq/pyzmq/issues/2000) , which can occasionally cause vLLM to hang depending on the machine configuration. The solution is to upgrade to the latest version of `vllm` to include the [fix](gh-pr:6759).