A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 23:15 UTC
[Doc] add debugging tips (#5409)
youkaichao committed
Jun 10, 2024 at 23:21 UTC
d8f31f2f8b3486c6767a8db29d881fa8cda620fa
2 files changed
+37
docs/source/getting_started/debugging.rst
new
+36
@@ -0,0 +1,36 @@
1
+.. _debugging:
2
+
3
+Debugging Tips
4
+===============
5
+
6
+Debugging hang/crash issues
7
+---------------------------
8
+
9
+When an vLLM instance hangs or crashes, it is very difficult to debug the issue. Here are some tips to help debug the issue:
10
+
11
+- Set the environment variable ``export VLLM_LOGGING_LEVEL=DEBUG`` to turn on more logging.
12
+- Set the environment variable ``export CUDA_LAUNCH_BLOCKING=1`` to know exactly which CUDA kernel is causing the trouble.
13
+- Set the environment variable ``export NCCL_DEBUG=TRACE`` to turn on more logging for NCCL.
14
+- Set the environment variable ``export VLLM_TRACE_FUNCTION=1`` . All the function calls in vLLM will be recorded. Inspect these log files, and tell which function crashes or hangs. **Note: it will generate a lot of logs and slow down the system. Only use it for debugging purposes.**
15
+
16
+With more logging, hopefully you can find the root cause of the issue.
17
+
18
+Here are some common issues that can cause hangs:
19
+
20
+- The network setup is incorrect. The vLLM instance cannot get the correct IP address. You can find the log such as ``DEBUG 06-10 21:32:17 parallel_state.py:88] world_size=8 rank=0 local_rank=0 distributed_init_method=tcp://xxx.xxx.xxx.xxx:54641 backend=nccl``. The IP address should be the correct one. If not, override the IP address by setting the environment variable ``export VLLM_HOST_IP=your_ip_address``.
21
+- Hardware/driver setup is incorrect. GPU communication cannot be established. You can run a sanity check script below to see if the GPU communication is working correctly.
22
+
23
+.. code-block:: python
24
+
25
+ # save it as `test.py`` , and run it with `NCCL_DEBUG=TRACE torchrun --nproc-per-node=8 test.py`
26
+ # adjust `--nproc-per-node` to the number of GPUs you want to use.
27
+ import torch
28
+ import torch.distributed as dist
29
+ dist.init_process_group(backend="nccl")
30
+ data = torch.FloatTensor([1,] * 128).to(f"cuda:{dist.get_rank()}")
31
+ dist.all_reduce(data, op=dist.ReduceOp.SUM)
32
+ torch.cuda.synchronize()
33
+ value = data.mean().item()
34
+ assert value == dist.get_world_size()
35
+
36
+If the problem persists, feel free to open an `issue <https://github.com/vllm-project/vllm/issues/new/choose>`_ on GitHub, with a detailed description of the issue, your environment, and the logs.
docs/source/index.rst
+1
@@ -66,6 +66,7 @@ Documentation
66
getting_started/neuron-installation
67
getting_started/cpu-installation
68
getting_started/quickstart
69
+ getting_started/debugging
70
getting_started/examples/examples_index
71
72
.. toctree::