A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 23:15 UTC
[Doc] add debugging tips for crash and multi-node debugging (#5581)
youkaichao committed
Jun 16, 2024 at 19:08 UTC
845a3f26f9706acafe8fa45ae452846d8cc3b97f
1 file changed
+16
-3
docs/source/getting_started/debugging.rst
+16
-3
@@ -24,6 +24,8 @@ If you have already taken care of the above issues, but the vLLM instance still
24
25
With more logging, hopefully you can find the root cause of the issue.
26
27
+If it crashes, and the error trace shows somewhere around ``self.graph.replay()`` in ``vllm/worker/model_runner.py``, it is a cuda error inside cudagraph. To know the particular cuda operation that causes the error, you can add ``--enforce-eager`` to the command line, or ``enforce_eager=True`` to the ``LLM`` class, to disable the cudagraph optimization. This way, you can locate the exact cuda operation that causes the error.
28
+
29
Here are some common issues that can cause hangs:
30
31
- **Incorrect network setup**: The vLLM instance cannot get the correct IP address. You can find the log such as ``DEBUG 06-10 21:32:17 parallel_state.py:88] world_size=8 rank=0 local_rank=0 distributed_init_method=tcp://xxx.xxx.xxx.xxx:54641 backend=nccl``. The IP address should be the correct one. If not, override the IP address by setting the environment variable ``export VLLM_HOST_IP=your_ip_address``.
@@ -31,15 +33,26 @@ Here are some common issues that can cause hangs:
33
34
.. code-block:: python
35
34
- # save it as `test.py` , and run it with `NCCL_DEBUG=TRACE torchrun --nproc-per-node=8 test.py`
35
- # adjust `--nproc-per-node` to the number of GPUs you want to use.
36
import torch
37
import torch.distributed as dist
38
dist.init_process_group(backend="nccl")
39
- data = torch.FloatTensor([1,] * 128).to(f"cuda:{dist.get_rank()}")
39
+ local_rank = dist.get_rank() % torch.cuda.device_count()
40
+ data = torch.FloatTensor([1,] * 128).to(f"cuda:{local_rank}")
41
dist.all_reduce(data, op=dist.ReduceOp.SUM)
42
torch.cuda.synchronize()
43
value = data.mean().item()
44
assert value == dist.get_world_size()
45
46
+.. tip::
47
+
48
+ Save the script as ``test.py``.
49
+
50
+ If you are testing in a single-node, run it with ``NCCL_DEBUG=TRACE torchrun --nproc-per-node=8 test.py``, adjust ``--nproc-per-node`` to the number of GPUs you want to use.
51
+
52
+ If you are testing with multi-nodes, run it with ``NCCL_DEBUG=TRACE torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=$MASTER_ADDR test.py``. Adjust ``--nproc-per-node`` and ``--nnodes`` according to your setup. Make sure ``MASTER_ADDR``:
53
+
54
+ - is the correct IP address of the master node
55
+ - is reachable from all nodes
56
+ - is set before running the script.
57
+
58
If the problem persists, feel free to `open an issue on GitHub <https://github.com/vllm-project/vllm/issues/new/choose>`_, with a detailed description of the issue, your environment, and the logs.