@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 2759a43a26
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 17:00 UTC

[doc] update doc on testing and debugging (#8514)

youkaichao committed Sep 16, 2024 at 12:10 UTC 2759a43a26e4eecb7ff7d741c2b6da0d544462ad
1 file changed +7
docs/source/getting_started/debugging.rst
+7
@@ -98,6 +98,13 @@ Here are some common issues that can cause hangs:
98
99 If the script runs successfully, you should see the message ``sanity check is successful!``.
100
101 + Note that multi-node environment is more complicated than single-node. If you see errors such as ``torch.distributed.DistNetworkError``, it is likely that the network/DNS setup is incorrect. In that case, you can manually assign node rank and specify the IP via command line arguments:
102 +
103 + - In the first node, run ``NCCL_DEBUG=TRACE torchrun --nnodes 2 --nproc-per-node=2 --node-rank 0 --master_addr $MASTER_ADDR test.py``.
104 + - In the second node, run ``NCCL_DEBUG=TRACE torchrun --nnodes 2 --nproc-per-node=2 --node-rank 1 --master_addr $MASTER_ADDR test.py``.
105 +
106 + Adjust ``--nproc-per-node``, ``--nnodes``, and ``--node-rank`` according to your setup. The difference is that you need to execute different commands (with different ``--node-rank``) on different nodes.
107 +
108 If the problem persists, feel free to `open an issue on GitHub <https://github.com/vllm-project/vllm/issues/new/choose>`_, with a detailed description of the issue, your environment, and the logs.
109
110 Some known issues: