@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / d1c6799b88
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 08:15 UTC

[doc] update debugging guide (#10236)

Signed-off-by: youkaichao <youkaichao@gmail.com>

youkaichao committed Nov 11, 2024 at 15:21 UTC d1c6799b8870e513bf4f2305cbf6cda9fc3d773b
1 file changed +2
docs/source/getting_started/debugging.rst
+2
@@ -122,6 +122,8 @@ If you are testing with multi-nodes, adjust ``--nproc-per-node`` and ``--nnodes`
122
123 If the script runs successfully, you should see the message ``sanity check is successful!``.
124
125 +If the test script hangs or crashes, usually it means the hardware/drivers are broken in some sense. You should try to contact your system administrator or hardware vendor for further assistance. As a common workaround, you can try to tune some NCCL environment variables, such as ``export NCCL_P2P_DISABLE=1`` to see if it helps. Please check `their documentation <https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html>`__ for more information. Please only use these environment variables as a temporary workaround, as they might affect the performance of the system. The best solution is still to fix the hardware/drivers so that the test script can run successfully.
126 +
127 .. note::
128
129 A multi-node environment is more complicated than a single-node one. If you see errors such as ``torch.distributed.DistNetworkError``, it is likely that the network/DNS setup is incorrect. In that case, you can manually assign node rank and specify the IP via command line arguments: