@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 9f4ccec761
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 20:45 UTC

[doc][misc] remind to cancel debugging environment variables (#6481)

[doc][misc] remind users to cancel debugging environment variables after debugging (#6481)

youkaichao committed Jul 16, 2024 at 09:45 UTC 9f4ccec76135083c96d15fbbade5eda7a2321bf1
1 file changed +4 -3
docs/source/getting_started/debugging.rst
+4 -3
@@ -19,9 +19,6 @@ If you have already taken care of the above issues, but the vLLM instance still
19 - Set the environment variable ``export NCCL_DEBUG=TRACE`` to turn on more logging for NCCL.
20 - Set the environment variable ``export VLLM_TRACE_FUNCTION=1``. All the function calls in vLLM will be recorded. Inspect these log files, and tell which function crashes or hangs.
21
22 - .. warning::
23 - vLLM function tracing will generate a lot of logs and slow down the system. Only use it for debugging purposes.
24 -
22 With more logging, hopefully you can find the root cause of the issue.
23
24 If it crashes, and the error trace shows somewhere around ``self.graph.replay()`` in ``vllm/worker/model_runner.py``, it is a cuda error inside cudagraph. To know the particular cuda operation that causes the error, you can add ``--enforce-eager`` to the command line, or ``enforce_eager=True`` to the ``LLM`` class, to disable the cudagraph optimization. This way, you can locate the exact cuda operation that causes the error.
@@ -67,3 +64,7 @@ Here are some common issues that can cause hangs:
64 If the script runs successfully, you should see the message ``sanity check is successful!``.
65
66 If the problem persists, feel free to `open an issue on GitHub <https://github.com/vllm-project/vllm/issues/new/choose>`_, with a detailed description of the issue, your environment, and the logs.
67 +
68 +.. warning::
69 +
70 + After you find the root cause and solve the issue, remember to turn off all the debugging environment variables defined above, or simply start a new shell to avoid being affected by the debugging settings. If you don't do this, the system might be slow because many debugging functionalities are turned on.