@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 376 lines 20.5 KB
Rendered Raw
1 # Troubleshooting
2
3 This document outlines some troubleshooting strategies you can consider. If you think you've discovered a bug, please [search existing issues](https://github.com/vllm-project/vllm/issues?q=is%3Aissue) first to see if it has already been reported. If not, please [file a new issue](https://github.com/vllm-project/vllm/issues/new/choose), providing as much relevant information as possible.
4
5 !!! note
6 Once you've debugged a problem, remember to turn off any debugging environment variables defined, or simply start a new shell to avoid being affected by lingering debugging settings. Otherwise, the system might be slow with debugging functionalities left activated.
7
8 ## Hangs downloading a model
9
10 If the model isn't already downloaded to disk, vLLM will download it from the internet which can take time and depend on your internet connection.
11 It's recommended to download the model first using the [`hf` CLI](https://huggingface.co/docs/huggingface_hub/en/guides/cli) and passing the local path to the model to vLLM. This way, you can isolate the issue.
12
13 ## Hangs loading a model from disk
14
15 If the model is large, it can take a long time to load it from disk. Pay attention to where you store the model. Some clusters have shared filesystems across nodes, e.g. a distributed filesystem or a network filesystem, which can be slow.
16 It'd be better to store the model in a local disk. Additionally, have a look at the CPU memory usage, when the model is too large it might take a lot of CPU memory, slowing down the operating system because it needs to frequently swap between disk and memory.
17
18 !!! note
19 To isolate the model downloading and loading issue, you can use the `--load-format dummy` argument to skip loading the model weights. This way, you can check if the model downloading and loading is the bottleneck.
20
21 ## Out of memory
22
23 If the model is too large to fit in a single GPU, you will get an out-of-memory (OOM) error. Consider adopting [these options](../configuration/conserving_memory.md) to reduce the memory consumption.
24
25 ## Generation quality changed
26
27 In v0.8.0, the source of default sampling parameters was changed in <https://github.com/vllm-project/vllm/pull/12622>. Prior to v0.8.0, the default sampling parameters came from vLLM's set of neutral defaults. From v0.8.0 onwards, the default sampling parameters come from the `generation_config.json` provided by the model creator.
28
29 In most cases, this should lead to higher quality responses, because the model creator is likely to know which sampling parameters are best for their model. However, in some cases the defaults provided by the model creator can lead to degraded performance.
30
31 You can check if this is happening by trying the old defaults with `--generation-config vllm` for online and `generation_config="vllm"` for offline. If, after trying this, your generation quality improves we would recommend continuing to use the vLLM defaults and petition the model creator on <https://huggingface.co> to update their default `generation_config.json` so that it produces better quality generations.
32
33 ## Enable more logging
34
35 If other strategies don't solve the problem, it's likely that the vLLM instance is stuck somewhere. You can use the following environment variables to help debug the issue:
36
37 - `export VLLM_LOGGING_LEVEL=DEBUG` to turn on more logging.
38 - `export VLLM_LOG_STATS_INTERVAL=1.` to get log statistics more frequently for tracking running queue, waiting queue and cache hit states.
39 - `export CUDA_LAUNCH_BLOCKING=1` to identify which CUDA kernel is causing the problem.
40 - `export NCCL_DEBUG=TRACE` to turn on more logging for NCCL.
41 - `export VLLM_TRACE_FUNCTION=1` to record all function calls for inspection in the log files to tell which function crashes or hangs. (WARNING: This flag will slow down the token generation by **over 100x**. Do not use unless absolutely needed.)
42
43 ## Breakpoints
44
45 Setting normal `pdb` breakpoints may not work in vLLM's codebase if they are executed in a subprocess. You will experience something like:
46
47 ``` text
48 File "/usr/local/uv/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/bdb.py", line 100, in trace_dispatch
49 return self.dispatch_line(frame)
50 ^^^^^^^^^^^^^^^^^^^^^^^^^
51 File "/usr/local/uv/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/bdb.py", line 125, in dispatch_line
52 if self.quitting: raise BdbQuit
53 ^^^^^^^^^^^^^
54 bdb.BdbQuit
55 ```
56
57 One solution is using [forked-pdb](https://github.com/Lightning-AI/forked-pdb). Install with `pip install fpdb` and set a breakpoint with something like:
58
59 ``` python
60 __import__('fpdb').ForkedPdb().set_trace()
61 ```
62
63 Another option is to disable multiprocessing entirely, with the `VLLM_ENABLE_V1_MULTIPROCESSING` environment variable.
64 This keeps the scheduler in the same process, so you can use stock `pdb` breakpoints:
65
66 ``` python
67 import os
68 os.environ["VLLM_ENABLE_V1_MULTIPROCESSING"] = "0"
69 ```
70
71 ## Incorrect network setup
72
73 The vLLM instance cannot get the correct IP address if you have a complicated network config. You can find a log such as `DEBUG 06-10 21:32:17 parallel_state.py:88] world_size=8 rank=0 local_rank=0 distributed_init_method=tcp://xxx.xxx.xxx.xxx:54641 backend=nccl` and the IP address should be the correct one.
74 If it's not, override the IP address using the environment variable `export VLLM_HOST_IP=<your_ip_address>`.
75
76 You might also need to set `export NCCL_SOCKET_IFNAME=<your_network_interface>` and `export GLOO_SOCKET_IFNAME=<your_network_interface>` to specify the network interface for the IP address.
77
78 ## Error near `self.graph.replay()`
79
80 If vLLM crashes and the error trace captures it somewhere around `self.graph.replay()` in `vllm/worker/model_runner.py`, it is a CUDA error inside CUDAGraph.
81 To identify the particular CUDA operation that causes the error, you can add `--enforce-eager` to the command line, or `enforce_eager=True` to the [LLM][vllm.LLM] class to disable the CUDAGraph optimization and isolate the exact CUDA operation that causes the error.
82
83 ## Incorrect hardware/driver
84
85 If GPU/CPU communication cannot be established, you can use the following Python script and follow the instructions below to confirm whether the GPU/CPU communication is working correctly.
86
87 ??? code
88
89 ```python
90 # Test PyTorch NCCL
91 import torch
92 import torch.distributed as dist
93 dist.init_process_group(backend="nccl")
94 local_rank = dist.get_rank() % torch.accelerator.device_count()
95 torch.accelerator.set_device_index(local_rank)
96 data = torch.FloatTensor([1,] * 128).to("cuda")
97 dist.all_reduce(data, op=dist.ReduceOp.SUM)
98 torch.accelerator.synchronize()
99 value = data.mean().item()
100 world_size = dist.get_world_size()
101 assert value == world_size, f"Expected {world_size}, got {value}"
102
103 print("PyTorch NCCL is successful!")
104
105 # Test PyTorch GLOO
106 gloo_group = dist.new_group(ranks=list(range(world_size)), backend="gloo")
107 cpu_data = torch.FloatTensor([1,] * 128)
108 dist.all_reduce(cpu_data, op=dist.ReduceOp.SUM, group=gloo_group)
109 value = cpu_data.mean().item()
110 assert value == world_size, f"Expected {world_size}, got {value}"
111
112 print("PyTorch GLOO is successful!")
113
114 if world_size <= 1:
115 exit()
116
117 # Test vLLM NCCL, with cuda graph
118 from vllm.distributed.device_communicators.pynccl import PyNcclCommunicator
119
120 pynccl = PyNcclCommunicator(group=gloo_group, device=local_rank)
121 # pynccl is enabled by default for 0.6.5+,
122 # but for 0.6.4 and below, we need to enable it manually.
123 # keep the code for backward compatibility when because people
124 # prefer to read the latest documentation.
125 pynccl.disabled = False
126
127 s = torch.cuda.Stream()
128 with torch.cuda.stream(s):
129 data.fill_(1)
130 out = pynccl.all_reduce(data, stream=s)
131 value = out.mean().item()
132 assert value == world_size, f"Expected {world_size}, got {value}"
133
134 print("vLLM NCCL is successful!")
135
136 g = torch.cuda.CUDAGraph()
137 with torch.cuda.graph(cuda_graph=g, stream=s):
138 out = pynccl.all_reduce(data, stream=torch.cuda.current_stream())
139
140 data.fill_(1)
141 g.replay()
142 torch.cuda.current_stream().synchronize()
143 value = out.mean().item()
144 assert value == world_size, f"Expected {world_size}, got {value}"
145
146 print("vLLM NCCL with cuda graph is successful!")
147
148 dist.destroy_process_group(gloo_group)
149 dist.destroy_process_group()
150 ```
151
152 If you are testing with a single node, adjust `--nproc-per-node` to the number of GPUs you want to use:
153
154 ```bash
155 NCCL_DEBUG=TRACE torchrun --nproc-per-node=<number-of-GPUs> test.py
156 ```
157
158 If you are testing with multi-nodes, adjust `--nproc-per-node` and `--nnodes` according to your setup and set `MASTER_ADDR` to the correct IP address and port of the master node (e.g., `10.0.0.1:29400`), reachable from all nodes. Then, run:
159
160 ```bash
161 NCCL_DEBUG=TRACE torchrun --nnodes 2 \
162 --nproc-per-node=2 \
163 --rdzv_backend=static \
164 --rdzv_endpoint=$MASTER_ADDR \
165 --node-rank $NODE_RANK test.py
166 ```
167
168 Set `MASTER_ADDR` to the IP address and port of the master node (e.g., `10.0.0.1:29400`), reachable from all nodes. Set `NODE_RANK` to `0` on the master node and `1`, `2`, ... on the workers. Adjust `--nproc-per-node` and `--nnodes` according to your setup.
169
170 !!! note
171 We use `--rdzv_backend=static` instead of `c10d` because the `c10d` rendezvous backend can fail with DNS resolution errors in multi-node setups (see [pytorch/pytorch#85300](https://github.com/pytorch/pytorch/issues/85300)). The `static` backend avoids this by requiring explicit node ranks.
172
173 If the script runs successfully, you should see the message `sanity check is successful!`.
174
175 If the test script hangs or crashes, usually it means the hardware/drivers are broken in some sense. You should try to contact your system administrator or hardware vendor for further assistance. As a common workaround, you can try to tune some NCCL environment variables, such as `export NCCL_P2P_DISABLE=1` to see if it helps. Please check [their documentation](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html) for more information. Please only use these environment variables as a temporary workaround, as they might affect the performance of the system. The best solution is still to fix the hardware/drivers so that the test script can run successfully.
176
177 ## Python multiprocessing
178
179 ### `RuntimeError` Exception
180
181 If you have seen a warning in your logs like this:
182
183 ```console
184 WARNING 12-11 14:50:37 multiproc_worker_utils.py:281] CUDA was previously
185 initialized. We must use the `spawn` multiprocessing start method. Setting
186 VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See
187 https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing
188 for more information.
189 ```
190
191 or an error from Python that looks like this:
192
193 ??? console "Logs"
194
195 ```console
196 RuntimeError:
197 An attempt has been made to start a new process before the
198 current process has finished its bootstrapping phase.
199
200 This probably means that you are not using fork to start your
201 child processes and you have forgotten to use the proper idiom
202 in the main module:
203
204 if __name__ == '__main__':
205 freeze_support()
206 ...
207
208 The "freeze_support()" line can be omitted if the program
209 is not going to be frozen to produce an executable.
210
211 To fix this issue, refer to the "Safe importing of main module"
212 section in https://docs.python.org/3/library/multiprocessing.html
213 ```
214
215 then you must update your Python code to guard usage of `vllm` behind a `if
216 __name__ == '__main__':` block. For example, instead of this:
217
218 ```python
219 import vllm
220
221 llm = vllm.LLM(...)
222 ```
223
224 try this instead:
225
226 ```python
227 if __name__ == '__main__':
228 import vllm
229
230 llm = vllm.LLM(...)
231 ```
232
233 ## `torch.compile` Error
234
235 vLLM heavily depends on `torch.compile` to optimize the model for better performance, which introduces the dependency on the `torch.compile` functionality and the `triton` library. By default, we use `torch.compile` to [optimize some functions](https://github.com/vllm-project/vllm/pull/10406) in the model. Before running vLLM, you can check if `torch.compile` is working as expected by running the following script:
236
237 ??? code
238
239 ```python
240 import torch
241
242 @torch.compile
243 def f(x):
244 # a simple function to test torch.compile
245 x = x + 1
246 x = x * 2
247 x = x.sin()
248 return x
249
250 x = torch.randn(4, 4).cuda()
251 print(f(x))
252 ```
253
254 If it raises errors from `torch/_inductor` directory, usually it means you have a custom `triton` library that is not compatible with the version of PyTorch you are using. See <https://github.com/vllm-project/vllm/issues/12219> for example.
255
256 ## Model failed to be inspected
257
258 If you see an error like:
259
260 ```text
261 File "vllm/model_executor/models/registry.py", line xxx, in _raise_for_unsupported
262 raise ValueError(
263 ValueError: Model architectures ['<arch>'] failed to be inspected. Please check the logs for more details.
264 ```
265
266 It means that vLLM failed to import the model file.
267 Usually, it is related to missing dependencies or outdated binaries in the vLLM build.
268 Please read the logs carefully to determine the root cause of the error.
269
270 ## Model not supported
271
272 If you see an error like:
273
274 ```text
275 Traceback (most recent call last):
276 ...
277 File "vllm/model_executor/models/registry.py", line xxx, in inspect_model_cls
278 for arch in architectures:
279 TypeError: 'NoneType' object is not iterable
280 ```
281
282 or:
283
284 ```text
285 File "vllm/model_executor/models/registry.py", line xxx, in _raise_for_unsupported
286 raise ValueError(
287 ValueError: Model architectures ['<arch>'] are not supported for now. Supported architectures: [...]
288 ```
289
290 But you are sure that the model is in the [list of supported models](../models/supported_models.md), there may be some issue with vLLM's model resolution. In that case, please follow [these steps](../configuration/model_resolution.md) to explicitly specify the vLLM implementation for the model.
291
292 ## Failed to infer device type
293
294 If you see an error like `RuntimeError: Failed to infer device type`, it means that vLLM failed to infer the device type of the runtime environment. You can check [the code](../../vllm/platforms/__init__.py) to see how vLLM infers the device type and why it is not working as expected. After [this PR](https://github.com/vllm-project/vllm/pull/14195), you can also set the environment variable `VLLM_LOGGING_LEVEL=DEBUG` to see more detailed logs to help debug the issue.
295
296 ## NCCL error: unhandled system error during `ncclCommInitRank`
297
298 If your serving workload uses GPUDirect RDMA for distributed serving across multiple nodes and encounters an error during `ncclCommInitRank`, with no clear error message even with `NCCL_DEBUG=INFO` set, it might look like this:
299
300 ```text
301 Error executing method 'init_device'. This might cause deadlock in distributed execution.
302 Traceback (most recent call last):
303 ...
304 File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 99, in __init__
305 self.comm: ncclComm_t = self.nccl.ncclCommInitRank(
306 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
307 File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 277, in ncclCommInitRank
308 self.NCCL_CHECK(self._funcs["ncclCommInitRank"](ctypes.byref(comm),
309 File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 256, in NCCL_CHECK
310 raise RuntimeError(f"NCCL error: {error_str}")
311 RuntimeError: NCCL error: unhandled system error (run with NCCL_DEBUG=INFO for details)
312 ...
313 ```
314
315 This indicates vLLM failed to initialize the NCCL communicator, possibly due to a missing `IPC_LOCK` linux capability or an unmounted `/dev/shm`. Refer to [Enabling GPUDirect RDMA](../serving/parallelism_scaling.md#enabling-gpudirect-rdma) for guidance on properly configuring the environment for GPUDirect RDMA.
316
317 ## CUDA error: the provided PTX was compiled with an unsupported toolchain
318
319 If you see an error like `RuntimeError: CUDA error: the provided PTX was compiled with an unsupported toolchain`, it means that the CUDA PTX in vLLM's wheels was compiled with a toolchain unsupported by your system. This section also applies if you get the error `RuntimeError: The NVIDIA driver on your system is too old`.
320
321 The released vLLM wheels are compiled with a specific version of CUDA toolkit, and the compiled code might fail to run on lower versions of CUDA drivers. Read [CUDA compatibility](https://docs.nvidia.com/deploy/cuda-compatibility/) for more details. **This is only supported on select professional and datacenter NVIDIA GPUs.**
322
323 If you are using the vLLM official Docker image, you can solve this by adding `-e VLLM_ENABLE_CUDA_COMPATIBILITY=1` to your `docker run` command. This will enable the pre-installed CUDA forward compatibility libraries.
324
325 If you are running vLLM outside of Docker, the solution is to install the `cuda-compat` package from your package manager with the [CUDA repository](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/) enabled. For example, on Ubuntu, you can run `sudo apt-get install cuda-compat-12-9`, and then set `export VLLM_ENABLE_CUDA_COMPATIBILITY=1` and `export VLLM_CUDA_COMPATIBILITY_PATH="/usr/local/cuda-12.9/compat"`.
326
327 On Conda, you can install the `conda-forge::cuda-compat` package (e.g., `conda install -c conda-forge cuda-compat=12.9`), then after activating the environment, set `export VLLM_ENABLE_CUDA_COMPATIBILITY=1` and `export VLLM_CUDA_COMPATIBILITY_PATH="${CONDA_PREFIX}/cuda-compat"`.
328
329 You can verify the configuration works by running a minimal Python script that initializes CUDA via vLLM:
330
331 ```bash
332 export VLLM_ENABLE_CUDA_COMPATIBILITY=1
333 export VLLM_CUDA_COMPATIBILITY_PATH="/usr/local/cuda-12.9/compat"
334
335 python3 - << 'EOF'
336 import vllm
337 import torch
338
339 print(f"CUDA available: {torch.cuda.is_available()}")
340 print(f"CUDA device count: {torch.accelerator.device_count()}")
341 EOF
342 ```
343
344 Note that we use CUDA 12.9 as an example here, and you may want to install a higher version of cuda-compat package in case vLLM's default CUDA version goes higher.
345
346 ## ptxas fatal: Value 'sm_110a' is not defined for option 'gpu-name'
347
348 If you use triton kernels with cuda 13, you might see an error like `ptxas fatal: Value 'sm_110a' is not defined for option 'gpu-name'`:
349
350 ```text
351 (EngineCore_0 pid=9492) triton.runtime.errors.PTXASError: PTXAS error: Internal Triton PTX codegen error
352 (EngineCore_0 pid=9492) `ptxas` stderr:
353 (EngineCore_0 pid=9492) ptxas fatal : Value 'sm_110a' is not defined for option 'gpu-name'
354 (EngineCore_0 pid=9492)
355 (EngineCore_0 pid=9492) Repro command: /home/jetson/.venv/lib/python3.12/site-packages/triton/backends/nvidia/bin/ptxas -lineinfo -v --gpu-name=sm_110a /tmp/tmp95oy_b9d.ptx -o /tmp/tmp95oy_b9d.ptx.o
356 (EngineCore_0 pid=9492)
357 outputs = self.engine_core.get_output()
358 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
359 File "/home/jetson/.venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 668, in get_output
360 raise self._format_exception(outputs) from None
361 vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
362 ```
363
364 It means that the ptxas in the triton bundle is not compatible with your device. You need to set `TRITON_PTXAS_PATH` environment variable to use cuda toolkit's ptxas manually instead:
365
366 ```shell
367 export CUDA_HOME=/usr/local/cuda
368 export TRITON_PTXAS_PATH="${CUDA_HOME}/bin/ptxas"
369 export PATH="${CUDA_HOME}/bin:$PATH"
370 ```
371
372 ## Known Issues
373
374 - In `v0.5.2`, `v0.5.3`, and `v0.5.3.post1`, there is a bug caused by [zmq](https://github.com/zeromq/pyzmq/issues/2000) , which can occasionally cause vLLM to hang depending on the machine configuration. The solution is to upgrade to the latest version of `vllm` to include the [fix](https://github.com/vllm-project/vllm/pull/6759).
375 - To address a memory overhead issue in older NCCL versions (see [bug](https://github.com/NVIDIA/nccl/issues/1234)), vLLM versions `>= 0.4.3, <= 0.10.1.1` would set the environment variable `NCCL_CUMEM_ENABLE=0`. External processes connecting to vLLM also needed to set this variable to prevent hangs or crashes. Since the underlying NCCL bug was fixed in NCCL 2.22.3, this override was removed in newer vLLM versions to allow for NCCL performance optimizations.
376 - In some PCIe machines (e.g. machines without NVLink), if you see an error like `transport/shm.cc:590 NCCL WARN Cuda failure 217 'peer access is not supported between these two devices'`, it's likely caused by a driver bug. See [this issue](https://github.com/NVIDIA/nccl/issues/1838) for more details. In that case, you can try to set `NCCL_CUMEM_HOST_ENABLE=0` to disable the feature, or upgrade your driver to the latest version.