@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / bd5038af07

Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 15, 03:30 UTC

[Doc] add config and troubleshooting guide for NCCL & GPUDirect RDMA (#15897)

Signed-off-by: Ernest Wong <chwong719@gmail.com>

Ernest Wong committed Jun 30, 2025 at 21:44 UTC bd5038af076a2e299d4781c3885415639a1ed3a5

2 files changed +65 -1

docs/serving/distributed_serving.md

+44 -1

@@ -100,7 +100,50 @@ vllm serve /path/to/the/model/in/the/container \

100 --tensor-parallel-size 16

101 ```

102

103 -To make tensor parallel performant, you should make sure the communication between nodes is efficient, e.g. using high-speed network cards like Infiniband. To correctly set up the cluster to use Infiniband, append additional arguments like `--privileged -e NCCL_IB_HCA=mlx5` to the `run_cluster.sh` script. Please contact your system administrator for more information on how to set up the flags. One way to confirm if the Infiniband is working is to run vLLM with `NCCL_DEBUG=TRACE` environment variable set, e.g. `NCCL_DEBUG=TRACE vllm serve ...` and check the logs for the NCCL version and the network used. If you find `[send] via NET/Socket` in the logs, it means NCCL uses raw TCP Socket, which is not efficient for cross-node tensor parallel. If you find `[send] via NET/IB/GDRDMA` in the logs, it means NCCL uses Infiniband with GPU-Direct RDMA, which is efficient.

103 +To make tensor parallel performant, you should make sure the communication between nodes is efficient, e.g. using high-speed network cards like InfiniBand. To correctly set up the cluster to use InfiniBand, append additional arguments like `--privileged -e NCCL_IB_HCA=mlx5` to the `run_cluster.sh` script. Please contact your system administrator for more information on how to set up the flags. One way to confirm if the InfiniBand is working is to run vLLM with `NCCL_DEBUG=TRACE` environment variable set, e.g. `NCCL_DEBUG=TRACE vllm serve ...` and check the logs for the NCCL version and the network used. If you find `[send] via NET/Socket` in the logs, it means NCCL uses raw TCP Socket, which is not efficient for cross-node tensor parallel. If you find `[send] via NET/IB/GDRDMA` in the logs, it means NCCL uses InfiniBand with GPUDirect RDMA, which is efficient.

104 +

105 +### GPUDirect RDMA

106 +

107 +To enable GPUDirect RDMA with vLLM, specific configuration tweaks are needed. This setup ensures:

108 +

109 +- `IPC_LOCK` Security Context: Add the `IPC_LOCK` capability to the container’s security context to lock memory pages and prevent swapping to disk.

110 +- Shared Memory with `/dev/shm`: Mount `/dev/shm` in the pod spec to provide shared memory for IPC.

111 +

112 +When using Docker, you can set up the container as follows:

113 +

114 +```bash

115 +docker run --gpus all \

116 + --ipc=host \

117 + --shm-size=16G \

118 + -v /dev/shm:/dev/shm \

119 + vllm/vllm-openai

120 +```

121 +

122 +When using Kubernetes, you can set up the pod spec as follows:

123 +

124 +```yaml

125 +...

126 +spec:

127 + containers:

128 + - name: vllm

129 + image: vllm/vllm-openai

130 + securityContext:

131 + capabilities:

132 + add: ["IPC_LOCK"]

133 + volumeMounts:

134 + - mountPath: /dev/shm

135 + name: dshm

136 + resources:

137 + limits:

138 + nvidia.com/gpu: 8

139 + requests:

140 + nvidia.com/gpu: 8

141 + volumes:

142 + - name: dshm

143 + emptyDir:

144 + medium: Memory

145 +...

146 +```

147

148 !!! warning

149 After you start the Ray cluster, you'd better also check the GPU-GPU communication between nodes. It can be non-trivial to set up. Please refer to the [sanity check script][troubleshooting-incorrect-hardware-driver] for more information. If you need to set some environment variables for the communication configuration, you can append them to the `run_cluster.sh` script, e.g. `-e NCCL_SOCKET_IFNAME=eth0`. Note that setting environment variables in the shell (e.g. `NCCL_SOCKET_IFNAME=eth0 vllm serve ...`) only works for the processes in the same node, not for the processes in the other nodes. Setting environment variables when you create the cluster is the recommended way. See <gh-issue:6803> for more information.

docs/usage/troubleshooting.md

+21

@@ -273,6 +273,27 @@ But you are sure that the model is in the [list of supported models][supported-m

273

274 If you see an error like `RuntimeError: Failed to infer device type`, it means that vLLM failed to infer the device type of the runtime environment. You can check [the code](gh-file:vllm/platforms/__init__.py) to see how vLLM infers the device type and why it is not working as expected. After [this PR](gh-pr:14195), you can also set the environment variable `VLLM_LOGGING_LEVEL=DEBUG` to see more detailed logs to help debug the issue.

275

276 +## NCCL error: unhandled system error during `ncclCommInitRank`

277 +

278 +If your serving workload uses GPUDirect RDMA for distributed serving across multiple nodes and encounters an error during `ncclCommInitRank`, with no clear error message even with `NCCL_DEBUG=INFO` set, it might look like this:

279 +

280 +```text

281 +Error executing method 'init_device'. This might cause deadlock in distributed execution.

282 +Traceback (most recent call last):

283 +...

284 + File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 99, in __init__

285 + self.comm: ncclComm_t = self.nccl.ncclCommInitRank(

286 + ^^^^^^^^^^^^^^^^^^^^^^^^^^^

287 + File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 277, in ncclCommInitRank

288 + self.NCCL_CHECK(self._funcs["ncclCommInitRank"](ctypes.byref(comm),

289 + File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 256, in NCCL_CHECK

290 + raise RuntimeError(f"NCCL error: {error_str}")

291 + RuntimeError: NCCL error: unhandled system error (run with NCCL_DEBUG=INFO for details)

292 +...

293 +```

294 +

295 +This indicates vLLM failed to initialize the NCCL communicator, possibly due to a missing `IPC_LOCK` linux capability or an unmounted `/dev/shm`. Refer to [Distributed Inference and Serving](../serving/distributed_serving.md#running-vllm-on-multiple-nodes) for guidance on properly configuring the environment for distributed serving.

296 +

297 ## Known Issues

298

299 - In `v0.5.2`, `v0.5.3`, and `v0.5.3.post1`, there is a bug caused by [zmq](https://github.com/zeromq/pyzmq/issues/2000) , which can occasionally cause vLLM to hang depending on the machine configuration. The solution is to upgrade to the latest version of `vllm` to include the [fix](gh-pr:6759).