A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 04:30 UTC
[doc] backward compatibility for 0.6.4 (#11359)
Signed-off-by: youkaichao <youkaichao@gmail.com>
youkaichao committed
Dec 19, 2024 at 21:33 UTC
1ecc645b8f5431f1404551ad24721a63f01aea4e
1 file changed
+5
docs/source/getting_started/debugging.rst
+5
@@ -86,6 +86,11 @@ If GPU/CPU communication cannot be established, you can use the following Python
86
from vllm.distributed.device_communicators.pynccl import PyNcclCommunicator
87
88
pynccl = PyNcclCommunicator(group=gloo_group, device=local_rank)
89
+ # pynccl is enabled by default for 0.6.5+,
90
+ # but for 0.6.4 and below, we need to enable it manually.
91
+ # keep the code for backward compatibility when because people
92
+ # prefer to read the latest documentation.
93
+ pynccl.disabled = False
94
95
s = torch.cuda.Stream()
96
with torch.cuda.stream(s):