@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 429284dc37
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 04:15 UTC

Fix `dist.broadcast` stall without group argument (#3408)

Junda Chen committed Mar 14, 2024 at 23:25 UTC 429284dc374bab79d4dfbb25053583901e6e5051
1 file changed +1 -1
vllm/model_executor/parallel_utils/communication_op.py
+1 -1
@@ -177,7 +177,7 @@ def broadcast_tensor_dict(
177 for key, value in metadata_list:
178 if isinstance(value, TensorMetadata):
179 tensor = tensor_dict[key]
180 - torch.distributed.broadcast(tensor, src=src)
180 + torch.distributed.broadcast(tensor, src=src, group=group)
181 else:
182 recv_metadata_list = [None]
183 torch.distributed.broadcast_object_list(recv_metadata_list,