A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 5, 18:30 UTC
Fix document of torchrun_example.py (#31113)
Maosheng Liao committed
Mar 31, 2026 at 18:54 UTC
aae3e688f8849bdad382943b78a7872f70b75046
1 file changed
+4
-3
examples/offline_inference/torchrun_example.py
+4
-3
@@ -4,9 +4,10 @@
4
experimental support for tensor-parallel inference with torchrun,
5
see https://github.com/vllm-project/vllm/issues/11400 for
6
the motivation and use case for this example.
7
-run the script with `torchrun --nproc-per-node=2 torchrun_example.py`,
8
-the argument 2 should match the `tensor_parallel_size` below.
9
-see `tests/distributed/test_torchrun_example.py` for the unit test.
7
+run the script with `torchrun --nproc-per-node=4 torchrun_example.py`,
8
+the argument `4` should match the product of `tensor_parallel_size` and
9
+`pipeline_parallel_size` below. see `tests/distributed/test_torchrun_example.py`
10
+for the unit test.
11
"""
12
13
import torch.distributed as dist