@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 3e051bda82

Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 14, 05:30 UTC

[UX] Replace VLLM_ALL2ALL_BACKEND with --all2all-backend (#26732)

Signed-off-by: mgoin <mgoin64@gmail.com>

Michael Goin committed Oct 13, 2025 at 21:12 UTC 3e051bda82efe351ecfb5bb21de1606accc976d4

12 files changed +90 -51

docs/design/dbo.md

+2 -2

@@ -34,10 +34,10 @@ To enable the DBO system pass in the `--enable-dbo` argument to your vllm serve

34 * `--dbo-decode-token-threshold` the minimum number of tokens in a decode-only batch required to enable DBO for that batch

35 * `--dbo-prefill-token-threshold` the minimum number of tokens in a batch containing at least one prefill required to enable DBO for that batch

37 -Currently, DBO is only supported with DeepEP, so DeepEP must be installed and the `VLLM_ALL2ALL_BACKEND` environment variable must be set to `deepep_low_latency` if your workload is primarily decode requests, or `deepep_high_throughput` if your workload is primarily prefill requests.

37 +Currently, DBO is only supported with DeepEP, so DeepEP must be installed and the `--all2all-backend` argument must be set to `deepep_low_latency` if your workload is primarily decode requests, or `deepep_high_throughput` if your workload is primarily prefill requests.

39 Below is a command that will spin up a two DP rank server with expert parallelism and DBO enabled.

40 -EX: `VLLM_ALL2ALL_BACKEND=deepep_low_latency vllm serve --model="deepseek-ai/DeepSeek-V2-Lite" --trust-remote-code --data-parallel-size 2 --enable-expert-parallel --enable-dbo`

40 +EX: `vllm serve deepseek-ai/DeepSeek-V2-Lite --trust-remote-code --data-parallel-size 2 --enable-expert-parallel --enable-dbo --all2all-backend deepep_low_latency`

42 Note that there must be at least two GPUs visible in `CUDA_VISIBLE_DEVICES`

docs/serving/expert_parallel_deployment.md

+21 -17

@@ -14,13 +14,16 @@ Before using EP, you need to install the necessary dependencies. We are actively

15 ### Backend Selection Guide

17 -vLLM provides three communication backends for EP:

17 +vLLM provides multiple communication backends for EP. Use `--all2all-backend` to select one:

20 |---------|----------|----------|----------|

28 ## Single Node Deployment

@@ -47,11 +50,11 @@ The following command serves a `DeepSeek-V3-0324` model with 1-way tensor parall

51 ```bash

52 # Single node EP deployment with pplx backend

50 -VLLM_ALL2ALL_BACKEND=pplx VLLM_USE_DEEP_GEMM=1 \

51 - vllm serve deepseek-ai/DeepSeek-V3-0324 \

52 - --tensor-parallel-size 1 \ # Tensor parallelism across 1 GPU

53 +vllm serve deepseek-ai/DeepSeek-V3-0324 \

54 + --tensor-parallel-size 1 \ # Tensor parallelism across 1 GPU

55 --data-parallel-size 8 \ # Data parallelism across 8 processes

54 - --enable-expert-parallel # Enable expert parallelism

56 + --enable-expert-parallel \ # Enable expert parallelism

57 + --all2all-backend pplx # Use pplx communication backend

58 ```

60 ## Multi-Node Deployment

@@ -70,8 +73,8 @@ The following example deploys `DeepSeek-V3-0324` across 2 nodes using `deepep_lo

74 ```bash

75 # Node 1 (Primary - handles incoming requests)

73 -VLLM_ALL2ALL_BACKEND=deepep_low_latency VLLM_USE_DEEP_GEMM=1 \

74 - vllm serve deepseek-ai/DeepSeek-V3-0324 \

76 +vllm serve deepseek-ai/DeepSeek-V3-0324 \

77 + --all2all-backend deepep_low_latency \

78 --tensor-parallel-size 1 \ # TP size per node

79 --enable-expert-parallel \ # Enable EP

80 --data-parallel-size 16 \ # Total DP size across all nodes

@@ -81,8 +84,8 @@ VLLM_ALL2ALL_BACKEND=deepep_low_latency VLLM_USE_DEEP_GEMM=1 \

84 --api-server-count=8 # Number of API servers for load handling (scaling this out to total ranks are recommended)

86 # Node 2 (Secondary - headless mode, no API server)

84 -VLLM_ALL2ALL_BACKEND=deepep_low_latency VLLM_USE_DEEP_GEMM=1 \

85 - vllm serve deepseek-ai/DeepSeek-V3-0324 \

87 +vllm serve deepseek-ai/DeepSeek-V3-0324 \

88 + --all2all-backend deepep_low_latency \

89 --tensor-parallel-size 1 \ # TP size per node

90 --enable-expert-parallel \ # Enable EP

91 --data-parallel-size 16 \ # Total DP size across all nodes

@@ -169,11 +172,12 @@ Single node deployment with EPLB enabled:

172

173 ```bash

174 # Single node with EPLB load balancing

172 -VLLM_ALL2ALL_BACKEND=pplx VLLM_USE_DEEP_GEMM=1 vllm serve deepseek-ai/DeepSeek-V3-0324 \

173 - --tensor-parallel-size 1 \ # Tensor parallelism

174 - --data-parallel-size 8 \ # Data parallelism

175 - --enable-expert-parallel \ # Enable EP

176 - --enable-eplb \ # Enable load balancer

175 +vllm serve deepseek-ai/DeepSeek-V3-0324 \

176 + --tensor-parallel-size 1 \ # Tensor parallelism

177 + --data-parallel-size 8 \ # Data parallelism

178 + --enable-expert-parallel \ # Enable EP

179 + --all2all-backend pplx \ # Use pplx communication backend

180 + --enable-eplb \ # Enable load balancer

181 --eplb-config '{"window_size":1000,"step_interval":3000,"num_redundant_experts":2,"log_balancedness":true}'

182 ```

183

vllm/config/parallel.py

+31 -2

@@ -113,6 +113,25 @@ class ParallelConfig:

113 with 4 experts and 2 ranks, rank 0 will have experts [0, 2] and rank 1

114 will have experts [1, 3]. This strategy can help improve load balancing

115 for grouped expert models with no redundant experts."""

116 + all2all_backend: (

117 + Literal[

118 + "naive",

119 + "pplx",

120 + "deepep_high_throughput",

121 + "deepep_low_latency",

122 + "allgather_reducescatter",

123 + "flashinfer_all2allv",

124 + ]

125 + | None

126 + ) = None

127 + """All2All backend for MoE expert parallel communication. If not set, uses

128 + the value from VLLM_ALL2ALL_BACKEND environment variable. Available options:

129 + - "naive": Naive all2all implementation using broadcasts

130 + - "allgather_reducescatter": All2all based on allgather and reducescatter

131 + - "pplx": Use pplx kernels

132 + - "deepep_high_throughput": Use deepep high-throughput kernels

133 + - "deepep_low_latency": Use deepep low-latency kernels

134 + - "flashinfer_all2allv": Use flashinfer alltoallv kernels for mnnvl"""

135 num_redundant_experts: int | None = None

136 """`num_redundant_experts` is deprecated and has been replaced with

137 `eplb_config.num_redundant_experts`. This will be removed in v0.12.0.

@@ -341,7 +360,7 @@ class ParallelConfig:

360 @property

361 def use_sequence_parallel_moe(self) -> bool:

362 return (

344 - envs.VLLM_ALL2ALL_BACKEND

363 + self.all2all_backend

364 in (

365 "allgather_reducescatter",

366 "naive",

@@ -390,7 +409,7 @@ class ParallelConfig:

409 factors.append(self.tensor_parallel_size)

410 factors.append(self.enable_expert_parallel)

411 factors.append(self.data_parallel_size)

393 - factors.append(envs.VLLM_ALL2ALL_BACKEND)

412 + factors.append(self.all2all_backend)

413 factors.append(self.enable_eplb)

414 if self.enable_eplb:

415 factors.append(self.eplb_config.log_balancedness)

@@ -400,6 +419,16 @@ class ParallelConfig:

419 return hashlib.sha256(str(factors).encode()).hexdigest()

420

421 def __post_init__(self) -> None:

422 + # Set all2all_backend from env var if not specified, with deprecation warning

423 + if self.all2all_backend is None:

424 + self.all2all_backend = envs.VLLM_ALL2ALL_BACKEND

425 + if envs.is_set("VLLM_ALL2ALL_BACKEND"):

426 + logger.warning_once(

427 + "VLLM_ALL2ALL_BACKEND environment variable is deprecated and "

428 + "will be removed in a future release. Please use the "

429 + "--all2all-backend command-line argument instead."

430 + )

431 +

432 # Forward deprecated fields to their new location

433 if self.num_redundant_experts is not None:

434 self.eplb_config.num_redundant_experts = self.num_redundant_experts

vllm/config/vllm.py

+4 -4

@@ -523,13 +523,13 @@ class VllmConfig:

523 )

524

525 if self.parallel_config.enable_dbo:

526 - a2a_backend = envs.VLLM_ALL2ALL_BACKEND

526 + a2a_backend = self.parallel_config.all2all_backend

527 assert a2a_backend in ["deepep_low_latency", "deepep_high_throughput"], (

528 "Microbatching currently only supports the deepep_low_latency and "

529 f"deepep_high_throughput all2all backend. {a2a_backend} is not "

530 - "supported. To fix set the VLLM_ALL2ALL_BACKEND environment "

531 - "variable to deepep_low_latency or deepep_high_throughput and "

532 - "install the DeepEP kernels."

530 + "supported. To fix use --all2all-backend=deepep_low_latency or "

531 + "--all2all-backend=deepep_high_throughput and install the DeepEP"

532 + " kernels."

533 )

534

535 if not self.model_config.disable_cascade_attn:

vllm/distributed/device_communicators/base_device_communicator.py

@@ -111,6 +111,7 @@ class DeviceCommunicatorBase:

111 self.rank_in_group = dist.get_group_rank(self.cpu_group, self.global_rank)

112

113 use_ep = False

114 + all2all_backend = None

115 from vllm.config import get_current_vllm_config

116

117 config = get_current_vllm_config()

@@ -119,9 +120,11 @@ class DeviceCommunicatorBase:

120 # where all data parallel ranks execute forward together),

121 # we initialize the all2all manager used in expert parallel.

122 use_ep = config.parallel_config.data_parallel_size > 1

123 + all2all_backend = config.parallel_config.all2all_backend

124

125 self.is_ep_communicator = "ep" in unique_name

126 self.use_all2all = self.is_ep_communicator and use_ep

127 + self.all2all_backend = all2all_backend

128 self.all2all_manager: All2AllManagerBase | None = None

129

130 def all_reduce(self, input_: torch.Tensor) -> torch.Tensor:

vllm/distributed/device_communicators/cuda_communicator.py

+7 -8

@@ -91,33 +91,32 @@ class CudaCommunicator(DeviceCommunicatorBase):

91 self.qr_comm = QuickAllReduce(group=self.cpu_group, device=self.device)

93 if self.use_all2all:

94 - all2all_backend = envs.VLLM_ALL2ALL_BACKEND

95 - if all2all_backend == "naive":

94 + if self.all2all_backend == "naive":

95 from .all2all import NaiveAll2AllManager

97 self.all2all_manager = NaiveAll2AllManager(self.cpu_group)

99 - elif all2all_backend == "allgather_reducescatter":

98 + elif self.all2all_backend == "allgather_reducescatter":

99 from .all2all import AgRsAll2AllManager

100

101 self.all2all_manager = AgRsAll2AllManager(self.cpu_group)

103 - elif all2all_backend == "pplx":

102 + elif self.all2all_backend == "pplx":

103 from .all2all import PPLXAll2AllManager

104

105 self.all2all_manager = PPLXAll2AllManager(self.cpu_group)

107 - elif all2all_backend == "deepep_high_throughput":

106 + elif self.all2all_backend == "deepep_high_throughput":

107 from .all2all import DeepEPHTAll2AllManager

108

109 self.all2all_manager = DeepEPHTAll2AllManager(self.cpu_group)

111 - elif all2all_backend == "deepep_low_latency":

110 + elif self.all2all_backend == "deepep_low_latency":

111 from .all2all import DeepEPLLAll2AllManager

112

113 self.all2all_manager = DeepEPLLAll2AllManager(self.cpu_group)

115 - elif all2all_backend == "flashinfer_all2allv":

114 + elif self.all2all_backend == "flashinfer_all2allv":

115 from .all2all import FlashInferAllToAllManager

116

117 self.all2all_manager = FlashInferAllToAllManager(self.cpu_group)

118 else:

120 - raise ValueError(f"Unknown all2all backend: {all2all_backend}")

119 + raise ValueError(f"Unknown all2all backend: {self.all2all_backend}")

120

121 if is_global_first_rank():

122 logger.info(

vllm/distributed/device_communicators/xpu_communicator.py

+5 -7

@@ -6,7 +6,6 @@ import torch

6 import torch.distributed as dist

7 from torch.distributed import ProcessGroup

9 -import vllm.envs as envs

9 from vllm.logger import init_logger

11 from .base_device_communicator import DeviceCommunicatorBase

@@ -24,15 +23,14 @@ class XpuCommunicator(DeviceCommunicatorBase):

23 ):

24 super().__init__(cpu_group, device, device_group, unique_name)

25 if self.use_all2all:

27 - all2all_backend = envs.VLLM_ALL2ALL_BACKEND

28 - if all2all_backend != "naive":

26 + if self.all2all_backend != "naive":

27 logger.warning(

30 - "`%s` all2all manager is not supported on XPU."

28 + "`%s` all2all manager is not supported on XPU. "

29 "Falling back to `naive` all2all manager for XPU.",

32 - all2all_backend,

30 + self.all2all_backend,

31 )

34 - all2all_backend = "naive"

35 - if all2all_backend == "naive":

32 + self.all2all_backend = "naive"

33 + if self.all2all_backend == "naive":

34 from .all2all import NaiveAll2AllManager

36 self.all2all_manager = NaiveAll2AllManager(self.cpu_group)

vllm/engine/arg_utils.py

@@ -371,6 +371,7 @@ class EngineArgs:

371 data_parallel_hybrid_lb: bool = False

372 data_parallel_backend: str = ParallelConfig.data_parallel_backend

373 enable_expert_parallel: bool = ParallelConfig.enable_expert_parallel

374 + all2all_backend: str | None = ParallelConfig.all2all_backend

375 enable_dbo: bool = ParallelConfig.enable_dbo

376 dbo_decode_token_threshold: int = ParallelConfig.dbo_decode_token_threshold

377 dbo_prefill_token_threshold: int = ParallelConfig.dbo_prefill_token_threshold

@@ -763,6 +764,9 @@ class EngineArgs:

764 parallel_group.add_argument(

765 "--enable-expert-parallel", **parallel_kwargs["enable_expert_parallel"]

766 )

767 + parallel_group.add_argument(

768 + "--all2all-backend", **parallel_kwargs["all2all_backend"]

769 + )

770 parallel_group.add_argument("--enable-dbo", **parallel_kwargs["enable_dbo"])

771 parallel_group.add_argument(

772 "--dbo-decode-token-threshold",

@@ -1461,6 +1465,7 @@ class EngineArgs:

1465 data_parallel_backend=self.data_parallel_backend,

1466 data_parallel_hybrid_lb=self.data_parallel_hybrid_lb,

1467 enable_expert_parallel=self.enable_expert_parallel,

1468 + all2all_backend=self.all2all_backend,

1469 enable_dbo=self.enable_dbo,

1470 dbo_decode_token_threshold=self.dbo_decode_token_threshold,

1471 dbo_prefill_token_threshold=self.dbo_prefill_token_threshold,

vllm/model_executor/layers/fused_moe/config.py

+6 -6

@@ -641,6 +641,7 @@ class FusedMoEParallelConfig:

641 ep_rank: int

642

643 use_ep: bool # whether to use EP or not

644 + all2all_backend: str # all2all backend for MoE communication

645

646 @property

647 def use_all2all_kernels(self):

@@ -648,21 +649,18 @@ class FusedMoEParallelConfig:

649

650 @property

651 def use_pplx_kernels(self):

651 - return self.use_all2all_kernels and envs.VLLM_ALL2ALL_BACKEND == "pplx"

652 + return self.use_all2all_kernels and self.all2all_backend == "pplx"

653

654 @property

655 def use_deepep_ht_kernels(self):

656 return (

657 self.use_all2all_kernels

657 - and envs.VLLM_ALL2ALL_BACKEND == "deepep_high_throughput"

658 + and self.all2all_backend == "deepep_high_throughput"

659 )

660

661 @property

662 def use_deepep_ll_kernels(self):

662 - return (

663 - self.use_all2all_kernels

664 - and envs.VLLM_ALL2ALL_BACKEND == "deepep_low_latency"

665 - )

663 + return self.use_all2all_kernels and self.all2all_backend == "deepep_low_latency"

664

665 @staticmethod

666 def make(

@@ -762,6 +760,7 @@ class FusedMoEParallelConfig:

760 ep_size=1,

761 ep_rank=0,

762 use_ep=False,

763 + all2all_backend=vllm_parallel_config.all2all_backend,

764 )

765 # DP + EP / TP + EP / DP + TP + EP

766 assert use_ep

@@ -777,6 +776,7 @@ class FusedMoEParallelConfig:

776 ep_size=ep_size,

777 ep_rank=ep_rank,

778 use_ep=True,

779 + all2all_backend=vllm_parallel_config.all2all_backend,

780 )

781

782

vllm/model_executor/layers/quantization/utils/flashinfer_fp4_moe.py

+1 -1

@@ -58,7 +58,7 @@ def build_flashinfer_fp4_cutlass_moe_prepare_finalize(

58 ) -> mk.FusedMoEPrepareAndFinalize:

59 """Create a FlashInfer CUTLASS fused-MoE prepare finalize kernel"""

60 use_dp = moe.moe_parallel_config.dp_size > 1

61 - enable_alltoallv = envs.VLLM_ALL2ALL_BACKEND == "flashinfer_all2allv"

61 + enable_alltoallv = moe.moe_parallel_config.all2all_backend == "flashinfer_all2allv"

62 return create_flashinfer_prepare_finalize(

63 use_dp=use_dp, use_nvfp4=True, enable_alltoallv=enable_alltoallv

64 )

vllm/platforms/cuda.py

+2 -2

@@ -192,7 +192,7 @@ class CudaPlatformBase(Platform):

192

193 compilation_config = vllm_config.compilation_config

194 if (

195 - envs.VLLM_ALL2ALL_BACKEND == "deepep_high_throughput"

195 + parallel_config.all2all_backend == "deepep_high_throughput"

196 and parallel_config.data_parallel_size > 1

197 and compilation_config.cudagraph_mode != CUDAGraphMode.NONE

198 ):

@@ -204,7 +204,7 @@ class CudaPlatformBase(Platform):

204 "kernels are optimized for prefill and are incompatible with "

205 "CUDA Graphs. "

206 "In order to use CUDA Graphs for decode-optimized workloads, "

207 - "set VLLM_ALL2ALL_BACKEND to another option, such as "

207 + "use --all2all-backend with another option, such as "

208 "deepep_low_latency, pplx, or allgather_reducescatter."

209 )

210 compilation_config.cudagraph_mode = CUDAGraphMode.NONE

vllm/v1/engine/utils.py

+3 -2

@@ -356,9 +356,10 @@ class CoreEngineActorManager:

356 )

357 device_str = current_platform.ray_device_key

358

359 + all2all_backend = vllm_config.parallel_config.all2all_backend

360 if envs.VLLM_RAY_DP_PACK_STRATEGY == "fill" and (

360 - envs.VLLM_ALL2ALL_BACKEND == "deepep_high_throughput"

361 - or envs.VLLM_ALL2ALL_BACKEND == "deepep_low_latency"

361 + all2all_backend == "deepep_high_throughput"

362 + or all2all_backend == "deepep_low_latency"

363 ):

364 raise ValueError(

365 "DeepEP kernels require EP ranks [0,7] (same for [8,15], ...) "