| 1 | # Expert parallel kernels |
| 2 | |
| 3 | Large-scale cluster-level expert parallel, as described in the [DeepSeek-V3 Technical Report](http://arxiv.org/abs/2412.19437), is an efficient way to deploy sparse MoE models with many experts. However, such deployment requires many components beyond a normal Python package, including system package support and system driver support. It is impossible to bundle all these components into a Python package. |
| 4 | |
| 5 | Here we break down the requirements in 2 steps: |
| 6 | |
| 7 | 1. Build and install the Python libraries ([DeepEP](https://github.com/deepseek-ai/DeepEP)), including necessary dependencies like NVSHMEM. This step does not require any privileged access. Any user can do this. |
| 8 | 2. Configure NVIDIA driver to enable IBGDA. This step requires root access, and must be done on the host machine. |
| 9 | |
| 10 | Step 2 is necessary for multi-node deployment. |
| 11 | |
| 12 | All scripts accept a positional argument as workspace path for staging the build, defaulting to `$(pwd)/ep_kernels_workspace`. |
| 13 | |
| 14 | ## NCCL version requirement (CUDA 13+) |
| 15 | |
| 16 | DeepEPv2 uses the NCCL GIN (GPU-Initiated Networking) backend, which requires |
| 17 | NCCL >= 2.30.4 at both compile time and runtime. PyTorch 2.11 pins |
| 18 | `nvidia-nccl-cu13==2.28.9` as a transitive dependency, so you need to |
| 19 | override it. |
| 20 | |
| 21 | **With uv** (recommended): |
| 22 | |
| 23 | ```bash |
| 24 | # Create an override file |
| 25 | echo "nvidia-nccl-cu13>=2.30.4" > /tmp/nccl-override.txt |
| 26 | export UV_OVERRIDE=/tmp/nccl-override.txt |
| 27 | |
| 28 | # All subsequent uv pip install commands will respect the override |
| 29 | uv pip install vllm |
| 30 | ``` |
| 31 | |
| 32 | **With pip**: |
| 33 | |
| 34 | ```bash |
| 35 | pip install vllm |
| 36 | pip install "nvidia-nccl-cu13>=2.30.4" --no-deps |
| 37 | ``` |
| 38 | |
| 39 | The override / reinstall must happen before building DeepEP (for GIN device |
| 40 | headers) and must remain in place at runtime. You can verify with: |
| 41 | |
| 42 | ```bash |
| 43 | python -c "from vllm.utils.import_utils import has_deep_ep_v2; print(has_deep_ep_v2())" |
| 44 | ``` |
| 45 | |
| 46 | ## Usage |
| 47 | |
| 48 | ```bash |
| 49 | # for hopper |
| 50 | TORCH_CUDA_ARCH_LIST="9.0" bash install_python_libraries.sh |
| 51 | # for blackwell |
| 52 | TORCH_CUDA_ARCH_LIST="10.0" bash install_python_libraries.sh |
| 53 | ``` |
| 54 | |
| 55 | Additional step for multi-node deployment: |
| 56 | |
| 57 | ```bash |
| 58 | sudo bash configure_system_drivers.sh # update-initramfs can take several minutes |
| 59 | sudo reboot # Reboot is required to load the new driver |
| 60 | ``` |