A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 11:00 UTC
[misc] add instructions on how to install nvshmem/pplx/deepep (#17964)
Signed-off-by: youkaichao <youkaichao@gmail.com>
youkaichao committed
May 12, 2025 at 09:02 UTC
d45fe333fb8d3ab73d73a6458e3cde73f14f0d7e
4 files changed
+146
tools/ep_kernels/README.md
new
+27
@@ -0,0 +1,27 @@
1
+Large-scale cluster-level expert parallel, as described in the [DeepSeek-V3 Technical Report](http://arxiv.org/abs/2412.19437), is an efficient way to deploy sparse MoE models with many experts. However, such deployment requires many components beyond a normal Python package, including system package support and system driver support. It is impossible to bundle all these components into a Python package.
2
+
3
+Here we break down the requirements in 3 steps:
4
+1. Build and install the Python libraries (both [pplx-kernels](https://github.com/ppl-ai/pplx-kernels) and [DeepEP](https://github.com/deepseek-ai/DeepEP)), including necessary dependencies like NVSHMEM. This step does not require any privileged access. Any user can do this.
5
+2. Build and install the system libraries (GDR Copy). This step requires root access. You can do it inside a Docker container so that they can be shipped as a single image.
6
+3. Build and install the system drivers (GDR Copy, and necessary modifications to NVIDIA driver to enable IBGDA). This step requires root access, and must be done on the host machine.
7
+
8
+2 and 3 are necessary for multi-node deployment.
9
+
10
+All scripts accept a positional argument as workspace path for staging the build, defaulting to `$(pwd)/ep_kernels_workspace`.
11
+
12
+# Usage
13
+
14
+## Single-node
15
+
16
+```bash
17
+bash install_python_libraries.sh
18
+```
19
+
20
+## Multi-node
21
+
22
+```bash
23
+bash install_python_libraries.sh
24
+sudo bash install_system_libraries.sh
25
+sudo bash install_system_drivers.sh
26
+sudo reboot # Reboot is required to load the new driver
27
+```
tools/ep_kernels/install_python_libraries.sh
new
+77
@@ -0,0 +1,77 @@
1
+set -ex
2
+
3
+# prepare workspace directory
4
+WORKSPACE=$1
5
+if [ -z "$WORKSPACE" ]; then
6
+ export WORKSPACE=$(pwd)/ep_kernels_workspace
7
+fi
8
+
9
+if [ ! -d "$WORKSPACE" ]; then
10
+ mkdir -p $WORKSPACE
11
+fi
12
+
13
+# install dependencies if not installed
14
+pip3 install cmake torch ninja
15
+
16
+# build gdrcopy, required by nvshmem
17
+pushd $WORKSPACE
18
+wget https://github.com/NVIDIA/gdrcopy/archive/refs/tags/v2.4.4.tar.gz
19
+mkdir -p gdrcopy_src
20
+tar -xvf v2.4.4.tar.gz -C gdrcopy_src --strip-components=1
21
+pushd gdrcopy_src
22
+make -j$(nproc)
23
+make prefix=$WORKSPACE/gdrcopy_install install
24
+popd
25
+
26
+# build nvshmem
27
+pushd $WORKSPACE
28
+mkdir -p nvshmem_src
29
+wget https://developer.download.nvidia.com/compute/redist/nvshmem/3.2.5/source/nvshmem_src_3.2.5-1.txz
30
+tar -xvf nvshmem_src_3.2.5-1.txz -C nvshmem_src --strip-components=1
31
+pushd nvshmem_src
32
+wget https://github.com/deepseek-ai/DeepEP/raw/main/third-party/nvshmem.patch
33
+git init
34
+git apply -vvv nvshmem.patch
35
+
36
+# assume CUDA_HOME is set correctly
37
+export GDRCOPY_HOME=$WORKSPACE/gdrcopy_install
38
+export NVSHMEM_SHMEM_SUPPORT=0
39
+export NVSHMEM_UCX_SUPPORT=0
40
+export NVSHMEM_USE_NCCL=0
41
+export NVSHMEM_IBGDA_SUPPORT=1
42
+export NVSHMEM_PMIX_SUPPORT=0
43
+export NVSHMEM_TIMEOUT_DEVICE_POLLING=0
44
+export NVSHMEM_USE_GDRCOPY=1
45
+export NVSHMEM_IBRC_SUPPORT=1
46
+
47
+# remove MPI dependency
48
+export NVSHMEM_BUILD_TESTS=0
49
+export NVSHMEM_BUILD_EXAMPLES=0
50
+export NVSHMEM_MPI_SUPPORT=0
51
+
52
+cmake -S . -B $WORKSPACE/nvshmem_build/ -DCMAKE_INSTALL_PREFIX=$WORKSPACE/nvshmem_install
53
+
54
+cd $WORKSPACE/nvshmem_build/
55
+make -j$(nproc)
56
+make install
57
+
58
+popd
59
+
60
+export CMAKE_PREFIX_PATH=$WORKSPACE/nvshmem_install:$CMAKE_PREFIX_PATH
61
+
62
+# build and install pplx, require pytorch installed
63
+pushd $WORKSPACE
64
+git clone https://github.com/ppl-ai/pplx-kernels
65
+cd pplx-kernels
66
+# see https://github.com/pypa/pip/issues/9955#issuecomment-838065925
67
+# PIP_NO_BUILD_ISOLATION=0 disables build isolation
68
+PIP_NO_BUILD_ISOLATION=0 TORCH_CUDA_ARCH_LIST=9.0a+PTX pip install -vvv -e .
69
+popd
70
+
71
+# build and install deepep, require pytorch installed
72
+pushd $WORKSPACE
73
+git clone https://github.com/deepseek-ai/DeepEP
74
+cd DeepEP
75
+export NVSHMEM_DIR=$WORKSPACE/nvshmem_install
76
+PIP_NO_BUILD_ISOLATION=0 pip install -vvv -e .
77
+popd
tools/ep_kernels/install_system_drivers.sh
new
+24
@@ -0,0 +1,24 @@
1
+set -ex
2
+
3
+# prepare workspace directory
4
+WORKSPACE=$1
5
+if [ -z "$WORKSPACE" ]; then
6
+ export WORKSPACE=$(pwd)/ep_kernels_workspace
7
+fi
8
+
9
+if [ ! -d "$WORKSPACE" ]; then
10
+ mkdir -p $WORKSPACE
11
+fi
12
+
13
+# build and install gdrcopy driver
14
+pushd $WORKSPACE
15
+cd gdrcopy_src
16
+./insmod.sh
17
+# run gdrcopy_copybw to test the installation
18
+$WORKSPACE/gdrcopy_install/bin/gdrcopy_copybw
19
+
20
+# turn on IBGDA
21
+echo 'options nvidia NVreg_EnableStreamMemOPs=1 NVreg_RegistryDwords="PeerMappingOverride=1;"' | tee -a /etc/modprobe.d/nvidia.conf
22
+update-initramfs -u
23
+
24
+echo "Please reboot the system to apply the changes"
tools/ep_kernels/install_system_libraries.sh
new
+18
@@ -0,0 +1,18 @@
1
+set -ex
2
+
3
+# prepare workspace directory
4
+WORKSPACE=$1
5
+if [ -z "$WORKSPACE" ]; then
6
+ export WORKSPACE=$(pwd)/ep_kernels_workspace
7
+fi
8
+
9
+if [ ! -d "$WORKSPACE" ]; then
10
+ mkdir -p $WORKSPACE
11
+fi
12
+
13
+# build and install gdrcopy system packages
14
+pushd $WORKSPACE
15
+cd gdrcopy_src/packages
16
+apt install devscripts -y
17
+CUDA=${CUDA_HOME:-/usr/local/cuda} ./build-deb-packages.sh
18
+dpkg -i *.deb