@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 175 lines 6 KB
Rendered Raw
1 <!-- markdownlint-disable MD041 -->
2 --8<-- [start:installation]
3
4 vLLM has experimental support for s390x architecture on IBM Z platform. For now, users must build from source to natively run on IBM Z platform.
5
6 Currently, the CPU implementation for s390x architecture supports FP32, BF16 and FP16, as well as AWQ and GPTQ 4-bit quantization and compressed-tensors INT8 W8A8.
7
8 --8<-- [end:installation]
9 --8<-- [start:requirements]
10
11 - OS: `Linux`
12 - SDK: `gcc/g++ >= 14.0.0` or later with Command Line Tools
13 - Instruction Set Architecture (ISA): VXE support is required. Works with Z15 and above.
14 - Build from source python packages (no pre-built s390x wheels): `torchvision`, `llvmlite`, `numba`, `opencv-python-headless`, `hf-xet`, `torchcodec`
15
16 --8<-- [end:requirements]
17 --8<-- [start:set-up-using-python]
18
19 --8<-- [end:set-up-using-python]
20 --8<-- [start:pre-built-wheels]
21
22 Currently, there are no pre-built IBM Z CPU wheels.
23
24 --8<-- [end:pre-built-wheels]
25 --8<-- [start:build-wheel-from-source]
26
27 Install the following packages from the package manager before building the vLLM. For example on RHEL 9.6:
28
29 ```bash
30 dnf install -y \
31 which procps findutils tar vim git patch xz ninja-build \
32 gcc-toolset-14 gcc-toolset-14-binutils gcc-toolset-14-libatomic-devel zlib-devel \
33 libjpeg-turbo-devel libtiff-devel libpng-devel libwebp-devel freetype-devel harfbuzz-devel \
34 openssl-devel openblas openblas-devel autoconf automake libtool cmake numpy libsndfile \
35 clang llvm-devel llvm-static clang-devel
36 ```
37
38 Build and install `numactl` from source:
39
40 ```bash
41 curl -LO https://github.com/numactl/numactl/archive/refs/tags/v2.0.19.tar.gz
42 tar -xvzf v2.0.19.tar.gz
43 cd numactl-2.0.19
44 ./autogen.sh && ./configure && make && make install
45 cd ..
46 ```
47
48 Install rust>=1.80 which is needed for `outlines-core`, `uvloop`, and `hf-xet` python packages installation.
49
50 ```bash
51 curl https://sh.rustup.rs -sSf | sh -s -- -y && \
52 . "$HOME/.cargo/env"
53 ```
54
55 Execute the following commands to build and install vLLM from source.
56
57 !!! tip
58 Pre-built wheels are not available for s390x for the following packages. Build them from source before building vLLM: `torchvision`, `llvmlite`, `numba`, `opencv-python-headless`, `hf-xet`, `torchcodec`.
59 `torchcodec` also needs a shared FFmpeg (built in `Dockerfile.s390x`). Video decoding only; there is no s390x NVDEC path.
60 See `docker/Dockerfile.s390x` for exact versions and build commands used in each multi-stage build.
61
62 !!! note "LLVM 20 required for llvmlite"
63 `llvmlite v0.47` requires LLVM 20, but UBI 9.6 repos ship LLVM 21 which is
64 not compatible. You must build LLVM 20 from source before building `llvmlite`:
65
66 ```bash
67 curl -LO https://github.com/llvm/llvm-project/releases/download/llvmorg-20.1.8/llvm-project-20.1.8.src.tar.xz
68 tar -xf llvm-project-20.1.8.src.tar.xz
69 cmake -G Ninja -S llvm-project-20.1.8.src/llvm -B llvm-build \
70 -DCMAKE_BUILD_TYPE=Release \
71 -DCMAKE_INSTALL_PREFIX=/opt/llvm20 \
72 -DLLVM_TARGETS_TO_BUILD="SystemZ" \
73 -DLLVM_ENABLE_RTTI=ON \
74 -DLLVM_BUILD_TOOLS=OFF \
75 -DLLVM_BUILD_UTILS=ON \
76 -DLLVM_BUILD_EXAMPLES=OFF \
77 -DLLVM_BUILD_TESTS=OFF \
78 -DLLVM_INCLUDE_TESTS=OFF \
79 -DLLVM_INCLUDE_EXAMPLES=OFF \
80 -DLLVM_INCLUDE_BENCHMARKS=OFF
81 ninja -C llvm-build install
82 ```
83
84 Then build `llvmlite` pointing to LLVM 20:
85
86 ```bash
87 CMAKE_PREFIX_PATH=/opt/llvm20 LLVM_CONFIG=/opt/llvm20/bin/llvm-config \
88 python setup.py bdist_wheel
89 ```
90
91 ```bash
92 uv pip install -v \
93 /path/to/torchvision.whl \
94 /path/to/llvmlite.whl \
95 /path/to/numba.whl \
96 /path/to/opencv_python_headless.whl \
97 /path/to/hf_xet.whl \
98 /path/to/torchcodec.whl \
99 -r requirements/build/cpu.txt \
100 -r requirements/cpu.txt \
101 --torch-backend cpu \
102 --index-strategy unsafe-best-match && \
103 VLLM_TARGET_DEVICE=cpu VLLM_CPU_MOE_PREPACK=0 python setup.py bdist_wheel && \
104 uv pip install dist/*.whl
105 ```
106
107 ??? console "pip"
108 ```bash
109 pip install -v \
110 --extra-index-url https://download.pytorch.org/whl/cpu \
111 /path/to/torchvision.whl \
112 /path/to/llvmlite.whl \
113 /path/to/numba.whl \
114 /path/to/opencv_python_headless.whl \
115 /path/to/hf_xet.whl \
116 /path/to/torchcodec.whl \
117 -r requirements/build/cpu.txt \
118 -r requirements/cpu.txt && \
119 VLLM_TARGET_DEVICE=cpu VLLM_CPU_MOE_PREPACK=0 python setup.py bdist_wheel && \
120 pip install dist/*.whl
121 ```
122
123 !!! warning "set `LD_PRELOAD` for TCMalloc"
124 For best memory allocation performance, build and install
125 [gperftools](https://github.com/gperftools/gperftools) (TCMalloc) from
126 source and add it to `LD_PRELOAD`:
127
128 ```bash
129 # Build and install TCMalloc from source
130 curl -LO https://github.com/gperftools/gperftools/releases/download/gperftools-2.16/gperftools-2.16.tar.gz
131 tar -xzf gperftools-2.16.tar.gz
132 cd gperftools-2.16
133 ./configure --enable-minimal && make -j$(nproc) && sudo make install
134 sudo ldconfig
135 cd ..
136
137 # Add to LD_PRELOAD
138 export LD_PRELOAD="/usr/local/lib/libtcmalloc_minimal.so.4:$LD_PRELOAD"
139 ```
140
141 The Docker image (`Dockerfile.s390x`) already includes TCMalloc and sets
142 `LD_PRELOAD` automatically.
143
144 --8<-- [end:build-wheel-from-source]
145 --8<-- [start:pre-built-images]
146
147 Currently, there are no pre-built IBM Z CPU images.
148
149 --8<-- [end:pre-built-images]
150 --8<-- [start:build-image-from-source]
151
152 ```bash
153 docker build -f docker/Dockerfile.s390x \
154 --tag vllm-cpu-env .
155
156 # Launch OpenAI server
157 docker run --rm \
158 --security-opt seccomp=unconfined \
159 --cap-add SYS_NICE \
160 --shm-size 4g \
161 -p 8000:8000 \
162 -e VLLM_CPU_KVCACHE_SPACE=<KV cache space> \
163 -e VLLM_CPU_OMP_THREADS_BIND=<CPU cores for inference> \
164 vllm-cpu-env \
165 --model meta-llama/Llama-3.2-1B-Instruct \
166 --dtype bfloat16 \
167 other vLLM OpenAI server arguments
168 ```
169
170 !!! tip
171 Alternatively, `--privileged=true` also works but is broader and not generally recommended.
172
173 --8<-- [end:build-image-from-source]
174 --8<-- [start:extra-information]
175 --8<-- [end:extra-information]