@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 292 lines 10.4 KB
Rendered Raw
1 <!-- markdownlint-disable MD041 MD051 -->
2 --8<-- [start:installation]
3
4 vLLM supports basic model inferencing and serving on x86 CPU platform, with data types FP32, FP16 and BF16.
5
6 --8<-- [end:installation]
7 --8<-- [start:requirements]
8
9 - OS: Linux
10 - CPU flags: `avx512f` (Recommended), `avx2` (Limited features)
11
12 !!! tip
13 Use `lscpu` to check the CPU flags.
14
15 --8<-- [end:requirements]
16 --8<-- [start:set-up-using-python]
17
18 --8<-- [end:set-up-using-python]
19 --8<-- [start:pre-built-wheels]
20
21 Pre-built vLLM wheels for x86 with AVX512/AVX2 are available since version 0.17.0. To install release wheels:
22
23 ```bash
24 export VLLM_VERSION=$(curl -s https://api.github.com/repos/vllm-project/vllm/releases/latest | jq -r .tag_name | sed 's/^v//')
25
26 # use uv
27 uv pip install "https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cpu-cp38-abi3-manylinux_2_34_x86_64.whl" --torch-backend cpu
28 ```
29
30 ??? console "pip"
31 ```bash
32 # use pip
33 pip install "https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cpu-cp38-abi3-manylinux_2_34_x86_64.whl" --extra-index-url https://download.pytorch.org/whl/cpu
34 ```
35 !!! warning "set `LD_PRELOAD`"
36 Before use vLLM CPU installed via wheels, make sure Intel OpenMP is added to `LD_PRELOAD`:
37 ```bash
38 # manually find the path
39 sudo find / -iname *libiomp5.so
40 IOMP_PATH=...
41
42 # add it to LD_PRELOAD
43 export LD_PRELOAD="$IOMP_PATH:$LD_PRELOAD"
44 ```
45
46 #### Install the latest code
47
48 To install the wheel built from the latest main branch:
49
50 ```bash
51 uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly/cpu --index-strategy first-index --torch-backend cpu
52 ```
53
54 #### Install specific revisions
55
56 If you want to access the wheels for previous commits (e.g. to bisect the behavior change, performance regression), you can specify the commit hash in the URL:
57
58 ```bash
59 export VLLM_COMMIT=730bd35378bf2a5b56b6d3a45be28b3092d26519 # use full commit hash from the main branch
60 uv pip install vllm --extra-index-url https://wheels.vllm.ai/${VLLM_COMMIT}/cpu --index-strategy first-index --torch-backend cpu
61 ```
62
63 --8<-- [end:pre-built-wheels]
64 --8<-- [start:build-wheel-from-source]
65
66 Install recommended compiler. We recommend to use `gcc/g++ >= 12.3.0` as the default compiler to avoid potential problems. For example, on Ubuntu 22.4, you can run:
67
68 ```bash
69 sudo apt-get update -y
70 sudo apt-get install -y gcc-12 g++-12 libnuma-dev
71 sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 10 --slave /usr/bin/g++ g++ /usr/bin/g++-12
72 ```
73
74 --8<-- "docs/getting_started/installation/python_env_setup.inc.md"
75
76 Clone the vLLM project:
77
78 ```bash
79 git clone https://github.com/vllm-project/vllm.git vllm_source
80 cd vllm_source
81 ```
82
83 Install the required dependencies:
84
85 ```bash
86 uv pip install -r requirements/build/cpu.txt --torch-backend cpu --index-strategy unsafe-best-match
87 uv pip install -r requirements/cpu.txt --torch-backend cpu --index-strategy unsafe-best-match
88 ```
89
90 ??? console "pip"
91 ```bash
92 pip install --upgrade pip
93 pip install -v -r requirements/build/cpu.txt --extra-index-url https://download.pytorch.org/whl/cpu
94 pip install -v -r requirements/cpu.txt --extra-index-url https://download.pytorch.org/whl/cpu
95 ```
96
97 Build and install vLLM:
98
99 ```bash
100 VLLM_TARGET_DEVICE=cpu uv pip install . --no-build-isolation
101 ```
102
103 If you want to develop vLLM, install it in editable mode instead.
104
105 ```bash
106 VLLM_TARGET_DEVICE=cpu python3 setup.py develop
107 ```
108
109 Optionally, build a portable wheel which you can then install elsewhere:
110
111 ```bash
112 VLLM_TARGET_DEVICE=cpu uv build --wheel --no-build-isolation
113 ```
114
115 ```bash
116 uv pip install dist/*.whl
117 ```
118
119 ??? console "pip"
120 ```bash
121 VLLM_TARGET_DEVICE=cpu python -m build --wheel --no-isolation
122 ```
123
124 ```bash
125 pip install dist/*.whl
126 ```
127
128 !!! warning "set `LD_PRELOAD`"
129 Before using vLLM CPU installed via wheels, make sure TCMalloc and Intel OpenMP are installed and added to `LD_PRELOAD`:
130 ```bash
131 # install TCMalloc, Intel OpenMP is installed with vLLM CPU
132 sudo apt-get install -y --no-install-recommends libtcmalloc-minimal4
133
134 # manually find the path
135 sudo find / -iname *libtcmalloc_minimal.so.4
136 sudo find / -iname *libiomp5.so
137 TC_PATH=...
138 IOMP_PATH=...
139
140 # add them to LD_PRELOAD
141 export LD_PRELOAD="$TC_PATH:$IOMP_PATH:$LD_PRELOAD"
142 ```
143
144 !!! example "Troubleshooting"
145 - **NumPy ≥2.0 error**: Downgrade using `pip install "numpy<2.0"`.
146 - **CMake picks up CUDA**: Add `CMAKE_DISABLE_FIND_PACKAGE_CUDA=ON` to prevent CUDA detection during CPU builds, even if CUDA is installed.
147 - `AMD` requires at least 4th gen processors (Zen 4/Genoa) or higher to support [AVX512](https://www.phoronix.com/review/amd-zen4-avx512) to run vLLM on CPU.
148 - If you receive an error such as: `Could not find a version that satisfies the requirement torch==X.Y.Z+cpu+cpu`, consider updating [pyproject.toml](https://github.com/vllm-project/vllm/blob/main/pyproject.toml) to help pip resolve the dependency.
149 ```toml title="pyproject.toml"
150 [build-system]
151 requires = [
152 "cmake>=3.26.1",
153 ...
154 "torch==X.Y.Z+cpu" # <-------
155 ]
156 ```
157
158 --8<-- [end:build-wheel-from-source]
159 --8<-- [start:pre-built-images]
160
161 You can pull the latest available CPU image from Docker Hub:
162
163 ```bash
164 docker pull vllm/vllm-openai-cpu:latest-x86_64
165 ```
166
167 To pull an image for a specific vLLM version:
168
169 ```bash
170 export VLLM_VERSION=$(curl -s https://api.github.com/repos/vllm-project/vllm/releases/latest | jq -r .tag_name | sed 's/^v//')
171 docker pull vllm/vllm-openai-cpu:v${VLLM_VERSION}-x86_64
172 ```
173
174 All available image tags are here: [https://hub.docker.com/r/vllm/vllm-openai-cpu/tags](https://hub.docker.com/r/vllm/vllm-openai-cpu/tags)
175
176 You can run these images via:
177
178 ```bash
179 docker run \
180 -v ~/.cache/huggingface:/root/.cache/huggingface \
181 -p 8000:8000 \
182 --env "HF_TOKEN=<secret>" \
183 vllm/vllm-openai-cpu:latest-x86_64 <args...>
184 ```
185
186 ### Serve with vLLM Recipes
187
188 The CPU image includes the [vLLM Recipes](https://recipes.vllm.ai/) deployment
189 tool. Recipe-assisted serving queries the site at runtime, so the image can use
190 the latest published recipe for the selected model and hardware.
191
192 To find the value for `--hardware`, open vLLM Recipes, select the model, and
193 choose the target in the **Hardware** picker. The selected page URL also
194 contains the hardware key; for example, `?hardware=xeon6` maps to
195 `--hardware xeon6`.
196
197 Keep the normal image entrypoint for regular serving. For recipe-assisted
198 serving, override it with `serve_with_recipe.sh`:
199
200 ```bash
201 docker run --rm \
202 --shm-size=4g \
203 -v ~/.cache/huggingface:/root/.cache/huggingface \
204 -p 8000:8000 \
205 --env "HF_TOKEN=<secret>" \
206 --entrypoint tools/recipes/serve_with_recipe.sh \
207 vllm/vllm-openai-cpu:latest-x86_64 \
208 --model meta-llama/Llama-3.1-8B-Instruct \
209 --hardware xeon6
210 ```
211
212 For Xeon 6, the script enables hardware detection automatically, generates
213 `config.yml` and `env.sh`, and starts `vllm serve`. The image keeps
214 `/vllm-workspace` as its working directory so relative recipe assets under
215 `examples/` continue to resolve.
216
217 See the [Recipes tool documentation](../../../tools/recipes/README.md) for
218 interactive discovery and additional options.
219
220 --8<-- [end:pre-built-images]
221 --8<-- [start:build-image-from-source]
222
223 #### Building for your target CPU
224
225 ```bash
226 docker build -f docker/Dockerfile.cpu \
227 --build-arg VLLM_CPU_X86=<false (default)|true> \ # For cross-compilation
228 --tag vllm-cpu-env \
229 --target vllm-openai .
230 ```
231
232 #### Building with AMD Zen optimizations
233
234 For AMD Zen 4 / Zen 5 hosts (`linux/amd64` only), use the `vllm-openai-zen` target. It extends the default `vllm-openai` image and adds `zentorch` via the `vllm[zen]` extra so `ZenCpuPlatform` auto-activates at runtime:
235
236 ```bash
237 docker build -f docker/Dockerfile.cpu \
238 --tag vllm-cpu-zen-env \
239 --target vllm-openai-zen .
240 ```
241
242 The resulting image accepts the same arguments and environment variables as `vllm-openai` (see [Launching the OpenAI server](#launching-the-openai-server) below); no extra flag is needed to engage Zen optimizations. See [AMD Zen optimizations](cpu.md#amd-zen-optimizations) for runtime behavior and the supported-dtype caveats.
243
244 #### Launching the OpenAI server {#launching-the-openai-server}
245
246 ```bash
247 docker run --rm \
248 --security-opt seccomp=unconfined \
249 --cap-add SYS_NICE \
250 --shm-size=4g \
251 -p 8000:8000 \
252 -e VLLM_CPU_KVCACHE_SPACE=<KV cache space> \
253 vllm-cpu-env \
254 meta-llama/Llama-3.2-1B-Instruct \
255 --dtype=bfloat16 \
256 other vLLM OpenAI server arguments
257 ```
258
259 --8<-- [end:build-image-from-source]
260 --8<-- [start:amd-zen-optimizations]
261
262 On AMD Zen CPUs, vLLM auto-selects `ZenCpuPlatform` (a subclass of `CpuPlatform`) which dispatches linear layers through [`zentorch`](https://github.com/amd/ZenDNN-pytorch-plugin)'s ZenDNN-optimized kernels. See the FAQ entry [How do I enable AMD Zen optimizations?](#how-do-i-enable-amd-zen-optimizations) for the install command.
263
264 ### Detection rules
265
266 `ZenCpuPlatform` is selected when **all** of the following hold:
267
268 - vLLM is built for CPU
269 - `/proc/cpuinfo` reports `AuthenticAMD` and `avx512`
270 - `import zentorch` succeeds
271
272 Otherwise, vLLM falls back to the default `CpuPlatform` (oneDNN / sgl-kernel paths).
273
274 ### Supported dtypes
275
276 `float16` is **not** supported on `ZenCpuPlatform`. `ZenCpuPlatform.supported_dtypes` advertises only `bfloat16` and `float32`, so models declared with `torch_dtype=float16` are auto-downcast to `bfloat16` at load time with the standard `"Your device 'cpu' doesn't support torch.float16. Falling back to torch.bfloat16 for compatibility."` warning emitted from `vllm/config/model.py`.
277
278 ### Environment variables
279
280 - `VLLM_ZENTORCH_WEIGHT_PREPACK` (default `1`): eagerly prepacks linear weights into ZenDNN's blocked layout at model load time, eliminating per-inference layout conversion overhead. Set to `0` to disable.
281
282 ### Docker
283
284 The `vllm-openai-zen` Docker target (in `docker/Dockerfile.cpu`) extends the default `vllm-openai` image with `vllm[zen]`. Build it with `docker build -f docker/Dockerfile.cpu --target vllm-openai-zen .` — see [Building with AMD Zen optimizations](#building-with-amd-zen-optimizations) for the full command and run instructions.
285
286 ### Reference
287
288 For the design rationale, see [RFC #35089: In-Tree AMD Zen CPU Backend via zentorch](https://github.com/vllm-project/vllm/issues/35089).
289
290 --8<-- [end:amd-zen-optimizations]
291 --8<-- [start:extra-information]
292 --8<-- [end:extra-information]