A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 12, 02:15 UTC
[CPU Backend] [Doc]: Update Installation Docs for CPUs (#29868)
Signed-off-by: Ioana Ghiban <ioana.ghiban@arm.com>
ioana ghiban committed
Dec 3, 2025 at 14:33 UTC
1bb17ecb396f911beaa26ab0d3926d46154c7155
5 files changed
+58
-14
docs/getting_started/installation/cpu.apple.inc.md
+4
-3
@@ -4,9 +4,6 @@ vLLM has experimental support for macOS with Apple Silicon. For now, users must
4
5
Currently the CPU implementation for macOS supports FP32 and FP16 datatypes.
6
7
-!!! warning
8
- There are no pre-built wheels or images for this device, so you must build vLLM from source.
9
-
7
# --8<-- [end:installation]
8
# --8<-- [start:requirements]
9
@@ -20,6 +17,8 @@ Currently the CPU implementation for macOS supports FP32 and FP16 datatypes.
17
# --8<-- [end:set-up-using-python]
18
# --8<-- [start:pre-built-wheels]
19
20
+Currently, there are no pre-built Apple silicon CPU wheels.
21
+
22
# --8<-- [end:pre-built-wheels]
23
# --8<-- [start:build-wheel-from-source]
24
@@ -78,6 +77,8 @@ uv pip install -e .
77
# --8<-- [end:build-wheel-from-source]
78
# --8<-- [start:pre-built-images]
79
80
+Currently, there are no pre-built Arm silicon CPU images.
81
+
82
# --8<-- [end:pre-built-images]
83
# --8<-- [start:build-image-from-source]
84
docs/getting_started/installation/cpu.arm.inc.md
+20
-6
@@ -1,11 +1,6 @@
1
# --8<-- [start:installation]
2
3
-vLLM has been adapted to work on ARM64 CPUs with NEON support, leveraging the CPU backend initially developed for the x86 platform.
4
-
5
-ARM CPU backend currently supports Float32, FP16 and BFloat16 datatypes.
6
-
7
-!!! warning
8
- There are no pre-built wheels or images for this device, so you must build vLLM from source.
3
+vLLM offers basic model inferencing and serving on Arm CPU platform, with support NEON, data types FP32, FP16 and BF16.
4
5
# --8<-- [end:installation]
6
# --8<-- [start:requirements]
@@ -20,6 +15,23 @@ ARM CPU backend currently supports Float32, FP16 and BFloat16 datatypes.
15
# --8<-- [end:set-up-using-python]
16
# --8<-- [start:pre-built-wheels]
17
18
+Pre-built vLLM wheels for Arm are available since version 0.11.2. These wheels contain pre-compiled C++ binaries.
19
+Please replace `<version>` in the commands below with a specific version string (e.g., `0.11.2`).
20
+
21
+```bash
22
+uv pip install --pre vllm==<version>+cpu --extra-index-url https://wheels.vllm.ai/<version>%2Bcpu/
23
+```
24
+
25
+??? console "pip"
26
+ ```bash
27
+ pip install --pre vllm==<version>+cpu --extra-index-url https://wheels.vllm.ai/<version>%2Bcpu/
28
+ ```
29
+
30
+The `uv` approach works for vLLM `v0.6.6` and later. A unique feature of `uv` is that packages in `--extra-index-url` have [higher priority than the default index](https://docs.astral.sh/uv/pip/compatibility/#packages-that-exist-on-multiple-indexes). If the latest public release is `v0.6.6.post1`, `uv`'s behavior allows installing a commit before `v0.6.6.post1` by specifying the `--extra-index-url`. In contrast, `pip` combines packages from `--extra-index-url` and the default index, choosing only the latest version, which makes it difficult to install a development version prior to the released version.
31
+
32
+!!! note
33
+ Nightly wheels are currently unsupported for this architecture. (e.g. to bisect the behavior change, performance regression).
34
+
35
# --8<-- [end:pre-built-wheels]
36
# --8<-- [start:build-wheel-from-source]
37
@@ -69,6 +81,8 @@ Testing has been conducted on AWS Graviton3 instances for compatibility.
81
# --8<-- [end:build-wheel-from-source]
82
# --8<-- [start:pre-built-images]
83
84
+Currently, there are no pre-built Arm CPU images.
85
+
86
# --8<-- [end:pre-built-images]
87
# --8<-- [start:build-image-from-source]
88
```bash
docs/getting_started/installation/cpu.md
+28
-2
@@ -46,11 +46,25 @@ vLLM is a Python library that supports the following CPU variants. Select your C
46
47
### Pre-built wheels
48
49
-Please refer to the instructions for [pre-built wheels on GPU](./gpu.md#pre-built-wheels).
50
-
49
When specifying the index URL, please make sure to use the `cpu` variant subdirectory.
50
For example, the nightly build index is: `https://wheels.vllm.ai/nightly/cpu/`.
51
52
+=== "Intel/AMD x86"
53
+
54
+ --8<-- "docs/getting_started/installation/cpu.x86.inc.md:pre-built-wheels"
55
+
56
+=== "ARM AArch64"
57
+
58
+ --8<-- "docs/getting_started/installation/cpu.arm.inc.md:pre-built-wheels"
59
+
60
+=== "Apple silicon"
61
+
62
+ --8<-- "docs/getting_started/installation/cpu.apple.inc.md:pre-built-wheels"
63
+
64
+=== "IBM Z (S390X)"
65
+
66
+ --8<-- "docs/getting_started/installation/cpu.s390x.inc.md:pre-built-wheels"
67
+
68
### Build wheel from source
69
70
#### Set up using Python-only build (without compilation) {#python-only-build}
@@ -87,6 +101,18 @@ VLLM_USE_PRECOMPILED=1 VLLM_PRECOMPILED_WHEEL_VARIANT=cpu VLLM_TARGET_DEVICE=cpu
101
102
--8<-- "docs/getting_started/installation/cpu.x86.inc.md:pre-built-images"
103
104
+=== "ARM AArch64"
105
+
106
+ --8<-- "docs/getting_started/installation/cpu.arm.inc.md:pre-built-images"
107
+
108
+=== "Apple silicon"
109
+
110
+ --8<-- "docs/getting_started/installation/cpu.apple.inc.md:pre-built-images"
111
+
112
+=== "IBM Z (S390X)"
113
+
114
+ --8<-- "docs/getting_started/installation/cpu.s390x.inc.md:pre-built-images"
115
+
116
### Build image from source
117
118
=== "Intel/AMD x86"
docs/getting_started/installation/cpu.s390x.inc.md
+4
-3
@@ -4,9 +4,6 @@ vLLM has experimental support for s390x architecture on IBM Z platform. For now,
4
5
Currently, the CPU implementation for s390x architecture supports FP32 datatype only.
6
7
-!!! warning
8
- There are no pre-built wheels or images for this device, so you must build vLLM from source.
9
-
7
# --8<-- [end:installation]
8
# --8<-- [start:requirements]
9
@@ -21,6 +18,8 @@ Currently, the CPU implementation for s390x architecture supports FP32 datatype
18
# --8<-- [end:set-up-using-python]
19
# --8<-- [start:pre-built-wheels]
20
21
+Currently, there are no pre-built IBM Z CPU wheels.
22
+
23
# --8<-- [end:pre-built-wheels]
24
# --8<-- [start:build-wheel-from-source]
25
@@ -69,6 +68,8 @@ Execute the following commands to build and install vLLM from source.
68
# --8<-- [end:build-wheel-from-source]
69
# --8<-- [start:pre-built-images]
70
71
+Currently, there are no pre-built IBM Z CPU images.
72
+
73
# --8<-- [end:pre-built-images]
74
# --8<-- [start:build-image-from-source]
75
docs/getting_started/installation/cpu.x86.inc.md
+2
@@ -17,6 +17,8 @@ vLLM supports basic model inferencing and serving on x86 CPU platform, with data
17
# --8<-- [end:set-up-using-python]
18
# --8<-- [start:pre-built-wheels]
19
20
+Currently, there are no pre-built x86 CPU wheels.
21
+
22
# --8<-- [end:pre-built-wheels]
23
# --8<-- [start:build-wheel-from-source]
24