| 1 | <!-- markdownlint-disable MD041 --> |
| 2 | --8<-- [start:installation] |
| 3 | |
| 4 | vLLM offers basic model inferencing and serving on Arm CPU platform, with support for NEON, data types FP32, FP16 and BF16. |
| 5 | |
| 6 | --8<-- [end:installation] |
| 7 | --8<-- [start:requirements] |
| 8 | |
| 9 | - OS: Linux |
| 10 | - Compiler: `gcc/g++ >= 12.3.0` (optional, recommended) |
| 11 | - Instruction Set Architecture (ISA): NEON support is required |
| 12 | |
| 13 | --8<-- [end:requirements] |
| 14 | --8<-- [start:set-up-using-python] |
| 15 | |
| 16 | --8<-- [end:set-up-using-python] |
| 17 | --8<-- [start:pre-built-wheels] |
| 18 | |
| 19 | Pre-built vLLM wheels for Arm are available since version 0.11.2. These wheels contain pre-compiled C++ binaries. |
| 20 | |
| 21 | ```bash |
| 22 | export VLLM_VERSION=$(curl -s https://api.github.com/repos/vllm-project/vllm/releases/latest | jq -r .tag_name | sed 's/^v//') |
| 23 | uv pip install "https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cpu-cp38-abi3-manylinux_2_34_aarch64.whl" --torch-backend cpu |
| 24 | ``` |
| 25 | |
| 26 | ??? console "pip" |
| 27 | ```bash |
| 28 | pip install "https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cpu-cp38-abi3-manylinux_2_34_aarch64.whl" --extra-index-url https://download.pytorch.org/whl/cpu |
| 29 | ``` |
| 30 | |
| 31 | The `uv` approach works for vLLM `v0.6.6` and later. A unique feature of `uv` is that packages in `--extra-index-url` have [higher priority than the default index](https://docs.astral.sh/uv/pip/compatibility/#packages-that-exist-on-multiple-indexes). If the latest public release is `v0.6.6.post1`, `uv`'s behavior allows installing a commit before `v0.6.6.post1` by specifying the `--extra-index-url`. In contrast, `pip` combines packages from `--extra-index-url` and the default index, choosing only the latest version, which makes it difficult to install a development version prior to the released version. |
| 32 | |
| 33 | #### Install the latest code |
| 34 | |
| 35 | LLM inference is a fast-evolving field, and the latest code may contain bug fixes, performance improvements, and new features that are not released yet. To allow users to try the latest code without waiting for the next release, vLLM provides working pre-built Arm CPU wheels for every commit since `v0.11.2` on <https://wheels.vllm.ai/nightly>. For native CPU wheels, this index should be used: |
| 36 | |
| 37 | - `https://wheels.vllm.ai/nightly/cpu/vllm` |
| 38 | |
| 39 | To install from nightly index, run: |
| 40 | |
| 41 | ```bash |
| 42 | uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly/cpu --index-strategy first-index --torch-backend cpu |
| 43 | ``` |
| 44 | |
| 45 | ??? console "pip (there's a caveat)" |
| 46 | |
| 47 | Using `pip` to install from nightly indices is _not supported_, because `pip` combines packages from `--extra-index-url` and the default index, choosing only the latest version, which makes it difficult to install a development version prior to the released version. In contrast, `uv` gives the extra index [higher priority than the default index](https://docs.astral.sh/uv/pip/compatibility/#packages-that-exist-on-multiple-indexes). |
| 48 | |
| 49 | If you insist on using `pip`, you have to specify the full URL (link address) of the wheel file (which can be obtained from https://wheels.vllm.ai/nightly/cpu/vllm). |
| 50 | |
| 51 | ```bash |
| 52 | pip install https://wheels.vllm.ai/2f3f441f84bd5b35ec8aa9fcfffb540f107da8a7/vllm-0.23.1rc1.dev901%2Bg2f3f441f8.cpu-cp38-abi3-manylinux_2_34_aarch64.whl --extra-index-url https://download.pytorch.org/whl/cpu # current nightly build (the filename will change!) |
| 53 | ``` |
| 54 | |
| 55 | #### Install specific revisions |
| 56 | |
| 57 | If you want to access the wheels for previous commits (e.g. to bisect the behavior change, performance regression), you can specify the commit hash in the URL: |
| 58 | |
| 59 | ```bash |
| 60 | export VLLM_COMMIT=730bd35378bf2a5b56b6d3a45be28b3092d26519 # use full commit hash from the main branch |
| 61 | uv pip install vllm --extra-index-url https://wheels.vllm.ai/${VLLM_COMMIT}/cpu --index-strategy first-index --torch-backend cpu |
| 62 | ``` |
| 63 | |
| 64 | --8<-- [end:pre-built-wheels] |
| 65 | --8<-- [start:build-wheel-from-source] |
| 66 | |
| 67 | First, install the recommended compiler. We recommend using `gcc/g++ >= 12.3.0` as the default compiler to avoid potential problems. For example, on Ubuntu 22.4, you can run: |
| 68 | |
| 69 | ```bash |
| 70 | sudo apt-get update -y |
| 71 | sudo apt-get install -y --no-install-recommends ccache git curl wget ca-certificates gcc-12 g++-12 libtcmalloc-minimal4 libnuma-dev ffmpeg libsm6 libxext6 libgl1 jq lsof |
| 72 | sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 10 --slave /usr/bin/g++ g++ /usr/bin/g++-12 |
| 73 | ``` |
| 74 | |
| 75 | Second, clone the vLLM project: |
| 76 | |
| 77 | ```bash |
| 78 | git clone https://github.com/vllm-project/vllm.git vllm_source |
| 79 | cd vllm_source |
| 80 | ``` |
| 81 | |
| 82 | Third, install required dependencies: |
| 83 | |
| 84 | ```bash |
| 85 | uv pip install -r requirements/build/cpu.txt --torch-backend cpu --index-strategy unsafe-best-match |
| 86 | uv pip install -r requirements/cpu.txt --torch-backend cpu --index-strategy unsafe-best-match |
| 87 | ``` |
| 88 | |
| 89 | ??? console "pip" |
| 90 | ```bash |
| 91 | pip install --upgrade pip |
| 92 | pip install -v -r requirements/build/cpu.txt --extra-index-url https://download.pytorch.org/whl/cpu |
| 93 | pip install -v -r requirements/cpu.txt --extra-index-url https://download.pytorch.org/whl/cpu |
| 94 | ``` |
| 95 | |
| 96 | Finally, build and install vLLM: |
| 97 | |
| 98 | ```bash |
| 99 | VLLM_TARGET_DEVICE=cpu uv pip install . --no-build-isolation |
| 100 | ``` |
| 101 | |
| 102 | If you want to develop vLLM, install it in editable mode instead. |
| 103 | |
| 104 | ```bash |
| 105 | VLLM_TARGET_DEVICE=cpu uv pip install -e . --no-build-isolation |
| 106 | ``` |
| 107 | |
| 108 | Testing has been conducted on AWS Graviton3 instances for compatibility. |
| 109 | |
| 110 | !!! warning "set `LD_PRELOAD`" |
| 111 | Before use vLLM CPU installed via wheels, make sure TCMalloc is installed and added to `LD_PRELOAD`: |
| 112 | ```bash |
| 113 | # install TCMalloc |
| 114 | sudo apt-get install -y --no-install-recommends libtcmalloc-minimal4 |
| 115 | |
| 116 | # manually find the path |
| 117 | sudo find / -iname *libtcmalloc_minimal.so.4 |
| 118 | TC_PATH=... |
| 119 | |
| 120 | # add them to LD_PRELOAD |
| 121 | export LD_PRELOAD="$TC_PATH:$LD_PRELOAD" |
| 122 | ``` |
| 123 | |
| 124 | --8<-- [end:build-wheel-from-source] |
| 125 | --8<-- [start:pre-built-images] |
| 126 | |
| 127 | To pull the latest image from Docker Hub: |
| 128 | |
| 129 | ```bash |
| 130 | docker pull vllm/vllm-openai-cpu:latest-arm64 |
| 131 | ``` |
| 132 | |
| 133 | To pull an image with a specific vLLM version: |
| 134 | |
| 135 | ```bash |
| 136 | export VLLM_VERSION=$(curl -s https://api.github.com/repos/vllm-project/vllm/releases/latest | jq -r .tag_name | sed 's/^v//') |
| 137 | docker pull vllm/vllm-openai-cpu:v${VLLM_VERSION}-arm64 |
| 138 | ``` |
| 139 | |
| 140 | All available image tags are here: [https://hub.docker.com/r/vllm/vllm-openai-cpu/tags](https://hub.docker.com/r/vllm/vllm-openai-cpu/tags). |
| 141 | |
| 142 | You can run these images via: |
| 143 | |
| 144 | ```bash |
| 145 | docker run \ |
| 146 | -v ~/.cache/huggingface:/root/.cache/huggingface \ |
| 147 | -p 8000:8000 \ |
| 148 | --env "HF_TOKEN=<secret>" \ |
| 149 | vllm/vllm-openai-cpu:latest-arm64 <args...> |
| 150 | ``` |
| 151 | |
| 152 | You can also access the latest code with Docker images. These are not intended for production use and are meant for CI and testing only. They will expire after several days. |
| 153 | |
| 154 | The latest code can contain bugs and may not be stable. Please use it with caution. |
| 155 | |
| 156 | ```bash |
| 157 | export VLLM_COMMIT=6299628d326f429eba78736acb44e76749b281f5 # use full commit hash from the main branch |
| 158 | docker pull public.ecr.aws/q9t5s3a7/vllm-ci-postmerge-repo:${VLLM_COMMIT}-arm64-cpu |
| 159 | ``` |
| 160 | |
| 161 | --8<-- [end:pre-built-images] |
| 162 | --8<-- [start:build-image-from-source] |
| 163 | |
| 164 | #### Building for your target ARM CPU |
| 165 | |
| 166 | ```bash |
| 167 | docker build -f docker/Dockerfile.cpu \ |
| 168 | --platform=linux/arm64 \ |
| 169 | --build-arg VLLM_CPU_ARM_BF16=<false (default)|true> \ |
| 170 | --tag vllm-cpu-env \ |
| 171 | --target vllm-openai . |
| 172 | ``` |
| 173 | |
| 174 | !!! note "Auto-detection by default" |
| 175 | By default, ARM CPU instruction sets (BF16, NEON, etc.) are automatically detected from the build system's CPU flags. The `VLLM_CPU_ARM_BF16` build argument is used for cross-compilation: |
| 176 | |
| 177 | - `VLLM_CPU_ARM_BF16=true` - Force-enable ARM BF16 support (build with BF16 regardless of build system capabilities) |
| 178 | - `VLLM_CPU_ARM_BF16=false` - Rely on auto-detection (default) |
| 179 | |
| 180 | ##### Examples |
| 181 | |
| 182 | ###### Auto-detection build (native ARM) |
| 183 | |
| 184 | ```bash |
| 185 | # Building on ARM64 system - platform auto-detected |
| 186 | docker build -f docker/Dockerfile.cpu \ |
| 187 | --tag vllm-cpu-arm64 \ |
| 188 | --target vllm-openai . |
| 189 | ``` |
| 190 | |
| 191 | ###### Cross-compile for ARM with BF16 support |
| 192 | |
| 193 | ```bash |
| 194 | # Building on ARM64 for newer ARM CPUs with BF16 |
| 195 | docker build -f docker/Dockerfile.cpu \ |
| 196 | --build-arg VLLM_CPU_ARM_BF16=true \ |
| 197 | --tag vllm-cpu-arm64-bf16 \ |
| 198 | --target vllm-openai . |
| 199 | ``` |
| 200 | |
| 201 | ###### Cross-compile from x86_64 to ARM64 with BF16 |
| 202 | |
| 203 | ```bash |
| 204 | # Requires Docker buildx with ARM emulation (QEMU) |
| 205 | docker buildx build -f docker/Dockerfile.cpu \ |
| 206 | --platform=linux/arm64 \ |
| 207 | --build-arg VLLM_CPU_ARM_BF16=true \ |
| 208 | --build-arg max_jobs=4 \ |
| 209 | --tag vllm-cpu-arm64-bf16 \ |
| 210 | --target vllm-openai \ |
| 211 | --load . |
| 212 | ``` |
| 213 | |
| 214 | !!! note "ARM BF16 requirements" |
| 215 | ARM BF16 support requires ARMv8.6-A or later (FEAT_BF16). Supported on AWS Graviton3/4, AmpereOne, and other recent ARM processors. |
| 216 | |
| 217 | #### Launching the OpenAI server |
| 218 | |
| 219 | ```bash |
| 220 | docker run --rm \ |
| 221 | --security-opt seccomp=unconfined \ |
| 222 | --cap-add SYS_NICE \ |
| 223 | --shm-size=4g \ |
| 224 | -p 8000:8000 \ |
| 225 | -e VLLM_CPU_KVCACHE_SPACE=<KV cache space> \ |
| 226 | -e VLLM_CPU_OMP_THREADS_BIND=<CPU cores for inference> \ |
| 227 | vllm-cpu-arm64 \ |
| 228 | meta-llama/Llama-3.2-1B-Instruct \ |
| 229 | --dtype=bfloat16 \ |
| 230 | other vLLM OpenAI server arguments |
| 231 | ``` |
| 232 | |
| 233 | !!! tip "Alternative to --privileged" |
| 234 | Instead of `--privileged=true`, use `--cap-add SYS_NICE --security-opt seccomp=unconfined` for better security. |
| 235 | |
| 236 | --8<-- [end:build-image-from-source] |
| 237 | --8<-- [start:extra-information] |
| 238 | --8<-- [end:extra-information] |