A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 9, 07:30 UTC
[Doc] Enhance documentation around CPU container images (#32286)
Signed-off-by: Nathan Weinberg <nweinber@redhat.com>
Nathan Weinberg committed
Jan 30, 2026 at 08:36 UTC
58cb55e4def7ca45a1db52d967aee44fc28f959d
3 files changed
+41
-5
docs/deployment/k8s.md
+6
-2
@@ -59,11 +59,15 @@ First, create a Kubernetes PVC and Secret for downloading and storing Hugging Fa
59
Here, the `token` field stores your **Hugging Face access token**. For details on how to generate a token,
60
see the [Hugging Face documentation](https://huggingface.co/docs/hub/en/security-tokens).
61
62
-Next, start the vLLM server as a Kubernetes Deployment and Service:
62
+Next, start the vLLM server as a Kubernetes Deployment and Service.
63
+
64
+Note that you will want to configure your vLLM image based on your processor arch:
65
66
??? console "Config"
67
68
```bash
69
+ VLLM_IMAGE=public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:latest # use this for x86_64
70
+ VLLM_IMAGE=public.ecr.aws/q9t5s3a7/vllm-arm64-cpu-release-repo:latest # use this for arm64
71
cat <<EOF |kubectl apply -f -
72
apiVersion: apps/v1
73
kind: Deployment
@@ -81,7 +85,7 @@ Next, start the vLLM server as a Kubernetes Deployment and Service:
85
spec:
86
containers:
87
- name: vllm
84
- image: vllm/vllm-openai:latest
88
+ image: $VLLM_IMAGE
89
command: ["/bin/sh", "-c"]
90
args: [
91
"vllm serve meta-llama/Llama-3.2-1B-Instruct"
docs/getting_started/installation/cpu.arm.inc.md
+18
-2
@@ -136,15 +136,31 @@ Testing has been conducted on AWS Graviton3 instances for compatibility.
136
# --8<-- [end:build-wheel-from-source]
137
# --8<-- [start:pre-built-images]
138
139
-See [Using Docker](../../deployment/docker.md) for instructions on using the official Docker image.
139
+To pull the latest image:
140
141
-Stable vLLM Docker images are being pre-built for Arm from version 0.12.0. Available image tags are here: [https://gallery.ecr.aws/q9t5s3a7/vllm-arm64-cpu-release-repo](https://gallery.ecr.aws/q9t5s3a7/vllm-arm64-cpu-release-repo).
141
+```bash
142
+docker pull public.ecr.aws/q9t5s3a7/vllm-arm64-cpu-release-repo:latest
143
+```
144
+
145
+To pull an image with a specific vLLM version:
146
147
```bash
148
export VLLM_VERSION=$(curl -s https://api.github.com/repos/vllm-project/vllm/releases/latest | jq -r .tag_name | sed 's/^v//')
149
docker pull public.ecr.aws/q9t5s3a7/vllm-arm64-cpu-release-repo:v${VLLM_VERSION}
150
```
151
152
+All available image tags are here: [https://gallery.ecr.aws/q9t5s3a7/vllm-arm64-cpu-release-repo](https://gallery.ecr.aws/q9t5s3a7/vllm-arm64-cpu-release-repo).
153
+
154
+You can run these images via:
155
+
156
+```bash
157
+docker run \
158
+ -v ~/.cache/huggingface:/root/.cache/huggingface \
159
+ -p 8000:8000 \
160
+ --env "HF_TOKEN=<secret>" \
161
+ public.ecr.aws/q9t5s3a7/vllm-arm64-cpu-release-repo:<tag> <args...>
162
+```
163
+
164
You can also access the latest code with Docker images. These are not intended for production use and are meant for CI and testing only. They will expire after several days.
165
166
The latest code can contain bugs and may not be stable. Please use it with caution.
docs/getting_started/installation/cpu.x86.inc.md
+17
-1
@@ -161,7 +161,23 @@ uv pip install dist/*.whl
161
# --8<-- [end:build-wheel-from-source]
162
# --8<-- [start:pre-built-images]
163
164
-[https://gallery.ecr.aws/q9t5s3a7/vllm-cpu-release-repo](https://gallery.ecr.aws/q9t5s3a7/vllm-cpu-release-repo)
164
+You can pull the latest available CPU image here via:
165
+
166
+```bash
167
+docker pull public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:latest
168
+```
169
+
170
+If you want a more specific build you can find all published CPU based images here: [https://gallery.ecr.aws/q9t5s3a7/vllm-cpu-release-repo](https://gallery.ecr.aws/q9t5s3a7/vllm-cpu-release-repo)
171
+
172
+You can run these images via:
173
+
174
+```bash
175
+docker run \
176
+ -v ~/.cache/huggingface:/root/.cache/huggingface \
177
+ -p 8000:8000 \
178
+ --env "HF_TOKEN=<secret>" \
179
+ public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:<tag> <args...>
180
+```
181
182
!!! warning
183
If deploying the pre-built images on machines without `avx512f`, `avx512_bf16`, or `avx512_vnni` support, an `Illegal instruction` error may be raised. See the build-image-from-source section below for build arguments to match your target CPU capabilities.