A high-throughput and memory-efficient inference and serving engine for LLMs
[ZenCPU] Add zencpu Platform Runtime Logging and Docs (#42726)
Signed-off-by: Lalithnarayan C <Lalithnarayan.C@amd.com> Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
Lalithnarayan C committed
Jun 16, 2026 at 17:53 UTC
405c7cf28352e45f68c338e643a4146550f3194e
6 files changed
+108
-3
docs/getting_started/installation/cpu.md
+25
@@ -142,6 +142,10 @@ VLLM_USE_PRECOMPILED=1 VLLM_PRECOMPILED_WHEEL_VARIANT=cpu VLLM_TARGET_DEVICE=cpu
142
=== "IBM Z (S390X)"
143
--8<-- "docs/getting_started/installation/cpu.s390x.inc.md:build-image-from-source"
144
145
+## AMD Zen optimizations {#amd-zen-optimizations}
146
+
147
+--8<-- "docs/getting_started/installation/cpu.x86.inc.md:amd-zen-optimizations"
148
+
149
## Related runtime environment variables
150
151
- `VLLM_CPU_KVCACHE_SPACE`: specify the KV Cache size (e.g, `VLLM_CPU_KVCACHE_SPACE=40` means 40 GiB space for KV cache), larger setting will allow vLLM to run more requests in parallel. This parameter should be set based on the hardware configuration and memory management pattern of users. Default value is `0`.
@@ -149,12 +153,14 @@ VLLM_USE_PRECOMPILED=1 VLLM_PRECOMPILED_WHEEL_VARIANT=cpu VLLM_TARGET_DEVICE=cpu
153
- `VLLM_CPU_NUM_OF_RESERVED_CPU`: specify the number of CPU cores which are not dedicated to the OpenMP threads for each rank. The variable only takes effect when VLLM_CPU_OMP_THREADS_BIND is set to `auto`. Default value is `None`. If the value is not set and use `auto` thread binding, no CPU will be reserved for `world_size == 1`, 1 CPU per rank will be reserved for `world_size > 1`.
154
- `CPU_VISIBLE_MEMORY_NODES`: specify visible NUMA memory nodes for vLLM CPU workers, similar to ```CUDA_VISIBLE_DEVICES```. The variable only takes effect when VLLM_CPU_OMP_THREADS_BIND is set to `auto`. The variable provides more control for the auto thread-binding feature, such as masking nodes and changing nodes binding sequence.
155
- `VLLM_CPU_SGL_KERNEL` (x86 only, Experimental): whether to use small-batch optimized kernels for linear layer and MoE layer, especially for low-latency requirements like online serving. The kernels require AMX instruction set, BFloat16 weight type and weight shapes divisible by 32. Default is `0` (False).
156
+- `VLLM_ZENTORCH_WEIGHT_PREPACK` (AMD Zen only): when `ZenCpuPlatform` is active, eagerly prepack linear weights into ZenDNN's blocked layout at model load time, eliminating per-inference layout conversion overhead. Default is `1` (enabled). See [AMD Zen optimizations](#amd-zen-optimizations).
157
158
## FAQ
159
160
### Which `dtype` should be used?
161
162
- Currently, vLLM CPU uses model default settings as `dtype`. However, due to unstable float16 support in torch CPU, it is recommended to explicitly set `dtype=bfloat16` if there are any performance or accuracy problem.
163
+- On AMD Zen CPUs (`ZenCpuPlatform`), `float16` is **not** supported. Only `bfloat16` and `float32` are accepted; models declared with `float16` are auto-downcast to `bfloat16` at model load time. See [AMD Zen optimizations](#amd-zen-optimizations).
164
165
### How to launch a vLLM service on CPU?
166
@@ -227,6 +233,25 @@ By providing MODEL_FILTER and DTYPE_FILTER, only commands for related model ID a
233
ON_CPU=1 SERVING_JSON=serving-tests-cpu-text.json DRY_RUN=1 MODEL_FILTER=meta-llama/Llama-3.1-8B-Instruct DTYPE_FILTER=bfloat16 bash .buildkite/performance-benchmarks/scripts/run-performance-benchmarks.sh
234
```
235
236
+### How do I enable AMD Zen optimizations? {#how-do-i-enable-amd-zen-optimizations}
237
+
238
+On an AMD Zen 4 / Zen 5 CPU, install the CPU wheel with the `zen` extra so vLLM pulls the tested `zentorch` version for that release:
239
+
240
+```bash
241
+export VLLM_VERSION=$(curl -s https://api.github.com/repos/vllm-project/vllm/releases/latest | jq -r .tag_name | sed 's/^v//')
242
+uv pip install "vllm[zen]" --extra-index-url https://wheels.vllm.ai/${VLLM_VERSION}/cpu --index-strategy first-index --torch-backend cpu
243
+```
244
+
245
+vLLM auto-detects the platform and routes linear layers through ZenDNN-optimized kernels - no flag needed. To verify it is engaged, look for the platform-selection line in the server's startup logs:
246
+
247
+```bash
248
+vllm serve Qwen/Qwen3-0.6B 2>&1 | grep "AMD Zen CPU detected with zentorch installed"
249
+```
250
+
251
+For per-backend dispatch details (which kernel each linear layer was bound to), re-run with `VLLM_LOGGING_LEVEL=DEBUG` and grep for `CPU unquantized GEMM dispatch`.
252
+
253
+See [AMD Zen optimizations](#amd-zen-optimizations) for detection rules, supported dtypes, and the `VLLM_ZENTORCH_WEIGHT_PREPACK` knob.
254
+
255
### How to decide `VLLM_CPU_OMP_THREADS_BIND`?
256
257
- Default `auto` thread-binding is recommended for most cases. Ideally, each OpenMP thread will be bound to a dedicated physical core respectively, threads of each rank will be bound to the same NUMA node respectively, and 1 CPU per rank will be reserved for other vLLM components when `world_size > 1`. If you have any performance problems or unexpected binding behaviours, please try to bind threads as following.
docs/getting_started/installation/cpu.x86.inc.md
+45
-2
@@ -1,4 +1,4 @@
1
-<!-- markdownlint-disable MD041 -->
1
+<!-- markdownlint-disable MD041 MD051 -->
2
--8<-- [start:installation]
3
4
vLLM supports basic model inferencing and serving on x86 CPU platform, with data types FP32, FP16 and BF16.
@@ -200,7 +200,19 @@ docker build -f docker/Dockerfile.cpu \
200
--target vllm-openai .
201
```
202
203
-#### Launching the OpenAI server
203
+#### Building with AMD Zen optimizations
204
+
205
+For AMD Zen 4 / Zen 5 hosts (`linux/amd64` only), use the `vllm-openai-zen` target. It extends the default `vllm-openai` image and adds `zentorch` via the `vllm[zen]` extra so `ZenCpuPlatform` auto-activates at runtime:
206
+
207
+```bash
208
+docker build -f docker/Dockerfile.cpu \
209
+ --tag vllm-cpu-zen-env \
210
+ --target vllm-openai-zen .
211
+```
212
+
213
+The resulting image accepts the same arguments and environment variables as `vllm-openai` (see [Launching the OpenAI server](#launching-the-openai-server) below); no extra flag is needed to engage Zen optimizations. See [AMD Zen optimizations](cpu.md#amd-zen-optimizations) for runtime behavior and the supported-dtype caveats.
214
+
215
+#### Launching the OpenAI server {#launching-the-openai-server}
216
217
```bash
218
docker run --rm \
@@ -216,5 +228,36 @@ docker run --rm \
228
```
229
230
--8<-- [end:build-image-from-source]
231
+--8<-- [start:amd-zen-optimizations]
232
+
233
+On AMD Zen CPUs, vLLM auto-selects `ZenCpuPlatform` (a subclass of `CpuPlatform`) which dispatches linear layers through [`zentorch`](https://github.com/amd/ZenDNN-pytorch-plugin)'s ZenDNN-optimized kernels. See the FAQ entry [How do I enable AMD Zen optimizations?](#how-do-i-enable-amd-zen-optimizations) for the install command.
234
+
235
+### Detection rules
236
+
237
+`ZenCpuPlatform` is selected when **all** of the following hold:
238
+
239
+- vLLM is built for CPU
240
+- `/proc/cpuinfo` reports `AuthenticAMD` and `avx512`
241
+- `import zentorch` succeeds
242
+
243
+Otherwise, vLLM falls back to the default `CpuPlatform` (oneDNN / sgl-kernel paths).
244
+
245
+### Supported dtypes
246
+
247
+`float16` is **not** supported on `ZenCpuPlatform`. `ZenCpuPlatform.supported_dtypes` advertises only `bfloat16` and `float32`, so models declared with `torch_dtype=float16` are auto-downcast to `bfloat16` at load time with the standard `"Your device 'cpu' doesn't support torch.float16. Falling back to torch.bfloat16 for compatibility."` warning emitted from `vllm/config/model.py`.
248
+
249
+### Environment variables
250
+
251
+- `VLLM_ZENTORCH_WEIGHT_PREPACK` (default `1`): eagerly prepacks linear weights into ZenDNN's blocked layout at model load time, eliminating per-inference layout conversion overhead. Set to `0` to disable.
252
+
253
+### Docker
254
+
255
+The `vllm-openai-zen` Docker target (in `docker/Dockerfile.cpu`) extends the default `vllm-openai` image with `vllm[zen]`. Build it with `docker build -f docker/Dockerfile.cpu --target vllm-openai-zen .` — see [Building with AMD Zen optimizations](#building-with-amd-zen-optimizations) for the full command and run instructions.
256
+
257
+### Reference
258
+
259
+For the design rationale, see [RFC #35089: In-Tree AMD Zen CPU Backend via zentorch](https://github.com/vllm-project/vllm/issues/35089).
260
+
261
+--8<-- [end:amd-zen-optimizations]
262
--8<-- [start:extra-information]
263
--8<-- [end:extra-information]
docs/models/hardware_supported_models/cpu.md
+3
@@ -1,5 +1,8 @@
1
# CPU - Intel® Xeon®
2
3
+!!! note "AMD Zen CPUs"
4
+ On AMD Zen 4 / Zen 5 CPUs, AMD Zen optimizations are auto-enabled when the [`zentorch`](https://github.com/amd/ZenDNN-pytorch-plugin) package is installed. All models supported by vLLM on CPU are supported on AMD Zen as well; model compatibility does not change. This page reflects the current CPU reference validation matrix on Intel systems. See [AMD Zen optimizations](../../getting_started/installation/cpu.md#amd-zen-optimizations) for details.
5
+
6
## Validated Hardware
7
8
| Hardware |
tests/model_executor/test_cpu_unquantized_gemm_dispatch.py
+23
@@ -66,3 +66,26 @@ def test_dispatch_cpu_unquantized_gemm_zen_remove_weight(monkeypatch):
66
utils.dispatch_cpu_unquantized_gemm(layer, remove_weight=True)
67
68
assert layer.weight.numel() == 0
69
+
70
+
71
+@pytest.mark.usefixtures("_mock_zentorch_linear_unary")
72
+def test_dispatch_cpu_unquantized_gemm_logs_zentorch_dispatch(monkeypatch):
73
+ monkeypatch.setattr(current_platform, "is_zen_cpu", lambda: True)
74
+ expected_prepacked = bool(utils.envs.VLLM_ZENTORCH_WEIGHT_PREPACK) and hasattr(
75
+ torch.ops.zentorch, "zentorch_weight_prepack_for_linear"
76
+ )
77
+
78
+ log_calls = []
79
+ monkeypatch.setattr(
80
+ utils.logger, "debug_once", lambda *args: log_calls.append(args)
81
+ )
82
+
83
+ layer = torch.nn.Linear(16, 8, bias=True)
84
+ utils.dispatch_cpu_unquantized_gemm(layer, remove_weight=False)
85
+
86
+ assert log_calls == [
87
+ (
88
+ "CPU unquantized GEMM dispatch: using zentorch_linear_unary (prepacked=%s)",
89
+ expected_prepacked,
90
+ )
91
+ ]
vllm/model_executor/layers/utils.py
+11
@@ -272,6 +272,10 @@ def dispatch_cpu_unquantized_gemm(
272
)
273
if remove_weight:
274
layer.weight = torch.nn.Parameter(torch.empty(0), requires_grad=False)
275
+ logger.debug_once(
276
+ "CPU unquantized GEMM dispatch: using zentorch_linear_unary (prepacked=%s)",
277
+ is_prepacked,
278
+ )
279
return
280
281
if envs.VLLM_CPU_SGL_KERNEL and check_cpu_sgl_kernel(N, K, dtype):
@@ -285,6 +289,9 @@ def dispatch_cpu_unquantized_gemm(
289
)
290
if remove_weight:
291
layer.weight = torch.nn.Parameter(torch.empty(0), requires_grad=False)
292
+ logger.debug_once(
293
+ "CPU unquantized GEMM dispatch: using sgl-kernel weight_packed_linear"
294
+ )
295
return
296
elif (
297
ops._supports_onednn
@@ -296,6 +303,7 @@ def dispatch_cpu_unquantized_gemm(
303
layer.cpu_linear = lambda x, weight, bias: ops.onednn_mm(handler, x, bias)
304
if remove_weight:
305
layer.weight = torch.nn.Parameter(torch.empty(0), requires_grad=False)
306
+ logger.debug_once("CPU unquantized GEMM dispatch: using oneDNN onednn_mm")
307
return
308
except RuntimeError as e:
309
logger.warning_once(
@@ -307,6 +315,9 @@ def dispatch_cpu_unquantized_gemm(
315
layer.cpu_linear = lambda x, weight, bias: torch.nn.functional.linear(
316
x, weight, bias
317
)
318
+ logger.debug_once(
319
+ "CPU unquantized GEMM dispatch: using torch.nn.functional.linear (fallback)"
320
+ )
321
322
323
def cpu_unquantized_gemm(
vllm/platforms/__init__.py
+1
-1
@@ -187,7 +187,7 @@ def cpu_platform_plugin() -> str | None:
187
try:
188
import zentorch # noqa: F401
189
190
- logger.debug(
190
+ logger.info(
191
"AMD Zen CPU detected with zentorch installed, using ZenCpuPlatform."
192
)
193
return "vllm.platforms.zen_cpu.ZenCpuPlatform"