[CPU] Align scheduler and NIXL CPU affinity per local rank (#53636)
Signed-off-by: Li, Tianmu <tianmu.li@intel.com> Co-authored-by: OpenAI Codex <noreply@openai.com> Co-authored-by: Li, Jiang <jiang1.li@intel.com>
Tianmu Li committed
Sep 17, 2026 at 01:43 UTC
23cd0346cec1cb98862d93e0c1bfa490f827e6a3
3 files changed
+6
-5
docs/getting_started/installation/cpu.md
+2
-2
@@ -150,7 +150,7 @@ VLLM_USE_PRECOMPILED=1 VLLM_PRECOMPILED_WHEEL_VARIANT=cpu VLLM_TARGET_DEVICE=cpu
150
151
- `VLLM_CPU_KVCACHE_SPACE`: specify the KV Cache size (e.g, `VLLM_CPU_KVCACHE_SPACE=40` means 40 GiB space for KV cache), larger setting will allow vLLM to run more requests in parallel. This parameter should be set based on the hardware configuration and memory management pattern of users. Default value is `0`.
152
- `VLLM_CPU_OMP_THREADS_BIND`: specify the CPU cores dedicated to the OpenMP threads, can be set as CPU id lists, `auto` (by default), or `nobind` (to disable binding to individual CPU cores and to inherit user-defined OpenMP variables). For example, `VLLM_CPU_OMP_THREADS_BIND=0-31` means there will be 32 OpenMP threads bound on 0-31 CPU cores. `VLLM_CPU_OMP_THREADS_BIND=0-31|32-63` means there will be 2 tensor parallel processes, 32 OpenMP threads of rank0 are bound on 0-31 CPU cores, and the OpenMP threads of rank1 are bound on 32-63 CPU cores. By setting to `auto`, the OpenMP threads of each rank are bound to the CPU cores in each NUMA node respectively. If set to `nobind`, the number of OpenMP threads is determined by the standard `OMP_NUM_THREADS` environment variable.
153
-- `VLLM_CPU_NUM_OF_RESERVED_CPU`: specify the number of CPU cores which are not dedicated to the OpenMP threads for each rank. The variable only takes effect when VLLM_CPU_OMP_THREADS_BIND is set to `auto`. Default value is `None`. If the value is not set and use `auto` thread binding, no CPU will be reserved for `world_size == 1`, 1 CPU per rank will be reserved for `world_size > 1`.
153
+- `VLLM_CPU_NUM_OF_RESERVED_CPU`: specify the total number of CPU cores which are not dedicated to OpenMP threads. The variable only takes effect when `VLLM_CPU_OMP_THREADS_BIND` is set to `auto`. By default, x86, ARM, and RISC-V reserve `local_world_size` CPUs in total, while PowerPC and S390X retain a one-CPU default. When KV transfer is enabled, one additional CPU is reserved per local rank. Reserved CPUs are distributed across the local ranks.
154
- `CPU_VISIBLE_MEMORY_NODES`: specify visible NUMA memory nodes for vLLM CPU workers, similar to ```CUDA_VISIBLE_DEVICES```. The variable only takes effect when VLLM_CPU_OMP_THREADS_BIND is set to `auto`. The variable provides more control for the auto thread-binding feature, such as masking nodes and changing nodes binding sequence.
155
- `VLLM_ZENTORCH_WEIGHT_PREPACK` (AMD Zen only): when `ZenCpuPlatform` is active, eagerly prepack linear weights into ZenDNN's blocked layout at model load time, eliminating per-inference layout conversion overhead. Default is `1` (enabled). See [AMD Zen optimizations](#amd-zen-optimizations).
156
@@ -253,7 +253,7 @@ See [AMD Zen optimizations](#amd-zen-optimizations) for detection rules, support
253
254
### How to decide `VLLM_CPU_OMP_THREADS_BIND`?
255
256
-- Default `auto` thread-binding is recommended for most cases. Ideally, each OpenMP thread will be bound to a dedicated physical core respectively, threads of each rank will be bound to the same NUMA node respectively, and 1 CPU per rank will be reserved for other vLLM components when `world_size > 1`. If you have any performance problems or unexpected binding behaviours, please try to bind threads as following.
256
+- Default `auto` thread-binding is recommended for most cases. Ideally, each OpenMP thread will be bound to a dedicated physical core respectively, threads of each rank will be bound to the same NUMA node respectively, and the default reserved CPUs will be kept for other vLLM components. On x86, ARM, and RISC-V, `local_world_size` CPUs are reserved in total, plus one additional CPU per local rank when KV transfer is enabled. If you have any performance problems or unexpected binding behaviours, please try to bind threads as following.
257
258
- On a hyper-threading enabled platform with 16 logical CPU cores / 8 physical CPU cores:
259
vllm/platforms/cpu.py
+2
-1
@@ -515,6 +515,7 @@ class CpuPlatform(Platform):
515
if not (os.path.exists(SYS_NODE) and os.path.exists(SYS_CPU)):
516
return []
517
518
+ use_highest_sibling = cls.get_cpu_architecture() == CpuArchEnum.X86
519
core_rsv_for_kv = []
520
for node in os.listdir(SYS_NODE):
521
if not node.startswith("node") or not node[4:].isdigit():
@@ -547,7 +548,7 @@ class CpuPlatform(Platform):
548
else:
549
siblings = [cpu_id]
550
550
- phys = min(siblings)
551
+ phys = max(siblings) if use_highest_sibling else min(siblings)
552
553
if phys not in seen_phys:
554
seen_phys.add(phys)
vllm/utils/ompmultiprocessing.py
+2
-2
@@ -46,12 +46,12 @@ class OMPProcessManager:
46
self.reserve_cpu_num = (
47
self.local_world_size
48
if current_platform.get_cpu_architecture()
49
- in (CpuArchEnum.ARM, CpuArchEnum.RISCV)
49
+ in (CpuArchEnum.X86, CpuArchEnum.ARM, CpuArchEnum.RISCV)
50
else 1
51
)
52
# reserve at one more core for nixl_connector under p/d case
53
if config.kv_transfer_config:
54
- self.reserve_cpu_num += 1
54
+ self.reserve_cpu_num += self.local_world_size
55
56
if envs.VLLM_CPU_NUM_OF_RESERVED_CPU is not None:
57
if self.reserve_cpu_num > envs.VLLM_CPU_NUM_OF_RESERVED_CPU: