A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 01:00 UTC
[bugfix] Fix auto thread-binding when world_size > 1 in CPU backend and refactor code (#21032)
Signed-off-by: jiang1.li <jiang1.li@intel.com>
Li, Jiang committed
Jul 19, 2025 at 20:13 UTC
e3a0e43d7f98fdd9631e5129005473eb25b98d7b
7 files changed
+144
-150
.buildkite/scripts/hardware_ci/run-cpu-test.sh
+2
-2
@@ -24,8 +24,8 @@ numactl -C "$CORE_RANGE" -N "$NUMA_NODE" docker build --tag cpu-test-"$NUMA_NODE
24
numactl -C "$CORE_RANGE" -N "$NUMA_NODE" docker build --build-arg VLLM_CPU_DISABLE_AVX512="true" --tag cpu-test-"$NUMA_NODE"-avx2 --target vllm-test -f docker/Dockerfile.cpu .
25
26
# Run the image, setting --shm-size=4g for tensor parallel.
27
-docker run -itd --cpuset-cpus="$CORE_RANGE" --cpuset-mems="$NUMA_NODE" --entrypoint /bin/bash -v ~/.cache/huggingface:/root/.cache/huggingface --privileged=true -e HF_TOKEN --env VLLM_CPU_KVCACHE_SPACE=4 --env VLLM_CPU_OMP_THREADS_BIND="$OMP_CORE_RANGE" --env VLLM_CPU_CI_ENV=1 --shm-size=4g --name cpu-test-"$NUMA_NODE" cpu-test-"$NUMA_NODE"
28
-docker run -itd --cpuset-cpus="$CORE_RANGE" --cpuset-mems="$NUMA_NODE" --entrypoint /bin/bash -v ~/.cache/huggingface:/root/.cache/huggingface --privileged=true -e HF_TOKEN --env VLLM_CPU_KVCACHE_SPACE=4 --env VLLM_CPU_OMP_THREADS_BIND="$OMP_CORE_RANGE" --env VLLM_CPU_CI_ENV=1 --shm-size=4g --name cpu-test-"$NUMA_NODE"-avx2 cpu-test-"$NUMA_NODE"-avx2
27
+docker run -itd --cpuset-cpus="$CORE_RANGE" --cpuset-mems="$NUMA_NODE" --entrypoint /bin/bash -v ~/.cache/huggingface:/root/.cache/huggingface --privileged=true -e HF_TOKEN --env VLLM_CPU_KVCACHE_SPACE=4 --env VLLM_CPU_CI_ENV=1 --shm-size=4g --name cpu-test-"$NUMA_NODE" cpu-test-"$NUMA_NODE"
28
+docker run -itd --cpuset-cpus="$CORE_RANGE" --cpuset-mems="$NUMA_NODE" --entrypoint /bin/bash -v ~/.cache/huggingface:/root/.cache/huggingface --privileged=true -e HF_TOKEN --env VLLM_CPU_KVCACHE_SPACE=4 --env VLLM_CPU_CI_ENV=1 --shm-size=4g --name cpu-test-"$NUMA_NODE"-avx2 cpu-test-"$NUMA_NODE"-avx2
29
30
function cpu_tests() {
31
set -e
docs/getting_started/installation/cpu.md
+7
-3
@@ -94,8 +94,8 @@ Currently, there are no pre-built CPU wheels.
94
## Related runtime environment variables
95
96
- `VLLM_CPU_KVCACHE_SPACE`: specify the KV Cache size (e.g, `VLLM_CPU_KVCACHE_SPACE=40` means 40 GiB space for KV cache), larger setting will allow vLLM running more requests in parallel. This parameter should be set based on the hardware configuration and memory management pattern of users. Default value is `0`.
97
-- `VLLM_CPU_OMP_THREADS_BIND`: specify the CPU cores dedicated to the OpenMP threads. For example, `VLLM_CPU_OMP_THREADS_BIND=0-31` means there will be 32 OpenMP threads bound on 0-31 CPU cores. `VLLM_CPU_OMP_THREADS_BIND=0-31|32-63` means there will be 2 tensor parallel processes, 32 OpenMP threads of rank0 are bound on 0-31 CPU cores, and the OpenMP threads of rank1 are bound on 32-63 CPU cores. By setting to `auto`, the OpenMP threads of each rank are bound to the CPU cores in each NUMA node. By setting to `all`, the OpenMP threads of each rank uses all CPU cores available on the system. Default value is `auto`.
98
-- `VLLM_CPU_NUM_OF_RESERVED_CPU`: specify the number of CPU cores which are not dedicated to the OpenMP threads for each rank. The variable only takes effect when VLLM_CPU_OMP_THREADS_BIND is set to `auto`. Default value is `0`.
97
+- `VLLM_CPU_OMP_THREADS_BIND`: specify the CPU cores dedicated to the OpenMP threads, can be set as CPU id lists or `auto` (by default). For example, `VLLM_CPU_OMP_THREADS_BIND=0-31` means there will be 32 OpenMP threads bound on 0-31 CPU cores. `VLLM_CPU_OMP_THREADS_BIND=0-31|32-63` means there will be 2 tensor parallel processes, 32 OpenMP threads of rank0 are bound on 0-31 CPU cores, and the OpenMP threads of rank1 are bound on 32-63 CPU cores. By setting to `auto`, the OpenMP threads of each rank are bound to the CPU cores in each NUMA node respectively.
98
+- `VLLM_CPU_NUM_OF_RESERVED_CPU`: specify the number of CPU cores which are not dedicated to the OpenMP threads for each rank. The variable only takes effect when VLLM_CPU_OMP_THREADS_BIND is set to `auto`. Default value is `None`. If the value is not set and use `auto` thread binding, no CPU will be reserved for `world_size == 1`, 1 CPU per rank will be reserved for `world_size > 1`.
99
- `VLLM_CPU_MOE_PREPACK` (x86 only): whether to use prepack for MoE layer. This will be passed to `ipex.llm.modules.GatedMLPMOE`. Default is `1` (True). On unsupported CPUs, you might need to set this to `0` (False).
100
- `VLLM_CPU_SGL_KERNEL` (x86 only, Experimental): whether to use small-batch optimized kernels for linear layer and MoE layer, especially for low-latency requirements like online serving. The kernels require AMX instruction set, BFloat16 weight type and weight shapes divisible by 32. Default is `0` (False).
101
@@ -123,9 +123,13 @@ export VLLM_CPU_NUM_OF_RESERVED_CPU=1
123
vllm serve facebook/opt-125m --dtype=bfloat16
124
```
125
126
+Note, it is recommended to manually reserve 1 CPU for vLLM front-end process when `world_size == 1`.
127
+
128
### How to decide `VLLM_CPU_OMP_THREADS_BIND`?
129
128
-- Bind each OpenMP thread to a dedicated physical CPU core respectively, or use auto thread binding feature by default. On a hyper-threading enabled platform with 16 logical CPU cores / 8 physical CPU cores:
130
+- Default `auto` thread-binding is recommended for most cases. Ideally, each OpenMP thread will be bound to a dedicated physical core respectively, threads of each rank will be bound to a same NUMA node respectively, and 1 CPU per rank will be reserved for other vLLM components when `world_size > 1`. If have any performance problems or unexpected binding behaviours, please try to bind threads as following.
131
+
132
+- On a hyper-threading enabled platform with 16 logical CPU cores / 8 physical CPU cores:
133
134
??? console "Commands"
135
requirements/cpu.txt
-2
@@ -24,6 +24,4 @@ datasets # for benchmark scripts
24
# Intel Extension for PyTorch, only for x86_64 CPUs
25
intel-openmp==2024.2.1; platform_machine == "x86_64"
26
intel_extension_for_pytorch==2.6.0; platform_machine == "x86_64" # torch>2.6.0+cpu has performance regression on x86 platform, see https://github.com/pytorch/pytorch/pull/151218
27
-py-libnuma; platform_system != "Darwin"
28
-psutil; platform_system != "Darwin"
27
triton==3.2.0; platform_machine == "x86_64" # Triton is required for torch 2.6+cpu, as it is imported in torch.compile.
vllm/envs.py
+3
-2
@@ -44,7 +44,7 @@ if TYPE_CHECKING:
44
VLLM_PP_LAYER_PARTITION: Optional[str] = None
45
VLLM_CPU_KVCACHE_SPACE: int = 0
46
VLLM_CPU_OMP_THREADS_BIND: str = ""
47
- VLLM_CPU_NUM_OF_RESERVED_CPU: int = 0
47
+ VLLM_CPU_NUM_OF_RESERVED_CPU: Optional[int] = None
48
VLLM_CPU_MOE_PREPACK: bool = True
49
VLLM_CPU_SGL_KERNEL: bool = False
50
VLLM_XLA_CACHE_PATH: str = os.path.join(VLLM_CACHE_ROOT, "xla_cache")
@@ -442,7 +442,8 @@ environment_variables: dict[str, Callable[[], Any]] = {
442
# (CPU backend only) CPU cores not used by OMP threads .
443
# Those CPU cores will not be used by OMP threads of a rank.
444
"VLLM_CPU_NUM_OF_RESERVED_CPU":
445
- lambda: int(os.getenv("VLLM_CPU_NUM_OF_RESERVED_CPU", "0")),
445
+ lambda: int(os.getenv("VLLM_CPU_NUM_OF_RESERVED_CPU", "0"))
446
+ if "VLLM_CPU_NUM_OF_RESERVED_CPU" in os.environ else None,
447
448
# (CPU backend only) whether to use prepack for MoE layer. This will be
449
# passed to ipex.llm.modules.GatedMLPMOE. On unsupported CPUs, you might
vllm/platforms/cpu.py
+64
@@ -1,9 +1,12 @@
1
# SPDX-License-Identifier: Apache-2.0
2
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
4
+import json
5
import os
6
import platform
7
+import subprocess
8
import sys
9
+from dataclasses import dataclass
10
from importlib.util import find_spec
11
from typing import TYPE_CHECKING, Optional
12
@@ -31,6 +34,35 @@ def get_max_threads(pid=0):
34
raise NotImplementedError("Unsupported OS")
35
36
37
+@dataclass
38
+class LogicalCPUInfo:
39
+ id: int = -1
40
+ physical_core: int = -1
41
+ numa_node: int = -1
42
+
43
+ @classmethod
44
+ def _int(cls, value: str) -> int:
45
+ try:
46
+ int_value = int(value)
47
+ except Exception:
48
+ int_value = -1
49
+ return int_value
50
+
51
+ @staticmethod
52
+ def json_decoder(obj_dict: dict):
53
+ id = obj_dict.get("cpu")
54
+ physical_core = obj_dict.get("core")
55
+ numa_node = obj_dict.get("node")
56
+
57
+ if not (id is None or physical_core is None or numa_node is None):
58
+ return LogicalCPUInfo(
59
+ id=LogicalCPUInfo._int(id),
60
+ physical_core=LogicalCPUInfo._int(physical_core),
61
+ numa_node=LogicalCPUInfo._int(numa_node))
62
+ else:
63
+ return obj_dict
64
+
65
+
66
class CpuPlatform(Platform):
67
_enum = PlatformEnum.CPU
68
device_name: str = "cpu"
@@ -240,6 +272,38 @@ class CpuPlatform(Platform):
272
vllm_config.scheduler_config.max_model_len,
273
DEFAULT_MAX_NUM_BATCHED_TOKENS)
274
275
+ @classmethod
276
+ def get_allowed_cpu_memory_node_list(
277
+ cls) -> tuple[list[int], list[LogicalCPUInfo]]:
278
+ assert platform.system() == "Linux"
279
+
280
+ # Init LogicalCPUInfo from lscpu
281
+ lscpu_output = subprocess.check_output("lscpu -J -e=CPU,CORE,NODE",
282
+ shell=True,
283
+ text=True)
284
+ logical_cpu_list: list[LogicalCPUInfo] = json.loads(
285
+ lscpu_output, object_hook=LogicalCPUInfo.json_decoder)['cpus']
286
+
287
+ # Filter CPUs with invalid attributes
288
+ logical_cpu_list = [
289
+ x for x in logical_cpu_list
290
+ if -1 not in (x.id, x.physical_core, x.numa_node)
291
+ ]
292
+
293
+ # Filter allowed CPUs
294
+ allowed_cpu_id_list = os.sched_getaffinity(0)
295
+ logical_cpu_list = [
296
+ x for x in logical_cpu_list if x.id in allowed_cpu_id_list
297
+ ]
298
+
299
+ # Get allowed NUMA nodes
300
+ allowed_numa_nodes = set()
301
+ for x in logical_cpu_list:
302
+ allowed_numa_nodes.add(x.numa_node) # type: ignore
303
+ allowed_numa_nodes_list = sorted(allowed_numa_nodes)
304
+
305
+ return allowed_numa_nodes_list, logical_cpu_list
306
+
307
@classmethod
308
def is_pin_memory_available(cls) -> bool:
309
logger.warning("Pin memory is not supported on CPU.")
vllm/v1/worker/cpu_model_runner.py
+4
-3
@@ -45,9 +45,10 @@ class CPUModelRunner(GPUModelRunner):
45
if k.endswith("_cpu_tensor") and isinstance(v, torch.Tensor):
46
replace_tensor(self.input_batch, k, k[:-11])
47
48
- for k, v in vars(self.input_batch.block_table).items():
49
- if k.endswith("_cpu") and isinstance(v, torch.Tensor):
50
- replace_tensor(self.input_batch.block_table, k, k[:-4])
48
+ for block_table in self.input_batch.block_table.block_tables:
49
+ for k, v in vars(block_table).items():
50
+ if k.endswith("_cpu") and isinstance(v, torch.Tensor):
51
+ replace_tensor(block_table, k, k[:-4])
52
53
def load_model(self, eep_scale_up: bool = False) -> None:
54
logger.info("Starting to load model %s...", self.model_config.model)
vllm/v1/worker/cpu_worker.py
+64
-138
@@ -1,8 +1,8 @@
1
# SPDX-License-Identifier: Apache-2.0
2
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
import os
4
-from importlib import util
5
-from typing import Optional
4
+import platform
5
+from typing import Callable, Optional
6
7
import torch
8
@@ -12,21 +12,14 @@ from vllm.distributed.parallel_state import get_pp_group, get_tp_group
12
from vllm.logger import init_logger
13
from vllm.model_executor.utils import set_random_seed
14
from vllm.platforms import CpuArchEnum, current_platform
15
+from vllm.platforms.cpu import CpuPlatform, LogicalCPUInfo
16
from vllm.sequence import IntermediateTensors
16
-from vllm.utils import PlaceholderModule
17
from vllm.v1.core.sched.output import SchedulerOutput
18
from vllm.v1.outputs import ModelRunnerOutput
19
from vllm.v1.worker.cpu_model_runner import CPUModelRunner
20
from vllm.v1.worker.gpu_worker import (Worker,
21
init_worker_distributed_environment)
22
23
-try:
24
- import psutil
25
- from numa import info
26
-except ImportError:
27
- psutil = PlaceholderModule("psutil") # type: ignore[assignment]
28
- numa = PlaceholderModule("numa") # type: ignore[assignment]
29
-
23
logger = init_logger(__name__)
24
25
@@ -45,20 +38,21 @@ class CPUWorker(Worker):
38
is_driver_worker=is_driver_worker)
39
40
self.parallel_config.disable_custom_all_reduce = True
48
- self.manually_bind_threads_suggestion = (
49
- "To get better performance, please try to manually bind threads.")
41
42
def init_device(self):
43
# Setup OpenMP threads affinity.
44
omp_cpuids = envs.VLLM_CPU_OMP_THREADS_BIND
54
- self.local_omp_cpuid = "all"
55
- if omp_cpuids == "auto":
45
+ if omp_cpuids == "auto" and platform.system() == "Linux":
46
if current_platform.get_cpu_architecture() == CpuArchEnum.POWERPC:
57
- self.local_omp_cpuid = (
58
- self.get_cpus_id_binding_based_on_numa_nodes_ppc64le())
47
+ # For POWERPC SMT-8/4/2
48
+ self.local_omp_cpuid = self._get_autobind_cpu_ids(
49
+ lambda cpus: [cpu for cpu in cpus if cpu.id % 8 < 4])
50
+ elif current_platform.get_cpu_architecture() == CpuArchEnum.X86:
51
+ # For x86 SMT-2, use 1 CPU per core
52
+ self.local_omp_cpuid = self._get_autobind_cpu_ids(
53
+ lambda cpus: cpus[-1:])
54
else:
60
- self.local_omp_cpuid = (
61
- self.get_cpus_id_binding_based_on_numa_nodes())
55
+ self.local_omp_cpuid = "all"
56
else:
57
self.local_omp_cpuid = omp_cpuids.split("|")[self.rank]
58
@@ -122,126 +116,58 @@ class CPUWorker(Worker):
116
assert isinstance(output, ModelRunnerOutput)
117
return output if self.is_driver_worker else None
118
125
- def warn_inability_to_detect_numa(self) -> None:
126
- logger.warning(
127
- "Auto thread-binding failed due to the "
128
- "inability to detect numa nodes. %s",
129
- self.manually_bind_threads_suggestion)
130
-
131
- def warn_lack_of_numa_and_psutil(self) -> None:
132
- logger.warning(
133
- "Auto thread-binding failed due to "
134
- "the lack of package numa and psutil. %s",
135
- self.manually_bind_threads_suggestion)
136
-
137
- def warn_world_size_too_large(self, world_size: int,
138
- node_to_cpus_len: int) -> None:
139
- logger.warning(
140
- "Auto thread-binding failed due to "
141
- "world size: %d being larger than "
142
- "allowed NUMA nodes number: %d. %s", world_size, node_to_cpus_len,
143
- self.manually_bind_threads_suggestion)
144
-
145
- def get_cpus_allow_list_and_numa_size(self):
146
- cpus_allow_list = psutil.Process().cpu_affinity()
147
- numa_size = info.get_num_configured_nodes()
148
- return cpus_allow_list, numa_size
149
-
150
- def auto_thread_binding_based_on_numa_nodes(self, world_size: int,
151
- rank_to_cpus: str) -> str:
152
- cpu_count = psutil.cpu_count(logical=False)
153
- cpus_allow_list, numa_size = self.get_cpus_allow_list_and_numa_size()
154
- if not numa_size:
155
- self.warn_inability_to_detect_numa()
156
- return rank_to_cpus
157
-
158
- cpu_count_per_numa = cpu_count // numa_size
159
- num_of_reserved_cpu = min(envs.VLLM_CPU_NUM_OF_RESERVED_CPU,
160
- cpu_count_per_numa // 2)
161
-
162
- node_to_cpus = []
163
- for i in range(numa_size):
164
- node_intersect = set(
165
- info.node_to_cpus(i)).intersection(cpus_allow_list)
166
- if bool(node_intersect):
167
- node_to_cpus.append(list(node_intersect))
168
-
169
- node_to_cpus_len = len(node_to_cpus)
170
- if world_size > node_to_cpus_len:
171
- self.warn_world_size_too_large(world_size, node_to_cpus_len)
172
- else:
173
- end = cpu_count_per_numa - num_of_reserved_cpu
174
- rank_to_cpus_list = node_to_cpus[self.rank][:end]
175
- rank_to_cpus = ','.join(str(x) for x in rank_to_cpus_list)
176
- logger.info("auto thread-binding list: %s", rank_to_cpus)
177
- return rank_to_cpus
178
-
179
- def libnuma_and_psutil_found(self) -> bool:
180
- libnuma_found = util.find_spec("numa") is not None
181
- psutil_found = util.find_spec("psutil") is not None
182
-
183
- return libnuma_found and psutil_found
184
-
185
- def get_cpus_id_binding_based_on_numa_nodes(self) -> str:
186
- """Return CPUs id binding based on NUMA nodes.
119
+ def _get_autobind_cpu_ids(
120
+ self, cpu_selector: Callable[[list[LogicalCPUInfo]],
121
+ list[LogicalCPUInfo]]
122
+ ) -> str:
123
"""
188
- rank_to_cpus = self.local_omp_cpuid
189
- # Setup OpenMP thread affinity based on NUMA nodes automatically
190
- world_size = self.vllm_config.parallel_config.world_size
191
- if self.libnuma_and_psutil_found():
192
- rank_to_cpus = self.auto_thread_binding_based_on_numa_nodes(
193
- world_size, rank_to_cpus)
194
- else:
195
- self.warn_lack_of_numa_and_psutil()
196
- return rank_to_cpus
197
-
198
- def select_threads_per_power_core(self,
199
- node_cpu_ids: list[int]) -> list[int]:
200
- return [cpu for cpu in node_cpu_ids if cpu % 8 < 4]
201
-
202
- def auto_thread_binding_based_on_numa_nodes_ppc64le(
203
- self, world_size: int, rank_to_cpus: str) -> str:
204
- cpus_allow_list, numa_size = self.get_cpus_allow_list_and_numa_size()
205
- if not numa_size:
206
- self.warn_inability_to_detect_numa()
207
- return rank_to_cpus
208
-
209
- node_to_cpus = []
210
- for i in range(numa_size):
211
- node_intersect = set(
212
- info.node_to_cpus(i)).intersection(cpus_allow_list)
213
- if bool(node_intersect):
214
- node_to_cpus.append(sorted(list(node_intersect)))
215
-
216
- node_to_cpus_len = len(node_to_cpus)
217
- if world_size > node_to_cpus_len:
218
- self.warn_world_size_too_large(world_size, node_to_cpus_len)
219
- else:
220
- node_cpus_this_rank = node_to_cpus[self.rank]
221
- node_cpus_this_rank = self.select_threads_per_power_core(
222
- node_cpus_this_rank)
223
- cpu_count_per_numa = len(node_cpus_this_rank)
224
- num_of_reserved_cpu = min(envs.VLLM_CPU_NUM_OF_RESERVED_CPU,
225
- cpu_count_per_numa // 2)
226
- end = cpu_count_per_numa - num_of_reserved_cpu
227
- rank_to_cpus_list = node_cpus_this_rank[:end]
228
- rank_to_cpus = ','.join(str(x) for x in rank_to_cpus_list)
229
- logger.info("ppc64le thread-binding list: %s", rank_to_cpus)
230
- return rank_to_cpus
231
-
232
- def get_cpus_id_binding_based_on_numa_nodes_ppc64le(self) -> str:
233
- """
234
- Power (ppc64le) specific: Selects a subset of threads per core for
235
- each NUMA node.This is robust to SMT mode (SMT-8, SMT-4, etc)
236
- because the OS only exposes available threads.This maximizes
237
- performance by avoiding oversubscription of logical CPUs on Power.
124
+ Return CPU ids to bind based on NUMA nodes.
125
+ Currently for rank N, only CPU ids on the N-th node in available NUMA
126
+ node list will be selected.
127
+ Args:
128
+ cpu_selector: a callable object to select CPUs from a CPU list
129
+ of a physical core. The input is a LogicalCPUInfo list, sorted by
130
+ the LogicalCPUInfo.id. A selected LogicalCPUInfo list should be
131
+ returned.
132
"""
133
240
- rank_to_cpus = self.local_omp_cpuid
241
- world_size = self.vllm_config.parallel_config.world_size
242
- if self.libnuma_and_psutil_found():
243
- rank_to_cpus = self.auto_thread_binding_based_on_numa_nodes_ppc64le(
244
- world_size, rank_to_cpus)
245
- else:
246
- self.warn_lack_of_numa_and_psutil()
247
- return rank_to_cpus
134
+ allowed_numa_nodes, logical_cpu_list = \
135
+ CpuPlatform.get_allowed_cpu_memory_node_list()
136
+ assert len(allowed_numa_nodes) >= self.parallel_config.world_size, (
137
+ f"No enough allowed NUMA nodes to bind threads of "
138
+ f"{self.parallel_config.world_size} CPUWorkers. "
139
+ f"Allowed NUMA nodes are {allowed_numa_nodes}. "
140
+ "Please try to bind threads manually.")
141
+
142
+ # Get CPUs on NUMA node `allowed_numa_nodes[local_rank]``
143
+ selected_numa_node = allowed_numa_nodes[
144
+ self.local_rank] # type: ignore
145
+ logical_cpu_list = [
146
+ x for x in logical_cpu_list if x.numa_node == selected_numa_node
147
+ ]
148
+
149
+ # Select CPUs from each physical core via cpu_selector
150
+ core_to_cpus: dict[int, list[LogicalCPUInfo]] = {}
151
+ for cpu_info in logical_cpu_list:
152
+ if cpu_info.physical_core not in core_to_cpus:
153
+ core_to_cpus[cpu_info.physical_core] = []
154
+ core_to_cpus[cpu_info.physical_core].append(cpu_info)
155
+ logical_cpu_list = []
156
+ for cpu_list in core_to_cpus.values():
157
+ cpu_list = sorted(cpu_list, key=lambda x: x.id)
158
+ logical_cpu_list.extend(cpu_selector(cpu_list))
159
+ logical_cpu_list = sorted(logical_cpu_list, key=lambda x: x.id)
160
+
161
+ # Reserve CPUs for other processes
162
+ reserve_cpu_num = envs.VLLM_CPU_NUM_OF_RESERVED_CPU
163
+ if reserve_cpu_num is None:
164
+ reserve_cpu_num = 1 if self.parallel_config.world_size > 1 else 0
165
+ assert len(logical_cpu_list) > reserve_cpu_num, (
166
+ f"VLLM_CPU_NUM_OF_RESERVED_CPU ({reserve_cpu_num}) "
167
+ f"should less than {len(logical_cpu_list)}.")
168
+ if reserve_cpu_num != 0:
169
+ logical_cpu_list = logical_cpu_list[:-reserve_cpu_num]
170
+
171
+ logger.info("auto thread-binding list (id, physical core): %s",
172
+ [(x.id, x.physical_core) for x in logical_cpu_list])
173
+ return ",".join([str(x.id) for x in logical_cpu_list])