@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 75e01a39a1
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 5, 11:00 UTC

[Feature] NUMA binding support for GPU workers (#38635)

Signed-off-by: Shengqi Chen <harry-chen@outlook.com> Co-authored-by: Jason Li <jasonlizhengjian@gmail.com> Co-authored-by: Roger Wang <hey@rogerw.io>

Shengqi Chen committed Apr 9, 2026 at 00:55 UTC 75e01a39a16c1f39d5e2cf37edb6525df1a76c9f
13 files changed +817 -7
docs/configuration/optimization.md
+74
@@ -140,6 +140,80 @@ Data parallelism replicates the entire model across multiple GPU sets and proces
140 Data parallelism can be combined with the other parallelism strategies and is set by `data_parallel_size=N`.
141 Note that MoE layers will be sharded according to the product of the tensor parallel size and data parallel size.
142
143 +### NUMA Binding for Multi-Socket GPU Nodes
144 +
145 +On multi-socket GPU servers, GPU worker processes can lose performance if their
146 +CPU execution and memory allocation drift away from the NUMA node nearest to the
147 +GPU. vLLM can pin each worker with `numactl` before the Python subprocess starts,
148 +so the interpreter, imports, and early allocator state are created with the
149 +desired NUMA policy from the beginning.
150 +
151 +Use `--numa-bind` to enable the feature. By default, vLLM auto-detects the
152 +GPU-to-NUMA mapping and uses `--cpunodebind=<node> --membind=<node>` for each
153 +worker. When you need a custom CPU policy, add `--numa-bind-cpus` and vLLM will
154 +switch to `--physcpubind=<cpu-list> --membind=<node>`.
155 +
156 +These `--numa-bind*` options only apply to GPU execution processes. They do not
157 +configure the CPU backend's separate thread-affinity controls. Automatic
158 +GPU-to-NUMA detection is currently implemented for CUDA/NVML-based platforms;
159 +other GPU backends must provide explicit binding lists if they use these
160 +options.
161 +
162 +`--numa-bind-nodes` takes one non-negative NUMA node index per visible GPU, in
163 +the same order as the GPU indices.
164 +`--numa-bind-cpus` takes one `numactl` CPU list per visible GPU, in the same
165 +order as the GPU indices. Each CPU list must use
166 +`numactl --physcpubind` syntax such as `0-3`, `0,2,4-7`, or `16-31,48-63`.
167 +
168 +```bash
169 +# Auto-detect NUMA nodes for visible GPUs
170 +vllm serve meta-llama/Llama-3.1-8B-Instruct \
171 + --tensor-parallel-size 4 \
172 + --numa-bind
173 +
174 +# Explicit NUMA-node mapping
175 +vllm serve meta-llama/Llama-3.1-8B-Instruct \
176 + --tensor-parallel-size 4 \
177 + --numa-bind \
178 + --numa-bind-nodes 0 0 1 1
179 +
180 +# Explicit CPU pinning, useful for PCT or other high-frequency core layouts
181 +vllm serve meta-llama/Llama-3.1-8B-Instruct \
182 + --tensor-parallel-size 4 \
183 + --numa-bind \
184 + --numa-bind-nodes 0 0 1 1 \
185 + --numa-bind-cpus 0-3 4-7 48-51 52-55
186 +```
187 +
188 +Notes:
189 +
190 +- CLI usage forces multiprocessing to use the `spawn` method automatically. If you enable NUMA binding through the Python API, also set `VLLM_WORKER_MULTIPROC_METHOD=spawn`.
191 +- Automatic detection relies on NVML and NUMA support from the host. If it cannot determine the mapping reliably, pass `--numa-bind-nodes` explicitly.
192 +- Explicit `--numa-bind-nodes` and `--numa-bind-cpus` values must be valid `numactl` inputs. vLLM does a small amount of validation, but the effective binding semantics are still determined by `numactl`.
193 +- The current implementation binds GPU execution processes such as `EngineCore` and multiprocessing workers. It does not apply NUMA binding to frontend API server processes or the DP coordinator.
194 +- In containerized environments, NUMA policy syscalls may require extra permissions, such as `--cap-add SYS_NICE` when running via `docker run`.
195 +
196 +### CPU Backend Thread Affinity
197 +
198 +The CPU backend uses a different mechanism from `--numa-bind`. CPU execution is
199 +configured through CPU-specific environment variables such as
200 +`VLLM_CPU_OMP_THREADS_BIND`, `VLLM_CPU_NUM_OF_RESERVED_CPU`, and
201 +`CPU_VISIBLE_MEMORY_NODES`, rather than the GPU-oriented `--numa-bind*` CLI
202 +options.
203 +
204 +By default, `VLLM_CPU_OMP_THREADS_BIND=auto` derives OpenMP placement from the
205 +available CPU and NUMA topology for each CPU worker. To override the automatic
206 +policy, set `VLLM_CPU_OMP_THREADS_BIND` explicitly using the CPU list format
207 +documented for the CPU backend, or use `nobind` to disable this behavior.
208 +
209 +For the current CPU backend setup and tuning guidance, see:
210 +
211 +- [Related runtime environment variables](../getting_started/installation/cpu.md#related-runtime-environment-variables)
212 +- [How to decide `VLLM_CPU_OMP_THREADS_BIND`](../getting_started/installation/cpu.md#how-to-decide-vllm_cpu_omp_threads_bind)
213 +
214 +The GPU-only `--numa-bind`, `--numa-bind-nodes`, and `--numa-bind-cpus` options
215 +do not configure CPU worker affinity.
216 +
217 ### Batch-level DP for Multi-Modal Encoders
218
219 By default, TP is used to shard the weights of multi-modal encoders just like for language decoders,
tests/engine/test_arg_utils.py
+23
@@ -525,6 +525,29 @@ def test_human_readable_model_len():
525 parser.parse_args(["--max-model-len", invalid])
526
527
528 +def test_numa_bind_args():
529 + parser = EngineArgs.add_cli_args(FlexibleArgumentParser())
530 + args = parser.parse_args(
531 + [
532 + "--numa-bind",
533 + "--numa-bind-nodes",
534 + "0",
535 + "0",
536 + "1",
537 + "1",
538 + "--numa-bind-cpus",
539 + "0-3",
540 + "4-7",
541 + "8-11",
542 + "12-15",
543 + ]
544 + )
545 + engine_args = EngineArgs.from_cli_args(args=args)
546 + assert engine_args.numa_bind is True
547 + assert engine_args.numa_bind_nodes == [0, 0, 1, 1]
548 + assert engine_args.numa_bind_cpus == ["0-3", "4-7", "8-11", "12-15"]
549 +
550 +
551 def test_ir_op_priority():
552 from vllm.config.kernel import IrOpPriorityConfig, KernelConfig
553
tests/utils_/test_numa_utils.py new
+132
@@ -0,0 +1,132 @@
1 +# SPDX-License-Identifier: Apache-2.0
2 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 +
4 +import os
5 +from types import SimpleNamespace
6 +
7 +import pytest
8 +
9 +from vllm.config import ParallelConfig
10 +from vllm.utils import numa_utils
11 +
12 +
13 +def _make_config(**parallel_kwargs):
14 + parallel_defaults = dict(
15 + numa_bind=False,
16 + numa_bind_nodes=None,
17 + numa_bind_cpus=None,
18 + distributed_executor_backend="mp",
19 + data_parallel_backend="mp",
20 + nnodes_within_dp=1,
21 + data_parallel_rank_local=0,
22 + data_parallel_index=0,
23 + pipeline_parallel_size=1,
24 + tensor_parallel_size=1,
25 + )
26 + parallel_defaults.update(parallel_kwargs)
27 + parallel_config = SimpleNamespace(**parallel_defaults)
28 + return SimpleNamespace(parallel_config=parallel_config)
29 +
30 +
31 +def test_get_numactl_args_with_node_binding():
32 + vllm_config = _make_config(numa_bind=True, numa_bind_nodes=[0, 1])
33 + assert (
34 + numa_utils._get_numactl_args(vllm_config, local_rank=1)
35 + == "--cpunodebind=1 --membind=1"
36 + )
37 +
38 +
39 +def test_get_numactl_args_with_cpu_binding():
40 + vllm_config = _make_config(
41 + numa_bind=True,
42 + numa_bind_nodes=[0, 1],
43 + numa_bind_cpus=["0-3", "4-7"],
44 + )
45 + assert (
46 + numa_utils._get_numactl_args(vllm_config, local_rank=1)
47 + == "--physcpubind=4-7 --membind=1"
48 + )
49 +
50 +
51 +def test_get_numactl_args_uses_dp_offset():
52 + vllm_config = _make_config(
53 + numa_bind=True,
54 + numa_bind_nodes=[0, 0, 1, 1],
55 + data_parallel_rank_local=1,
56 + pipeline_parallel_size=1,
57 + tensor_parallel_size=2,
58 + )
59 + assert (
60 + numa_utils._get_numactl_args(vllm_config, local_rank=1)
61 + == "--cpunodebind=1 --membind=1"
62 + )
63 +
64 +
65 +def test_get_numactl_args_requires_detectable_nodes(monkeypatch):
66 + vllm_config = _make_config(numa_bind=True)
67 + monkeypatch.setattr(numa_utils, "get_auto_numa_nodes", lambda: None)
68 + with pytest.raises(RuntimeError):
69 + numa_utils._get_numactl_args(vllm_config, local_rank=0)
70 +
71 +
72 +def test_log_numactl_show(monkeypatch):
73 + log_lines = []
74 +
75 + def fake_debug(msg, *args):
76 + log_lines.append(msg % args)
77 +
78 + monkeypatch.setattr(numa_utils.logger, "debug", fake_debug)
79 + monkeypatch.setattr(
80 + numa_utils.subprocess,
81 + "run",
82 + lambda *args, **kwargs: SimpleNamespace(
83 + stdout="policy: bind\nphyscpubind: 0 1 2 3\n", returncode=0
84 + ),
85 + )
86 +
87 + assert numa_utils._log_numactl_show("Worker_0") is True
88 + assert log_lines == [
89 + "Worker_0 affinity: policy: bind, physcpubind: 0 1 2 3",
90 + ]
91 +
92 +
93 +def test_get_numactl_executable_points_to_fixed_wrapper(monkeypatch):
94 + monkeypatch.setattr("shutil.which", lambda name: "/usr/bin/numactl")
95 + executable, debug_str = numa_utils._get_numactl_executable()
96 + assert executable.endswith("/vllm/utils/numa_wrapper.sh")
97 + assert "_VLLM_INTERNAL_NUMACTL_ARGS" in debug_str
98 +
99 +
100 +def test_set_numa_wrapper_env_restores_previous_values():
101 + os.environ[numa_utils._NUMACTL_ARGS_ENV] = "old-args"
102 + os.environ[numa_utils._NUMACTL_PYTHON_EXECUTABLE_ENV] = "old-python"
103 +
104 + with numa_utils._set_numa_wrapper_env("new-args", "new-python"):
105 + assert os.environ[numa_utils._NUMACTL_ARGS_ENV] == "new-args"
106 + assert os.environ[numa_utils._NUMACTL_PYTHON_EXECUTABLE_ENV] == "new-python"
107 +
108 + assert os.environ[numa_utils._NUMACTL_ARGS_ENV] == "old-args"
109 + assert os.environ[numa_utils._NUMACTL_PYTHON_EXECUTABLE_ENV] == "old-python"
110 +
111 +
112 +def test_set_numa_wrapper_env_clears_values_when_unset():
113 + os.environ.pop(numa_utils._NUMACTL_ARGS_ENV, None)
114 + os.environ.pop(numa_utils._NUMACTL_PYTHON_EXECUTABLE_ENV, None)
115 +
116 + with numa_utils._set_numa_wrapper_env("new-args", "new-python"):
117 + assert os.environ[numa_utils._NUMACTL_ARGS_ENV] == "new-args"
118 + assert os.environ[numa_utils._NUMACTL_PYTHON_EXECUTABLE_ENV] == "new-python"
119 +
120 + assert numa_utils._NUMACTL_ARGS_ENV not in os.environ
121 + assert numa_utils._NUMACTL_PYTHON_EXECUTABLE_ENV not in os.environ
122 +
123 +
124 +def test_parallel_config_validates_numa_bind_nodes():
125 + with pytest.raises(ValueError, match="non-negative"):
126 + ParallelConfig(numa_bind_nodes=[0, -1])
127 +
128 +
129 +@pytest.mark.parametrize("cpuset", ["", "abc", "1-", "4-1", "1,,2", "1:2"])
130 +def test_parallel_config_rejects_invalid_numa_bind_cpus(cpuset):
131 + with pytest.raises(ValueError, match="numa_bind_cpus"):
132 + ParallelConfig(numa_bind_cpus=[cpuset])
tests/utils_/test_system_utils.py
+9 -1
@@ -1,10 +1,11 @@
1 # SPDX-License-Identifier: Apache-2.0
2 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
4 +import os
5 import tempfile
6 from pathlib import Path
7
7 -from vllm.utils.system_utils import unique_filepath
8 +from vllm.utils.system_utils import _maybe_force_spawn, unique_filepath
9
10
11 def test_unique_filepath():
@@ -17,3 +18,10 @@ def test_unique_filepath():
18 paths.add(path)
19 assert len(paths) == 10
20 assert len(list(Path(temp_dir).glob("*.txt"))) == 10
21 +
22 +
23 +def test_numa_bind_forces_spawn(monkeypatch):
24 + monkeypatch.delenv("VLLM_WORKER_MULTIPROC_METHOD", raising=False)
25 + monkeypatch.setattr("sys.argv", ["vllm", "serve", "--numa-bind"])
26 + _maybe_force_spawn()
27 + assert os.environ["VLLM_WORKER_MULTIPROC_METHOD"] == "spawn"
vllm/config/parallel.py
+67
@@ -6,6 +6,7 @@ import socket
6 from collections.abc import Callable
7 from typing import TYPE_CHECKING, Any, Literal, overload
8
9 +import regex as re
10 import torch
11 from pydantic import Field, field_validator, model_validator
12 from torch.distributed import ProcessGroup, ReduceOp, Store
@@ -28,6 +29,7 @@ else:
29 Executor = Any
30
31 logger = init_logger(__name__)
32 +_NUMACTL_CPUSET_PATTERN = re.compile(r"^\d+(?:-\d+)?(?:,\d+(?:-\d+)?)*$")
33
34 ExpertPlacementStrategy = Literal["linear", "round_robin"]
35 DistributedExecutorBackend = Literal["ray", "mp", "uni", "external_launcher"]
@@ -259,6 +261,27 @@ class ParallelConfig:
261 nnodes: int = 1
262 """num of nodes for multi-node distributed
263 inference when distributed_executor_backend is mp."""
264 + numa_bind: bool = False
265 + """Enable NUMA binding for GPU worker subprocesses."""
266 + numa_bind_nodes: list[int] | None = None
267 + """NUMA node to bind each GPU worker to.
268 +
269 + Specify one NUMA node per visible GPU, for example `[0, 0, 1, 1]`
270 + for a 4-GPU system with GPUs 0-1 on NUMA node 0 and GPUs 2-3 on
271 + NUMA node 1. If unset and `numa_bind=True`, vLLM auto-detects the
272 + GPU-to-NUMA topology. The values are passed to `numactl --membind`
273 + and `--cpunodebind`, so they must be valid `numactl` NUMA node indices.
274 + """
275 + numa_bind_cpus: list[str] | None = None
276 + """Optional CPU lists to bind each GPU worker to.
277 +
278 + Specify one CPU list per visible GPU, for example
279 + `["0-3", "4-7", "8-11", "12-15"]`. When set, vLLM uses
280 + `numactl --physcpubind` instead of `--cpunodebind`. This is useful
281 + for custom policies such as binding to PCT or other high-frequency cores.
282 + Each entry must use `numactl --physcpubind` CPU-list syntax, for example
283 + `"0-3"` or `"0,2,4-7"`.
284 + """
285
286 distributed_timeout_seconds: int | None = None
287 """Timeout in seconds for distributed operations (e.g., init_process_group).
@@ -345,6 +368,43 @@ class ParallelConfig:
368 """Skip validation if the value is `None` when initialisation is delayed."""
369 return None if value is None else handler(value)
370
371 + @field_validator("numa_bind_nodes")
372 + @classmethod
373 + def _validate_numa_bind_nodes(cls, value: list[int] | None) -> list[int] | None:
374 + if value is None:
375 + return None
376 + if not value:
377 + raise ValueError("numa_bind_nodes must not be empty.")
378 + if any(node < 0 for node in value):
379 + raise ValueError("numa_bind_nodes must contain non-negative integers.")
380 + return value
381 +
382 + @field_validator("numa_bind_cpus")
383 + @classmethod
384 + def _validate_numa_bind_cpus(cls, value: list[str] | None) -> list[str] | None:
385 + if value is None:
386 + return None
387 + if not value:
388 + raise ValueError("numa_bind_cpus must not be empty.")
389 +
390 + for cpuset in value:
391 + if not cpuset:
392 + raise ValueError("numa_bind_cpus entries must not be empty.")
393 + if not _NUMACTL_CPUSET_PATTERN.fullmatch(cpuset):
394 + raise ValueError(
395 + "numa_bind_cpus entries must use numactl CPU list syntax, "
396 + "for example '0-3' or '0,2,4-7'."
397 + )
398 + for part in cpuset.split(","):
399 + if "-" not in part:
400 + continue
401 + start_str, end_str = part.split("-", 1)
402 + if int(start_str) > int(end_str):
403 + raise ValueError(
404 + f"numa_bind_cpus ranges must be ascending, but got '{cpuset}'."
405 + )
406 + return value
407 +
408 @model_validator(mode="after")
409 def _validate_parallel_config(self) -> Self:
410 if self._api_process_rank >= self._api_process_count:
@@ -373,6 +433,13 @@ class ParallelConfig:
433 "data_parallel_external_lb can only be set when data_parallel_size > 1"
434 )
435
436 + if not self.numa_bind and (
437 + self.numa_bind_nodes is not None or self.numa_bind_cpus is not None
438 + ):
439 + raise ValueError(
440 + "numa_bind_nodes and numa_bind_cpus require numa_bind=True."
441 + )
442 +
443 if self.enable_eplb:
444 if not current_platform.is_cuda_alike():
445 raise ValueError(
vllm/engine/arg_utils.py
+13
@@ -416,6 +416,9 @@ class EngineArgs:
416 nnodes: int = ParallelConfig.nnodes
417 node_rank: int = ParallelConfig.node_rank
418 distributed_timeout_seconds: int | None = ParallelConfig.distributed_timeout_seconds
419 + numa_bind: bool = ParallelConfig.numa_bind
420 + numa_bind_nodes: list[int] | None = ParallelConfig.numa_bind_nodes
421 + numa_bind_cpus: list[str] | None = ParallelConfig.numa_bind_cpus
422 tensor_parallel_size: int = ParallelConfig.tensor_parallel_size
423 prefill_context_parallel_size: int = ParallelConfig.prefill_context_parallel_size
424 decode_context_parallel_size: int = ParallelConfig.decode_context_parallel_size
@@ -861,6 +864,13 @@ class EngineArgs:
864 "--distributed-timeout-seconds",
865 **parallel_kwargs["distributed_timeout_seconds"],
866 )
867 + parallel_group.add_argument("--numa-bind", **parallel_kwargs["numa_bind"])
868 + parallel_group.add_argument(
869 + "--numa-bind-nodes", **parallel_kwargs["numa_bind_nodes"]
870 + )
871 + parallel_group.add_argument(
872 + "--numa-bind-cpus", **parallel_kwargs["numa_bind_cpus"]
873 + )
874 parallel_group.add_argument(
875 "--tensor-parallel-size", "-tp", **parallel_kwargs["tensor_parallel_size"]
876 )
@@ -1826,6 +1836,9 @@ class EngineArgs:
1836 cp_kv_cache_interleave_size=self.cp_kv_cache_interleave_size,
1837 _api_process_count=self._api_process_count,
1838 _api_process_rank=self._api_process_rank,
1839 + numa_bind=self.numa_bind,
1840 + numa_bind_nodes=self.numa_bind_nodes,
1841 + numa_bind_cpus=self.numa_bind_cpus,
1842 )
1843
1844 speculative_config = self.create_speculative_config(
vllm/platforms/cuda.py
+113
@@ -653,6 +653,111 @@ class NvmlCudaPlatform(CudaPlatformBase):
653 handle = pynvml.nvmlDeviceGetHandleByIndex(device_id)
654 return pynvml.nvmlDeviceGetName(handle)
655
656 + @classmethod
657 + @with_nvml_context
658 + def get_device_numa_node(cls, device_id: int = 0) -> int | None:
659 + """Get the NUMA node ID for a GPU device."""
660 + physical_device_id = cls.device_id_to_physical_device_id(device_id)
661 + handle = pynvml.nvmlDeviceGetHandleByIndex(physical_device_id)
662 +
663 + try:
664 + return pynvml.nvmlDeviceGetNumaNodeId(handle)
665 + except Exception:
666 + pass
667 +
668 + try:
669 + cpu_ids = cls._get_device_cpu_affinity(handle)
670 + if cpu_ids:
671 + numa_node = cls._get_numa_node_for_cpu(cpu_ids[0])
672 + if numa_node is not None:
673 + logger.debug(
674 + "Determined NUMA node %d for GPU %d via CPU affinity",
675 + numa_node,
676 + device_id,
677 + )
678 + return numa_node
679 + except Exception as e:
680 + logger.warning("Failed to get NUMA node for GPU %d: %s", device_id, e)
681 +
682 + return None
683 +
684 + @classmethod
685 + def _get_device_cpu_affinity(cls, handle) -> list[int]:
686 + """Get the list of CPU IDs associated with a GPU via NVML."""
687 + cpu_count = os.cpu_count()
688 + if cpu_count is None:
689 + return []
690 +
691 + cpu_set_size = (cpu_count + 63) // 64
692 + cpu_affinity_mask = pynvml.nvmlDeviceGetCpuAffinity(handle, cpu_set_size)
693 +
694 + cpu_ids = []
695 + for i, mask in enumerate(cpu_affinity_mask):
696 + for bit in range(64):
697 + cpu_id = i * 64 + bit
698 + if cpu_id >= cpu_count:
699 + break
700 + if mask & (1 << bit):
701 + cpu_ids.append(cpu_id)
702 + return cpu_ids
703 +
704 + @classmethod
705 + def _get_numa_node_for_cpu(cls, cpu_id: int) -> int | None:
706 + """Determine which NUMA node a CPU belongs to."""
707 + from pathlib import Path
708 +
709 + node_path = Path("/sys/devices/system/node")
710 + if not node_path.exists():
711 + return None
712 +
713 + for node_dir in node_path.iterdir():
714 + if not node_dir.name.startswith("node"):
715 + continue
716 + try:
717 + node_id = int(node_dir.name[4:])
718 + cpulist_file = node_dir / "cpulist"
719 + if cpulist_file.exists():
720 + cpulist = cpulist_file.read_text().strip()
721 + if cls._cpu_in_cpulist(cpu_id, cpulist):
722 + return node_id
723 + except (ValueError, OSError):
724 + continue
725 + return None
726 +
727 + @classmethod
728 + def _cpu_in_cpulist(cls, cpu_id: int, cpulist: str) -> bool:
729 + """Check if a CPU ID is in a cpulist string such as '0-3,8-11'."""
730 + for part in cpulist.split(","):
731 + part = part.strip()
732 + if "-" in part:
733 + start, end = part.split("-", 1)
734 + if int(start) <= cpu_id <= int(end):
735 + return True
736 + elif part.isdigit() and int(part) == cpu_id:
737 + return True
738 + return False
739 +
740 + @classmethod
741 + @with_nvml_context
742 + def get_all_device_numa_nodes(cls) -> list[int] | None:
743 + """Get NUMA nodes for all visible GPU devices."""
744 + try:
745 + numa_nodes = []
746 + for device_id in range(cls.device_count()):
747 + numa_node = cls.get_device_numa_node(device_id)
748 + if numa_node is None:
749 + logger.warning(
750 + "Could not detect NUMA node for GPU %d, "
751 + "disabling automatic NUMA binding",
752 + device_id,
753 + )
754 + return None
755 + numa_nodes.append(numa_node)
756 + return numa_nodes
757 + except Exception as e:
758 + logger.warning("Failed to get NUMA nodes for GPUs: %s", e)
759 + return None
760 +
761 @classmethod
762 @with_nvml_context
763 def log_warnings(cls):
@@ -695,6 +800,14 @@ class NonNvmlCudaPlatform(CudaPlatformBase):
800 )
801 return False
802
803 + @classmethod
804 + def get_device_numa_node(cls, device_id: int = 0) -> int | None:
805 + return None
806 +
807 + @classmethod
808 + def get_all_device_numa_nodes(cls) -> list[int] | None:
809 + return None
810 +
811
812 # Autodetect either NVML-enabled or non-NVML platform
813 # based on whether NVML is available.
vllm/utils/numa_utils.py new
+317
@@ -0,0 +1,317 @@
1 +# SPDX-License-Identifier: Apache-2.0
2 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 +"""NUMA binding utilities for vLLM worker processes.
4 +
5 +Adapted in part from SGLang's NUMA helper implementation:
6 +https://github.com/sgl-project/sglang/blob/ba6d54d0f08f82f42b8224908ae2459a496b31b3/python/sglang/srt/utils/numa_utils.py
7 +"""
8 +
9 +import ctypes
10 +import logging
11 +import multiprocessing
12 +import os
13 +import subprocess
14 +from contextlib import contextmanager
15 +from functools import cache
16 +from pathlib import Path
17 +from typing import TYPE_CHECKING
18 +
19 +import psutil
20 +
21 +from vllm import envs
22 +
23 +if TYPE_CHECKING:
24 + from vllm.config import VllmConfig
25 +
26 +logger = logging.getLogger(__name__)
27 +_NUMACTL_ARGS_ENV = "_VLLM_INTERNAL_NUMACTL_ARGS"
28 +_NUMACTL_PYTHON_EXECUTABLE_ENV = "_VLLM_INTERNAL_NUMACTL_PYTHON_EXECUTABLE"
29 +
30 +
31 +@cache
32 +def get_libnuma():
33 + libnuma = None
34 + for libnuma_so in ["libnuma.so", "libnuma.so.1"]:
35 + try:
36 + libnuma = ctypes.CDLL(libnuma_so)
37 + except OSError:
38 + libnuma = None
39 + if libnuma is not None:
40 + break
41 + return libnuma
42 +
43 +
44 +def _can_set_mempolicy() -> bool:
45 + """Check whether the current process can use NUMA memory policy syscalls."""
46 + try:
47 + libnuma = get_libnuma()
48 + if libnuma is None or libnuma.numa_available() < 0:
49 + return False
50 + mode = ctypes.c_int()
51 + ret = libnuma.get_mempolicy(
52 + ctypes.byref(mode), None, ctypes.c_ulong(0), None, ctypes.c_ulong(0)
53 + )
54 + return ret == 0
55 + except Exception:
56 + return False
57 +
58 +
59 +def _is_auto_numa_available() -> bool:
60 + """Check whether automatic GPU-to-NUMA detection should be attempted."""
61 + from vllm.platforms import current_platform
62 +
63 + if not current_platform.is_cuda_alike():
64 + return False
65 +
66 + if not os.path.isdir("/sys/devices/system/node/node1"):
67 + return False
68 +
69 + try:
70 + process = psutil.Process(os.getpid())
71 + cpu_affinity = process.cpu_affinity()
72 + cpu_count = psutil.cpu_count()
73 + if cpu_count is not None and cpu_affinity != list(range(cpu_count)):
74 + logger.warning(
75 + "CPU affinity is already constrained for this process. "
76 + "Skipping automatic NUMA binding; pass --numa-bind-nodes "
77 + "explicitly to override."
78 + )
79 + return False
80 + except (AttributeError, NotImplementedError, psutil.Error):
81 + pass
82 +
83 + if not _can_set_mempolicy():
84 + logger.warning(
85 + "User lacks permission to set NUMA memory policy. "
86 + "Automatic NUMA detection may not work; if you are using Docker, "
87 + "try adding --cap-add SYS_NICE."
88 + )
89 + return False
90 +
91 + if not hasattr(current_platform, "get_all_device_numa_nodes"):
92 + logger.warning(
93 + "Platform %s does not support automatic NUMA detection",
94 + type(current_platform).__name__,
95 + )
96 + return False
97 +
98 + return True
99 +
100 +
101 +@cache
102 +def get_auto_numa_nodes() -> list[int] | None:
103 + """Auto-detect NUMA nodes for all visible GPUs."""
104 + from vllm.platforms import current_platform
105 +
106 + if not _is_auto_numa_available():
107 + return None
108 +
109 + numa_nodes = current_platform.get_all_device_numa_nodes()
110 + if numa_nodes is not None:
111 + logger.info("Auto-detected NUMA nodes for GPUs: %s", numa_nodes)
112 + return numa_nodes
113 +
114 +
115 +def _get_gpu_index(
116 + parallel_config, local_rank: int, dp_local_rank: int | None = None
117 +) -> int:
118 + """Compute the physical GPU index used for NUMA lookup."""
119 + if (
120 + parallel_config.distributed_executor_backend not in ("ray", "external_launcher")
121 + and parallel_config.data_parallel_backend != "ray"
122 + and parallel_config.nnodes_within_dp == 1
123 + ):
124 + if dp_local_rank is None:
125 + dp_local_rank = parallel_config.data_parallel_rank_local
126 + if dp_local_rank is None:
127 + dp_local_rank = parallel_config.data_parallel_index
128 +
129 + tp_pp_world_size = (
130 + parallel_config.pipeline_parallel_size
131 + * parallel_config.tensor_parallel_size
132 + )
133 + return local_rank + dp_local_rank * tp_pp_world_size
134 +
135 + return local_rank
136 +
137 +
138 +def _get_numa_node(parallel_config, gpu_index: int) -> int:
139 + numa_nodes = parallel_config.numa_bind_nodes
140 + if numa_nodes is None:
141 + numa_nodes = get_auto_numa_nodes()
142 + if numa_nodes is None:
143 + raise RuntimeError(
144 + "NUMA binding was requested, but vLLM could not detect the "
145 + "GPU-to-NUMA topology automatically. Pass --numa-bind-nodes "
146 + "explicitly or disable --numa-bind."
147 + )
148 + parallel_config.numa_bind_nodes = numa_nodes
149 +
150 + if gpu_index >= len(numa_nodes):
151 + raise ValueError(
152 + f"GPU index {gpu_index} exceeds numa_bind_nodes size {len(numa_nodes)}. "
153 + "Ensure the binding lists cover every visible GPU."
154 + )
155 +
156 + return numa_nodes[gpu_index]
157 +
158 +
159 +def _get_cpu_binding(parallel_config, gpu_index: int) -> str | None:
160 + cpu_bindings = parallel_config.numa_bind_cpus
161 + if cpu_bindings is None:
162 + return None
163 +
164 + if gpu_index >= len(cpu_bindings):
165 + raise ValueError(
166 + f"GPU index {gpu_index} exceeds numa_bind_cpus size "
167 + f"{len(cpu_bindings)}. Ensure the binding lists cover every visible GPU."
168 + )
169 +
170 + return cpu_bindings[gpu_index]
171 +
172 +
173 +def _get_numactl_args(
174 + vllm_config: "VllmConfig",
175 + local_rank: int,
176 + dp_local_rank: int | None = None,
177 + process_kind: str = "worker",
178 +) -> str | None:
179 + parallel_config = vllm_config.parallel_config
180 + if not parallel_config.numa_bind:
181 + return None
182 +
183 + gpu_index = _get_gpu_index(parallel_config, local_rank, dp_local_rank)
184 + numa_node = _get_numa_node(parallel_config, gpu_index)
185 + cpu_binding = _get_cpu_binding(parallel_config, gpu_index)
186 +
187 + if cpu_binding is not None:
188 + bind_arg = f"--physcpubind={cpu_binding}"
189 + logger.info(
190 + "Binding %s subprocess (local_rank=%s, gpu_index=%s) to CPUs %s and NUMA node %s", # noqa: E501
191 + process_kind,
192 + local_rank,
193 + gpu_index,
194 + cpu_binding,
195 + numa_node,
196 + )
197 + else:
198 + bind_arg = f"--cpunodebind={numa_node}"
199 + logger.info(
200 + "Binding %s subprocess (local_rank=%s, gpu_index=%s) to NUMA node %s",
201 + process_kind,
202 + local_rank,
203 + gpu_index,
204 + numa_node,
205 + )
206 +
207 + return f"{bind_arg} --membind={numa_node}"
208 +
209 +
210 +def _log_numactl_show(label: str) -> bool:
211 + try:
212 + result = subprocess.run(
213 + ["numactl", "--show"],
214 + check=True,
215 + capture_output=True,
216 + text=True,
217 + )
218 + except (FileNotFoundError, subprocess.CalledProcessError) as e:
219 + logger.warning("Failed to run `numactl --show` for %s: %s", label, e)
220 + return False
221 +
222 + output = result.stdout.strip()
223 + if not output:
224 + logger.warning("`numactl --show` returned no output for %s", label)
225 + return False
226 +
227 + summary = ", ".join(line.strip() for line in output.splitlines() if line.strip())
228 + logger.debug("%s affinity: %s", label, summary)
229 + return True
230 +
231 +
232 +def log_current_affinity_state(label: str) -> None:
233 + """Log the process's effective NUMA affinity state."""
234 + _log_numactl_show(label)
235 +
236 +
237 +@contextmanager
238 +def configure_subprocess(
239 + vllm_config: "VllmConfig",
240 + local_rank: int,
241 + dp_local_rank: int | None = None,
242 + process_kind: str = "worker",
243 +):
244 + """Temporarily replace the multiprocessing executable with a numactl wrapper."""
245 + numactl_args = _get_numactl_args(
246 + vllm_config, local_rank, dp_local_rank, process_kind
247 + )
248 + if numactl_args is None:
249 + yield
250 + return
251 +
252 + executable, debug_str = _get_numactl_executable()
253 + python_executable = os.fsdecode(multiprocessing.spawn.get_executable())
254 + with (
255 + _set_numa_wrapper_env(numactl_args, python_executable),
256 + _mp_set_executable(executable, debug_str),
257 + ):
258 + yield
259 +
260 +
261 +def _get_numactl_executable() -> tuple[str, str]:
262 + """Return the fixed wrapper executable used to launch numactl."""
263 + from shutil import which
264 +
265 + if which("numactl") is None:
266 + raise RuntimeError(
267 + "numactl is required for NUMA binding but is not installed or "
268 + "not available on PATH."
269 + )
270 +
271 + script_path = Path(__file__).with_name("numa_wrapper.sh")
272 + return str(script_path), f"{script_path} via {_NUMACTL_ARGS_ENV}"
273 +
274 +
275 +@contextmanager
276 +def _set_numa_wrapper_env(numactl_args: str, python_executable: str):
277 + old_numactl_args = os.environ.get(_NUMACTL_ARGS_ENV)
278 + old_python_executable = os.environ.get(_NUMACTL_PYTHON_EXECUTABLE_ENV)
279 + os.environ[_NUMACTL_ARGS_ENV] = numactl_args
280 + os.environ[_NUMACTL_PYTHON_EXECUTABLE_ENV] = python_executable
281 + try:
282 + yield
283 + finally:
284 + if old_numactl_args is None:
285 + os.environ.pop(_NUMACTL_ARGS_ENV, None)
286 + else:
287 + os.environ[_NUMACTL_ARGS_ENV] = old_numactl_args
288 +
289 + if old_python_executable is None:
290 + os.environ.pop(_NUMACTL_PYTHON_EXECUTABLE_ENV, None)
291 + else:
292 + os.environ[_NUMACTL_PYTHON_EXECUTABLE_ENV] = old_python_executable
293 +
294 +
295 +@contextmanager
296 +def _mp_set_executable(executable: str, debug_str: str):
297 + start_method = envs.VLLM_WORKER_MULTIPROC_METHOD
298 + if start_method != "spawn":
299 + logger.warning(
300 + "NUMA binding requires spawn method but got '%s'. "
301 + "NUMA binding will be ineffective. "
302 + "Set VLLM_WORKER_MULTIPROC_METHOD=spawn to enable NUMA binding.",
303 + start_method,
304 + )
305 + yield
306 + return
307 +
308 + old_executable = os.fsdecode(multiprocessing.spawn.get_executable())
309 + multiprocessing.spawn.set_executable(executable)
310 + try:
311 + yield
312 + finally:
313 + assert os.fsdecode(multiprocessing.spawn.get_executable()) == executable, (
314 + "Executable was changed during NUMA binding context: "
315 + f"expected {executable}, got {multiprocessing.spawn.get_executable()}"
316 + )
317 + multiprocessing.spawn.set_executable(old_executable)
vllm/utils/numa_wrapper.sh new
+25
@@ -0,0 +1,25 @@
1 +#!/bin/sh
2 +
3 +if [ -z "${_VLLM_INTERNAL_NUMACTL_ARGS:-}" ]; then
4 + echo "_VLLM_INTERNAL_NUMACTL_ARGS is not set" >&2
5 + exit 1
6 +fi
7 +
8 +if [ -z "${_VLLM_INTERNAL_NUMACTL_PYTHON_EXECUTABLE:-}" ]; then
9 + echo "_VLLM_INTERNAL_NUMACTL_PYTHON_EXECUTABLE is not set" >&2
10 + exit 1
11 +fi
12 +
13 +if ! command -v numactl >/dev/null 2>&1; then
14 + echo "numactl is not available on PATH" >&2
15 + exit 1
16 +fi
17 +
18 +case "${_VLLM_INTERNAL_NUMACTL_ARGS}" in
19 + *[![:alnum:]\ \-\_=,./]*)
20 + echo "Invalid characters in _VLLM_INTERNAL_NUMACTL_ARGS" >&2
21 + exit 1
22 + ;;
23 +esac
24 +
25 +exec numactl ${_VLLM_INTERNAL_NUMACTL_ARGS} "${_VLLM_INTERNAL_NUMACTL_PYTHON_EXECUTABLE}" "$@"
vllm/utils/system_utils.py
+5
@@ -140,6 +140,11 @@ def _maybe_force_spawn():
140 os.environ["RAY_ADDRESS"] = ray.get_runtime_context().gcs_address
141 reasons.append("In a Ray actor and can only be spawned")
142
143 + # Force spawn if NUMA binding is enabled via --numa-bind.
144 + # NUMA binding uses executable hijacking which requires spawn
145 + if "--numa-bind" in sys.argv:
146 + reasons.append("NUMA binding requires spawn method")
147 +
148 if cuda_is_initialized():
149 reasons.append("CUDA is initialized")
150 elif xpu_is_initialized():
vllm/v1/engine/core.py
+6 -2
@@ -30,6 +30,7 @@ from vllm.multimodal import MULTIMODAL_REGISTRY
30 from vllm.tasks import POOLING_TASKS, SupportedTask
31 from vllm.tracing import instrument, maybe_init_worker_tracer
32 from vllm.transformers_utils.config import maybe_register_config_serialize_by_value
33 +from vllm.utils import numa_utils
34 from vllm.utils.gc_utils import (
35 freeze_gc_heap,
36 maybe_attach_gc_debug_callback,
@@ -1055,6 +1056,8 @@ class EngineCoreProc(EngineCore):
1056 set_process_title(process_title)
1057 maybe_init_worker_tracer("vllm.engine_core", "engine_core", process_title)
1058 decorate_logs()
1059 + if parallel_config.numa_bind:
1060 + numa_utils.log_current_affinity_state(process_title)
1061
1062 if data_parallel and vllm_config.kv_transfer_config is not None:
1063 # modify the engine_id and append the local_dp_rank to it to ensure
@@ -1255,8 +1258,9 @@ class EngineCoreProc(EngineCore):
1258 return
1259 output = UtilityOutput(call_id)
1260 # Lazily look-up utility method so that failure will be handled/returned.
1258 - get_result = lambda: (method := getattr(self, method_name)) and method(
1259 - *self._convert_msgspec_args(method, args)
1261 + get_result = lambda: (
1262 + (method := getattr(self, method_name))
1263 + and method(*self._convert_msgspec_args(method, args))
1264 )
1265 enqueue_output = lambda out: self.output_queue.put_nowait(
1266 (client_idx, EngineCoreOutputs(utility_output=out))
vllm/v1/engine/utils.py
+24 -3
@@ -22,6 +22,7 @@ from vllm.config import CacheConfig, ParallelConfig, VllmConfig
22 from vllm.logger import init_logger
23 from vllm.platforms import current_platform
24 from vllm.ray.ray_env import get_env_vars_to_copy
25 +from vllm.utils import numa_utils
26 from vllm.utils.network_utils import get_open_zmq_ipc_path, zmq_socket_ctx
27 from vllm.utils.system_utils import get_mp_context
28 from vllm.v1.engine.coordinator import DPCoordinator
@@ -141,13 +142,33 @@ class CoreEngineProcManager:
142 # Adjust device control in DP for non-CUDA platforms
143 # as well as external and ray launchers
144 # For CUDA platforms, we use torch.accelerator.set_device_index()()
145 + device_control_context: contextlib.AbstractContextManager[None] = (
146 + contextlib.nullcontext()
147 + )
148 if is_dp and (
149 not current_platform.is_cuda_alike()
150 or vllm_config.parallel_config.use_ray
151 ):
148 - with set_device_control_env_var(vllm_config, local_dp_rank):
149 - proc.start()
150 - else:
152 + device_control_context = set_device_control_env_var(
153 + vllm_config, local_dp_rank
154 + )
155 +
156 + with (
157 + device_control_context,
158 + numa_utils.configure_subprocess(
159 + # EngineCore itself does not have a TP/PP-local rank.
160 + # When DP is enabled, set_device_control_env_var()
161 + # narrows visible devices to this DP shard first, so
162 + # local_rank=0 means "the first local GPU in this
163 + # shard". The actual TP/PP worker processes spawned by
164 + # the executor are bound separately with their own
165 + # local_rank values.
166 + vllm_config,
167 + local_rank=0,
168 + dp_local_rank=local_dp_rank,
169 + process_kind="EngineCore",
170 + ),
171 + ):
172 proc.start()
173 finally:
174 # Kill other procs if not all are running.
vllm/v1/executor/multiproc_executor.py
+9 -1
@@ -44,6 +44,7 @@ from vllm.envs import enable_envs_cache
44 from vllm.logger import init_logger
45 from vllm.platforms import current_platform
46 from vllm.tracing import instrument, maybe_init_worker_tracer
47 +from vllm.utils import numa_utils
48 from vllm.utils.network_utils import (
49 get_distributed_init_method,
50 get_ip,
@@ -688,7 +689,12 @@ class WorkerProc:
689 daemon=True,
690 )
691
691 - proc.start()
692 + # Apply NUMA binding if configured
693 + with numa_utils.configure_subprocess(
694 + vllm_config, local_rank, process_kind="worker"
695 + ):
696 + proc.start()
697 +
698 # Close child ends of pipes here in the parent
699 ready_writer.close()
700 death_reader.close()
@@ -839,6 +845,8 @@ class WorkerProc:
845
846 worker = WorkerProc(*args, **kwargs)
847 assert worker.worker_response_mq is not None
848 + if kwargs["vllm_config"].parallel_config.numa_bind:
849 + numa_utils.log_current_affinity_state(f"Worker_{worker.rank}")
850
851 worker.monitor_death_pipe(death_pipe, shutdown_requested)
852