@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / cfb7e55515

Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 11, 06:15 UTC

[Doc][CPU] Update CPU doc (#30765)

Signed-off-by: jiang1.li <jiang1.li@intel.com> Signed-off-by: Li, Jiang <bigpyj64@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Li, Jiang committed Dec 18, 2025 at 12:59 UTC cfb7e55515a5558be3a7199044411953017352d3

5 files changed +106 -16

docker/Dockerfile.cpu

+2 -2

@@ -17,7 +17,7 @@

17 # VLLM_CPU_DISABLE_AVX512=false (default)|true

18 # VLLM_CPU_AVX512BF16=false (default)|true

19 # VLLM_CPU_AVX512VNNI=false (default)|true

20 -# VLLM_CPU_AMXBF16=false (default)|true

20 +# VLLM_CPU_AMXBF16=false |true (default)

21 #

23 ######################### COMMON BASE IMAGE #########################

@@ -95,7 +95,7 @@ ENV VLLM_CPU_AVX512BF16=${VLLM_CPU_AVX512BF16}

95 ARG VLLM_CPU_AVX512VNNI=0

96 ENV VLLM_CPU_AVX512VNNI=${VLLM_CPU_AVX512VNNI}

97 # Support for building with AMXBF16 ISA: docker build --build-arg VLLM_CPU_AMXBF16="true" ...

98 -ARG VLLM_CPU_AMXBF16=0

98 +ARG VLLM_CPU_AMXBF16=1

99 ENV VLLM_CPU_AMXBF16=${VLLM_CPU_AMXBF16}

100

101 WORKDIR /workspace/vllm

docs/getting_started/installation/cpu.arm.inc.md

+31 -3

@@ -19,7 +19,7 @@ Pre-built vLLM wheels for Arm are available since version 0.11.2. These wheels c

20 ```bash

21 export VLLM_VERSION=$(curl -s https://api.github.com/repos/vllm-project/vllm/releases/latest | jq -r .tag_name | sed 's/^v//')

22 -uv pip install vllm --extra-index-url https://wheels.vllm.ai/${VLLM_VERSION}/cpu

22 +uv pip install vllm --extra-index-url https://wheels.vllm.ai/${VLLM_VERSION}/cpu --index-strategy first-index

23 ```

25 ??? console "pip"

@@ -27,6 +27,20 @@ uv pip install vllm --extra-index-url https://wheels.vllm.ai/${VLLM_VERSION}/cpu

27 pip install vllm==${VLLM_VERSION}+cpu --extra-index-url https://wheels.vllm.ai/${VLLM_VERSION}/cpu

28 ```

30 +!!! warning "set `LD_PRELOAD`"

31 + Before use vLLM CPU installed via wheels, make sure TCMalloc is installed and added to `LD_PRELOAD`:

32 + ```bash

33 + # install TCMalloc

34 + sudo apt-get install -y --no-install-recommends libtcmalloc-minimal4

35 +

36 + # manually find the path

37 + sudo find / -iname *libtcmalloc_minimal.so.4

38 + TC_PATH=...

39 +

40 + # add them to LD_PRELOAD

41 + export LD_PRELOAD="$TC_PATH:$LD_PRELOAD"

42 + ```

43 +

44 The `uv` approach works for vLLM `v0.6.6` and later. A unique feature of `uv` is that packages in `--extra-index-url` have [higher priority than the default index](https://docs.astral.sh/uv/pip/compatibility/#packages-that-exist-on-multiple-indexes). If the latest public release is `v0.6.6.post1`, `uv`'s behavior allows installing a commit before `v0.6.6.post1` by specifying the `--extra-index-url`. In contrast, `pip` combines packages from `--extra-index-url` and the default index, choosing only the latest version, which makes it difficult to install a development version prior to the released version.

46 **Install the latest code**

@@ -37,7 +51,7 @@ LLM inference is a fast-evolving field, and the latest code may contain bug fixe

52 To install from nightly index, run:

53 ```bash

40 -uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly/cpu

54 +uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly/cpu --index-strategy first-index

55 ```

57 ??? console "pip (there's a caveat)"

@@ -56,7 +70,7 @@ If you want to access the wheels for previous commits (e.g. to bisect the behavi

71 ```bash

72 export VLLM_COMMIT=730bd35378bf2a5b56b6d3a45be28b3092d26519 # use full commit hash from the main branch

59 -uv pip install vllm --extra-index-url https://wheels.vllm.ai/${VLLM_COMMIT}/cpu

73 +uv pip install vllm --extra-index-url https://wheels.vllm.ai/${VLLM_COMMIT}/cpu --index-strategy first-index

74 ```

76 # --8<-- [end:pre-built-wheels]

@@ -105,6 +119,20 @@ VLLM_TARGET_DEVICE=cpu uv pip install -e . --no-build-isolation

119

120 Testing has been conducted on AWS Graviton3 instances for compatibility.

121

122 +!!! warning "set `LD_PRELOAD`"

123 + Before use vLLM CPU installed via wheels, make sure TCMalloc is installed and added to `LD_PRELOAD`:

124 + ```bash

125 + # install TCMalloc

126 + sudo apt-get install -y --no-install-recommends libtcmalloc-minimal4

127 +

128 + # manually find the path

129 + sudo find / -iname *libtcmalloc_minimal.so.4

130 + TC_PATH=...

131 +

132 + # add them to LD_PRELOAD

133 + export LD_PRELOAD="$TC_PATH:$LD_PRELOAD"

134 + ```

135 +

136 # --8<-- [end:build-wheel-from-source]

137 # --8<-- [start:pre-built-images]

138

docs/getting_started/installation/cpu.md

+6 -5

@@ -18,6 +18,12 @@ vLLM is a Python library that supports the following CPU variants. Select your C

19 --8<-- "docs/getting_started/installation/cpu.s390x.inc.md:installation"

21 +## Technical Discussions

22 +

23 +The main discussions happen in the `#sig-cpu` channel of [vLLM Slack](https://slack.vllm.ai/).

24 +

25 +When open a Github issue about the CPU backend, please add `[CPU Backend]` in the title and it will be labeled with `cpu` for better awareness.

26 +

27 ## Requirements

29 - Python: 3.10 -- 3.13

@@ -258,11 +264,6 @@ vLLM CPU supports data parallel (DP), tensor parallel (TP) and pipeline parallel

264 - GPTQ (x86 only)

265 - compressed-tensor INT8 W8A8 (x86, s390x)

266

261 -### (x86 only) What is the purpose of `VLLM_CPU_SGL_KERNEL`?

262 -

263 -- Both of them require `amx` CPU flag.

264 - - `VLLM_CPU_SGL_KERNEL` can provide better performance for MoE models and small-batch scenarios.

265 -

267 ### Why do I see `get_mempolicy: Operation not permitted` when running in Docker?

268

269 In some container environments (like Docker), NUMA-related syscalls used by vLLM (e.g., `get_mempolicy`, `migrate_pages`) are blocked/denied in the runtime's default seccomp/capabilities settings. This may lead to warnings like `get_mempolicy: Operation not permitted`. Functionality is not affected, but NUMA memory binding/migration optimizations may not take effect and performance can be suboptimal.

docs/getting_started/installation/cpu.x86.inc.md

+66 -5

@@ -17,7 +17,51 @@ vLLM supports basic model inferencing and serving on x86 CPU platform, with data

17 # --8<-- [end:set-up-using-python]

18 # --8<-- [start:pre-built-wheels]

20 -Currently, there are no pre-built x86 CPU wheels.

20 +Pre-built vLLM wheels for x86 with AVX512 are available since version 0.13.0. To install release wheels:

21 +

22 +```bash

23 +export VLLM_VERSION=$(curl -s https://api.github.com/repos/vllm-project/vllm/releases/latest | jq -r .tag_name | sed 's/^v//')

24 +

25 +# use uv

26 +uv pip install vllm --extra-index-url https://wheels.vllm.ai/${VLLM_VERSION}/cpu --index-strategy first-index --torch-backend cpu

27 +```

28 +??? console "pip"

29 + ```bash

30 + # use pip

31 + pip install vllm==${VLLM_VERSION}+cpu --extra-index-url https://wheels.vllm.ai/${VLLM_VERSION}/cpu --extra-index-url https://download.pytorch.org/whl/cpu

32 + ```

33 +!!! warning "set `LD_PRELOAD`"

34 + Before use vLLM CPU installed via wheels, make sure TCMalloc and Intel OpenMP are installed and added to `LD_PRELOAD`:

35 + ```bash

36 + # install TCMalloc, Intel OpenMP is installed with vLLM CPU

37 + sudo apt-get install -y --no-install-recommends libtcmalloc-minimal4

38 +

39 + # manually find the path

40 + sudo find / -iname *libtcmalloc_minimal.so.4

41 + sudo find / -iname *libiomp5.so

42 + TC_PATH=...

43 + IOMP_PATH=...

44 +

45 + # add them to LD_PRELOAD

46 + export LD_PRELOAD="$TC_PATH:$IOMP_PATH:$LD_PRELOAD"

47 + ```

48 +

49 +**Install the latest code**

50 +

51 +To install the wheel built from the latest main branch:

52 +

53 +```bash

54 +uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly/cpu --index-strategy first-index --torch-backend cpu

55 +```

56 +

57 +**Install specific revisions**

58 +

59 +If you want to access the wheels for previous commits (e.g. to bisect the behavior change, performance regression), you can specify the commit hash in the URL:

60 +

61 +```bash

62 +export VLLM_COMMIT=730bd35378bf2a5b56b6d3a45be28b3092d26519 # use full commit hash from the main branch

63 +uv pip install vllm --extra-index-url https://wheels.vllm.ai/${VLLM_COMMIT}/cpu --index-strategy first-index --torch-backend cpu

64 +```

66 # --8<-- [end:pre-built-wheels]

67 # --8<-- [start:build-wheel-from-source]

@@ -26,10 +70,12 @@ Install recommended compiler. We recommend to use `gcc/g++ >= 12.3.0` as the def

71 ```bash

72 sudo apt-get update -y

29 -sudo apt-get install -y gcc-12 g++-12 libnuma-dev python3-dev

73 +sudo apt-get install -y gcc-12 g++-12 libnuma-dev

74 sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 10 --slave /usr/bin/g++ g++ /usr/bin/g++-12

75 ```

77 +--8<-- "docs/getting_started/installation/python_env_setup.inc.md"

78 +

79 Clone the vLLM project:

81 ```bash

@@ -82,6 +128,22 @@ uv pip install dist/*.whl

128 pip install dist/*.whl

129 ```

130

131 +!!! warning "set `LD_PRELOAD`"

132 + Before use vLLM CPU installed via wheels, make sure TCMalloc and Intel OpenMP are installed and added to `LD_PRELOAD`:

133 + ```bash

134 + # install TCMalloc, Intel OpenMP is installed with vLLM CPU

135 + sudo apt-get install -y --no-install-recommends libtcmalloc-minimal4

136 +

137 + # manually find the path

138 + sudo find / -iname *libtcmalloc_minimal.so.4

139 + sudo find / -iname *libiomp5.so

140 + TC_PATH=...

141 + IOMP_PATH=...

142 +

143 + # add them to LD_PRELOAD

144 + export LD_PRELOAD="$TC_PATH:$IOMP_PATH:$LD_PRELOAD"

145 + ```

146 +

147 !!! example "Troubleshooting"

148 - **NumPy ≥2.0 error**: Downgrade using `pip install "numpy<2.0"`.

149 - **CMake picks up CUDA**: Add `CMAKE_DISABLE_FIND_PACKAGE_CUDA=ON` to prevent CUDA detection during CPU builds, even if CUDA is installed.

@@ -95,7 +157,6 @@ uv pip install dist/*.whl

157 "torch==X.Y.Z+cpu" # <-------

158 ]

159 ```

98 - - If you are building vLLM from source and not using the pre-built images, remember to set `LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4:$LD_PRELOAD"` on x86 machines before running vLLM.

160

161 # --8<-- [end:build-wheel-from-source]

162 # --8<-- [start:pre-built-images]

@@ -112,6 +173,7 @@ uv pip install dist/*.whl

173 docker build -f docker/Dockerfile.cpu \

174 --build-arg VLLM_CPU_AVX512BF16=false (default)|true \

175 --build-arg VLLM_CPU_AVX512VNNI=false (default)|true \

176 + --build-arg VLLM_CPU_AMXBF16=false|true (default) \

177 --build-arg VLLM_CPU_DISABLE_AVX512=false (default)|true \

178 --tag vllm-cpu-env \

179 --target vllm-openai .

@@ -123,9 +185,8 @@ docker run --rm \

185 --shm-size=4g \

186 -p 8000:8000 \

187 -e VLLM_CPU_KVCACHE_SPACE=<KV cache space> \

126 - -e VLLM_CPU_OMP_THREADS_BIND=<CPU cores for inference> \

188 vllm-cpu-env \

128 - --model=meta-llama/Llama-3.2-1B-Instruct \

189 + meta-llama/Llama-3.2-1B-Instruct \

190 --dtype=bfloat16 \

191 other vLLM OpenAI server arguments

192 ```

docs/getting_started/installation/python_env_setup.inc.md

+1 -1

@@ -1,4 +1,4 @@

1 -On NVIDIA CUDA only, it's recommended to use [uv](https://docs.astral.sh/uv/), a very fast Python environment manager, to create and manage Python environments. Please follow the [documentation](https://docs.astral.sh/uv/#getting-started) to install `uv`. After installing `uv`, you can create a new Python environment using the following commands:

1 +It's recommended to use [uv](https://docs.astral.sh/uv/), a very fast Python environment manager, to create and manage Python environments. Please follow the [documentation](https://docs.astral.sh/uv/#getting-started) to install `uv`. After installing `uv`, you can create a new Python environment using the following commands:

3 ```bash

4 uv venv --python 3.12 --seed