A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 05:30 UTC
[docs] add SYS_NICE cap & `security-opt` for docker/k8s (#24017)
Signed-off-by: Peter Pan <Peter.Pan@daocloud.io> Signed-off-by: Peter Pan <peter.pan@daocloud.io> Co-authored-by: Li, Jiang <bigpyj64@gmail.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Peter Pan committed
Sep 3, 2025 at 01:27 UTC
0e1759cd54f98e628d6660becb7f5626d084d68f
4 files changed
+40
docs/getting_started/installation/cpu.md
+32
@@ -194,3 +194,35 @@ vLLM CPU supports data parallel (DP), tensor parallel (TP) and pipeline parallel
194
- Both of them require `amx` CPU flag.
195
- `VLLM_CPU_MOE_PREPACK` can provides better performance for MoE models
196
- `VLLM_CPU_SGL_KERNEL` can provides better performance for MoE models and small-batch scenarios.
197
+
198
+### Why do I see `get_mempolicy: Operation not permitted` when running in Docker?
199
+
200
+In some container environments (like Docker), NUMA-related syscalls used by vLLM (e.g., `get_mempolicy`, `migrate_pages`) are blocked/denied in the runtime's default seccomp/capabilities settings. This may lead to warnings like `get_mempolicy: Operation not permitted`. Functionality is not affected, but NUMA memory binding/migration optimizations may not take effect and performance can be suboptimal.
201
+
202
+To enable these optimizations inside Docker with the least privilege, you can follow below tips:
203
+
204
+```bash
205
+docker run ... --cap-add SYS_NICE --security-opt seccomp=unconfined ...
206
+
207
+# 1) `--cap-add SYS_NICE` is to address `get_mempolicy` EPERM issue.
208
+
209
+# 2) `--security-opt seccomp=unconfined` is to enable `migrate_pages` for `numa_migrate_pages()`.
210
+# Actually, `seccomp=unconfined` bypasses the seccomp for container,
211
+# if it's unacceptable, you can customize your own seccomp profile,
212
+# based on docker/runtime default.json and add `migrate_pages` to `SCMP_ACT_ALLOW` list.
213
+
214
+# reference : https://docs.docker.com/engine/security/seccomp/
215
+```
216
+
217
+Alternatively, running with `--privileged=true` also works but is broader and not generally recommended.
218
+
219
+In K8S, the following configuration can be added to workload yaml to achieve the same effect as above:
220
+
221
+```yaml
222
+securityContext:
223
+ seccompProfile:
224
+ type: Unconfined
225
+ capabilities:
226
+ add:
227
+ - SYS_NICE
228
+```
docs/getting_started/installation/cpu/arm.inc.md
+4
@@ -48,6 +48,10 @@ docker run --rm \
48
--dtype=bfloat16 \
49
other vLLM OpenAI server arguments
50
```
51
+
52
+!!! tip
53
+ An alternative of `--privileged=true` is `--cap-add SYS_NICE --security-opt seccomp=unconfined`.
54
+
55
# --8<-- [end:build-image-from-source]
56
# --8<-- [start:extra-information]
57
# --8<-- [end:extra-information]
docs/getting_started/installation/cpu/s390x.inc.md
+3
@@ -89,6 +89,9 @@ docker run --rm \
89
other vLLM OpenAI server arguments
90
```
91
92
+!!! tip
93
+ An alternative of `--privileged true` is `--cap-add SYS_NICE --security-opt seccomp=unconfined`.
94
+
95
# --8<-- [end:build-image-from-source]
96
# --8<-- [start:extra-information]
97
# --8<-- [end:extra-information]
docs/getting_started/installation/cpu/x86.inc.md
+1
@@ -44,6 +44,7 @@ docker build -f docker/Dockerfile.cpu \
44
# Launching OpenAI server
45
docker run --rm \
46
--security-opt seccomp=unconfined \
47
+ --cap-add SYS_NICE \
48
--shm-size=4g \
49
-p 8000:8000 \
50
-e VLLM_CPU_KVCACHE_SPACE=<KV cache space> \