@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 112 lines 3.2 KB
Rendered Raw
1 ---
2 toc_depth: 2
3 ---
4
5 # Using Docker
6
7 ## Pre-built images
8
9 --8<-- "docs/getting_started/installation/gpu.md:pre-built-images"
10
11 ## Run a vLLM Recipes configuration
12
13 [vLLM Recipes](https://recipes.vllm.ai/) can be converted into `config.yaml`
14 and `env.sh`. See the
15 [Recipes conversion tool README](../../tools/recipes/README.md) for usage.
16
17 For Docker, mount both files and source `env.sh` inside the container before
18 starting vLLM:
19
20 ```bash
21 docker run --rm --gpus all \
22 -v ~/.cache/huggingface:/root/.cache/huggingface \
23 -v "$PWD/config.yaml:/recipe/config.yaml:ro" \
24 -v "$PWD/env.sh:/recipe/env.sh:ro" \
25 -p 8000:8000 \
26 --ipc=host \
27 --entrypoint /bin/bash \
28 vllm/vllm-openai:latest \
29 -lc 'source /recipe/env.sh && exec vllm serve --config /recipe/config.yaml'
30 ```
31
32 ## Persist the compile cache across containers
33
34 Mounting the Hugging Face cache keeps model weights across containers, but each
35 new container still starts with an empty `VLLM_CACHE_ROOT` (default
36 `~/.cache/vllm`) and recompiles the model's `torch.compile` artifacts. Mount a
37 named volume at that path to reuse the inductor, Triton, and AOT artifacts from
38 the second container onward:
39
40 ```bash
41 docker run --rm --gpus all \
42 -v ~/.cache/huggingface:/root/.cache/huggingface \
43 -v vllm-cache:/root/.cache/vllm \
44 -p 8000:8000 \
45 vllm/vllm-openai:latest \
46 meta-llama/Llama-3.1-8B-Instruct
47 ```
48
49 See [Faster Startup](../configuration/optimization.md#faster-startup) for the
50 mechanism and for what invalidates the cache.
51
52 ## Run as a non-root user
53
54 The CUDA `vllm/vllm-openai` image runs as root by default for backward
55 compatibility. It is also prepared to run as the built-in `vllm` user
56 (UID 2000, GID 0):
57
58 ```bash
59 docker run --rm --gpus all \
60 --user 2000:0 \
61 -p 8000:8000 \
62 vllm/vllm-openai:latest \
63 meta-llama/Llama-3.1-8B-Instruct
64 ```
65
66 When mounting model or cache volumes for a non-root container, mount writable
67 paths under `/home/vllm` instead of `/root`. For example, mount the Hugging
68 Face cache at `/home/vllm/.cache/huggingface` and make the mounted directory
69 writable by group 0.
70
71 ```bash
72 docker run --rm --gpus all \
73 --user 2000:0 \
74 -v ~/.cache/huggingface:/home/vllm/.cache/huggingface \
75 -p 8000:8000 \
76 vllm/vllm-openai:latest \
77 meta-llama/Llama-3.1-8B-Instruct
78 ```
79
80 To build an image that defaults to the non-root `vllm` user, use the opt-in
81 `vllm-openai-nonroot` target:
82
83 ```bash
84 docker build --target vllm-openai-nonroot \
85 -t vllm-openai-nonroot:local \
86 -f docker/Dockerfile .
87
88 docker run --rm --gpus all \
89 -p 8000:8000 \
90 vllm-openai-nonroot:local \
91 meta-llama/Llama-3.1-8B-Instruct
92 ```
93
94 The `vllm-openai-nonroot` target also supports OpenShift-style arbitrary UIDs
95 when the runtime UID is a member of group 0. In Kubernetes manifests, set the
96 container security context accordingly and keep mounted cache/model paths
97 writable by group 0:
98
99 ```yaml
100 securityContext:
101 runAsNonRoot: true
102 runAsUser: 1000540000
103 runAsGroup: 0
104 fsGroup: 0
105 ```
106
107 Runtime UIDs outside group 0 are not part of the documented support matrix
108 because they may be unable to write to `/home/vllm` or `/opt/uv/cache`.
109
110 ## Build image from source
111
112 --8<-- "docs/getting_started/installation/gpu.md:build-image-from-source"