A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 8, 17:45 UTC
[DOC] [ROCm] Update docker deployment doc (#33971)
Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: TJian <tunjian.tan@embeddedllm.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
vllmellm committed
Feb 7, 2026 at 02:05 UTC
aaa2efbe9861200056a8f2dc31d2482fec89b880
4 files changed
+237
-235
docs/deployment/docker.md
+8
-156
@@ -1,161 +1,13 @@
1
-# Using Docker
2
-
3
-## Use vLLM's Official Docker Image
4
-
5
-vLLM offers an official Docker image for deployment.
6
-The image can be used to run OpenAI compatible server and is available on Docker Hub as [vllm/vllm-openai](https://hub.docker.com/r/vllm/vllm-openai/tags).
7
-
8
-```bash
9
-docker run --runtime nvidia --gpus all \
10
- -v ~/.cache/huggingface:/root/.cache/huggingface \
11
- --env "HF_TOKEN=$HF_TOKEN" \
12
- -p 8000:8000 \
13
- --ipc=host \
14
- vllm/vllm-openai:latest \
15
- --model Qwen/Qwen3-0.6B
16
-```
17
-
18
-This image can also be used with other container engines such as [Podman](https://podman.io/).
19
-
20
-```bash
21
-podman run --device nvidia.com/gpu=all \
22
- -v ~/.cache/huggingface:/root/.cache/huggingface \
23
- --env "HF_TOKEN=$HF_TOKEN" \
24
- -p 8000:8000 \
25
- --ipc=host \
26
- docker.io/vllm/vllm-openai:latest \
27
- --model Qwen/Qwen3-0.6B
28
-```
29
-
30
-You can add any other [engine-args](../configuration/engine_args.md) you need after the image tag (`vllm/vllm-openai:latest`).
31
-
32
-!!! note
33
- You can either use the `ipc=host` flag or `--shm-size` flag to allow the
34
- container to access the host's shared memory. vLLM uses PyTorch, which uses shared
35
- memory to share data between processes under the hood, particularly for tensor parallel inference.
36
-
37
-!!! note
38
- Optional dependencies are not included in order to avoid licensing issues (e.g. <https://github.com/vllm-project/vllm/issues/8030>).
39
-
40
- If you need to use those dependencies (having accepted the license terms),
41
- create a custom Dockerfile on top of the base image with an extra layer that installs them:
42
-
43
- ```Dockerfile
44
- FROM vllm/vllm-openai:v0.11.0
45
-
46
- # e.g. install the `audio` optional dependencies
47
- # NOTE: Make sure the version of vLLM matches the base image!
48
- RUN uv pip install --system vllm[audio]==0.11.0
49
- ```
50
-
51
-!!! tip
52
- Some new models may only be available on the main branch of [HF Transformers](https://github.com/huggingface/transformers).
53
-
54
- To use the development version of `transformers`, create a custom Dockerfile on top of the base image
55
- with an extra layer that installs their code from source:
56
-
57
- ```Dockerfile
58
- FROM vllm/vllm-openai:latest
59
-
60
- RUN uv pip install --system git+https://github.com/huggingface/transformers.git
61
- ```
62
-
63
-## Building vLLM's Docker Image from Source
64
-
65
-You can build and run vLLM from source via the provided [docker/Dockerfile](../../docker/Dockerfile). To build vLLM:
66
-
67
-```bash
68
-# optionally specifies: --build-arg max_jobs=8 --build-arg nvcc_threads=2
69
-DOCKER_BUILDKIT=1 docker build . \
70
- --target vllm-openai \
71
- --tag vllm/vllm-openai \
72
- --file docker/Dockerfile
73
-```
1
+---
2
+toc_depth: 2
3
+---
4
75
-!!! note
76
- By default vLLM will build for all GPU types for widest distribution. If you are just building for the
77
- current GPU type the machine is running on, you can add the argument `--build-arg torch_cuda_arch_list=""`
78
- for vLLM to find the current GPU type and build for that.
79
-
80
- If you are using Podman instead of Docker, you might need to disable SELinux labeling by
81
- adding `--security-opt label=disable` when running `podman build` command to avoid certain [existing issues](https://github.com/containers/buildah/discussions/4184).
82
-
83
-!!! note
84
- If you have not changed any C++ or CUDA kernel code, you can use precompiled wheels to significantly reduce Docker build time.
85
-
86
- * **Enable the feature** by adding the build argument: `--build-arg VLLM_USE_PRECOMPILED="1"`.
87
- * **How it works**: By default, vLLM automatically finds the correct wheels from our [Nightly Builds](../contributing/ci/nightly_builds.md) by using the merge-base commit with the upstream `main` branch.
88
- * **Override commit**: To use wheels from a specific commit, provide the `--build-arg VLLM_PRECOMPILED_WHEEL_COMMIT=<commit_hash>` argument.
89
-
90
- For a detailed explanation, refer to the documentation on 'Set up using Python-only build (without compilation)' part in [Build wheel from source](../contributing/ci/nightly_builds.md#precompiled-wheels-usage), these args are similar.
91
-
92
-## Building for Arm64/aarch64
93
-
94
-A docker container can be built for aarch64 systems such as the Nvidia Grace-Hopper and Grace-Blackwell. Using the flag `--platform "linux/arm64"` will build for arm64.
95
-
96
-!!! note
97
- Multiple modules must be compiled, so this process can take a while. Recommend using `--build-arg max_jobs=` & `--build-arg nvcc_threads=`
98
- flags to speed up build process. However, ensure your `max_jobs` is substantially larger than `nvcc_threads` to get the most benefits.
99
- Keep an eye on memory usage with parallel jobs as it can be substantial (see example below).
100
-
101
-??? console "Command"
102
-
103
- ```bash
104
- # Example of building on Nvidia GH200 server. (Memory usage: ~15GB, Build time: ~1475s / ~25 min, Image size: 6.93GB)
105
- DOCKER_BUILDKIT=1 docker build . \
106
- --file docker/Dockerfile \
107
- --target vllm-openai \
108
- --platform "linux/arm64" \
109
- -t vllm/vllm-gh200-openai:latest \
110
- --build-arg max_jobs=66 \
111
- --build-arg nvcc_threads=2 \
112
- --build-arg torch_cuda_arch_list="9.0 10.0+PTX" \
113
- --build-arg RUN_WHEEL_CHECK=false
114
- ```
115
-
116
-For (G)B300, we recommend using CUDA 13, as shown in the following command.
117
-
118
-??? console "Command"
119
-
120
- ```bash
121
- DOCKER_BUILDKIT=1 docker build \
122
- --build-arg CUDA_VERSION=13.0.1 \
123
- --build-arg BUILD_BASE_IMAGE=nvidia/cuda:13.0.1-devel-ubuntu22.04 \
124
- --build-arg max_jobs=256 \
125
- --build-arg nvcc_threads=2 \
126
- --build-arg RUN_WHEEL_CHECK=false \
127
- --build-arg torch_cuda_arch_list='9.0 10.0+PTX' \
128
- --platform "linux/arm64" \
129
- --tag vllm/vllm-gb300-openai:latest \
130
- --target vllm-openai \
131
- -f docker/Dockerfile \
132
- .
133
- ```
134
-
135
-!!! note
136
- If you are building the `linux/arm64` image on a non-ARM host (e.g., an x86_64 machine), you need to ensure your system is set up for cross-compilation using QEMU. This allows your host machine to emulate ARM64 execution.
137
-
138
- Run the following command on your host machine to register QEMU user static handlers:
139
-
140
- ```bash
141
- docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
142
- ```
143
-
144
- After setting up QEMU, you can use the `--platform "linux/arm64"` flag in your `docker build` command.
145
-
146
-## Use the custom-built vLLM Docker image
5
+# Using Docker
6
148
-To run vLLM with the custom-built Docker image:
7
+## Pre-built images
8
150
-```bash
151
-docker run --runtime nvidia --gpus all \
152
- -v ~/.cache/huggingface:/root/.cache/huggingface \
153
- -p 8000:8000 \
154
- --env "HF_TOKEN=<secret>" \
155
- vllm/vllm-openai <args...>
156
-```
9
+--8<-- "docs/getting_started/installation/gpu.md:pre-built-images"
10
158
-The argument `vllm/vllm-openai` specifies the image to run, and should be replaced with the name of the custom-built image (the `-t` tag from the build command).
11
+## Build image from source
12
160
-!!! note
161
- **For version 0.4.1 and 0.4.2 only** - the vLLM docker images under these versions are supposed to be run under the root user since a library under the root user's home directory, i.e. `/root/.config/vllm/nccl/cu12/libnccl.so.2.18.1` is required to be loaded during runtime. If you are running the container under a different user, you may need to first change the permissions of the library (and all the parent directories) to allow the user to access it, then run vLLM with environment variable `VLLM_NCCL_SO_PATH=/root/.config/vllm/nccl/cu12/libnccl.so.2.18.1` .
13
+--8<-- "docs/getting_started/installation/gpu.md:build-image-from-source"
docs/getting_started/installation/gpu.cuda.inc.md
+149
-8
@@ -239,27 +239,168 @@ uv pip install -e .
239
# --8<-- [end:build-wheel-from-source]
240
# --8<-- [start:pre-built-images]
241
242
-See [Using Docker](../../deployment/docker.md) for instructions on using the official Docker image.
242
+vLLM offers an official Docker image for deployment.
243
+The image can be used to run OpenAI compatible server and is available on Docker Hub as [vllm/vllm-openai](https://hub.docker.com/r/vllm/vllm-openai/tags).
244
244
-Another way to access the latest code is to use the docker images:
245
+```bash
246
+docker run --runtime nvidia --gpus all \
247
+ -v ~/.cache/huggingface:/root/.cache/huggingface \
248
+ --env "HF_TOKEN=$HF_TOKEN" \
249
+ -p 8000:8000 \
250
+ --ipc=host \
251
+ vllm/vllm-openai:latest \
252
+ --model Qwen/Qwen3-0.6B
253
+```
254
+
255
+This image can also be used with other container engines such as [Podman](https://podman.io/).
256
257
```bash
247
-export VLLM_COMMIT=33f460b17a54acb3b6cc0b03f4a17876cff5eafd # use full commit hash from the main branch
248
-docker pull public.ecr.aws/q9t5s3a7/vllm-ci-postmerge-repo:${VLLM_COMMIT}
258
+podman run --device nvidia.com/gpu=all \
259
+-v ~/.cache/huggingface:/root/.cache/huggingface \
260
+--env "HF_TOKEN=$HF_TOKEN" \
261
+-p 8000:8000 \
262
+--ipc=host \
263
+docker.io/vllm/vllm-openai:latest \
264
+--model Qwen/Qwen3-0.6B
265
```
266
251
-These docker images are used for CI and testing only, and they are not intended for production use. They will be expired after several days.
267
+You can add any other [engine-args](https://docs.vllm.ai/en/latest/configuration/engine_args/) you need after the image tag (`vllm/vllm-openai:latest`).
268
+
269
+!!! note
270
+ You can either use the `ipc=host` flag or `--shm-size` flag to allow the
271
+ container to access the host's shared memory. vLLM uses PyTorch, which uses shared
272
+ memory to share data between processes under the hood, particularly for tensor parallel inference.
273
+
274
+!!! note
275
+ Optional dependencies are not included in order to avoid licensing issues (e.g. <https://github.com/vllm-project/vllm/issues/8030>).
276
+
277
+ If you need to use those dependencies (having accepted the license terms),
278
+ create a custom Dockerfile on top of the base image with an extra layer that installs them:
279
+
280
+ ```Dockerfile
281
+ FROM vllm/vllm-openai:v0.11.0
282
+
283
+ # e.g. install the `audio` optional dependencies
284
+ # NOTE: Make sure the version of vLLM matches the base image!
285
+ RUN uv pip install --system vllm[audio]==0.11.0
286
+ ```
287
+
288
+!!! tip
289
+ Some new models may only be available on the main branch of [HF Transformers](https://github.com/huggingface/transformers).
290
+
291
+ To use the development version of `transformers`, create a custom Dockerfile on top of the base image
292
+ with an extra layer that installs their code from source:
293
253
-The latest code can contain bugs and may not be stable. Please use it with caution.
294
+ ```Dockerfile
295
+ FROM vllm/vllm-openai:latest
296
+
297
+ RUN uv pip install --system git+https://github.com/huggingface/transformers.git
298
+ ```
299
300
# --8<-- [end:pre-built-images]
301
# --8<-- [start:build-image-from-source]
302
258
-See [Building vLLM's Docker Image from Source](../../deployment/docker.md#building-vllms-docker-image-from-source) for instructions on building the Docker image.
303
+You can build and run vLLM from source via the provided [docker/Dockerfile](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile). To build vLLM:
304
+
305
+```bash
306
+# optionally specifies: --build-arg max_jobs=8 --build-arg nvcc_threads=2
307
+DOCKER_BUILDKIT=1 docker build . \
308
+ --target vllm-openai \
309
+ --tag vllm/vllm-openai \
310
+ --file docker/Dockerfile
311
+```
312
+
313
+!!! note
314
+ By default vLLM will build for all GPU types for widest distribution. If you are just building for the
315
+ current GPU type the machine is running on, you can add the argument `--build-arg torch_cuda_arch_list=""`
316
+ for vLLM to find the current GPU type and build for that.
317
+
318
+ If you are using Podman instead of Docker, you might need to disable SELinux labeling by
319
+ adding `--security-opt label=disable` when running `podman build` command to avoid certain [existing issues](https://github.com/containers/buildah/discussions/4184).
320
+
321
+!!! note
322
+ If you have not changed any C++ or CUDA kernel code, you can use precompiled wheels to significantly reduce Docker build time.
323
+
324
+ * **Enable the feature** by adding the build argument: `--build-arg VLLM_USE_PRECOMPILED="1"`.
325
+ * **How it works**: By default, vLLM automatically finds the correct wheels from our [Nightly Builds](https://docs.vllm.ai/en/latest/contributing/ci/nightly_builds/) by using the merge-base commit with the upstream `main` branch.
326
+ * **Override commit**: To use wheels from a specific commit, provide the `--build-arg VLLM_PRECOMPILED_WHEEL_COMMIT=<commit_hash>` argument.
327
+
328
+ For a detailed explanation, refer to the documentation on 'Set up using Python-only build (without compilation)' part in [Build wheel from source](https://docs.vllm.ai/en/latest/contributing/ci/nightly_builds/#precompiled-wheels-usage), these args are similar.
329
+
330
+#### Building vLLM's Docker Image from Source for Arm64/aarch64
331
+
332
+A docker container can be built for aarch64 systems such as the Nvidia Grace-Hopper and Grace-Blackwell. Using the flag `--platform "linux/arm64"` will build for arm64.
333
+
334
+!!! note
335
+ Multiple modules must be compiled, so this process can take a while. Recommend using `--build-arg max_jobs=` & `--build-arg nvcc_threads=`
336
+ flags to speed up build process. However, ensure your `max_jobs` is substantially larger than `nvcc_threads` to get the most benefits.
337
+ Keep an eye on memory usage with parallel jobs as it can be substantial (see example below).
338
+
339
+??? console "Command"
340
+
341
+ ```bash
342
+ # Example of building on Nvidia GH200 server. (Memory usage: ~15GB, Build time: ~1475s / ~25 min, Image size: 6.93GB)
343
+ DOCKER_BUILDKIT=1 docker build . \
344
+ --file docker/Dockerfile \
345
+ --target vllm-openai \
346
+ --platform "linux/arm64" \
347
+ -t vllm/vllm-gh200-openai:latest \
348
+ --build-arg max_jobs=66 \
349
+ --build-arg nvcc_threads=2 \
350
+ --build-arg torch_cuda_arch_list="9.0 10.0+PTX" \
351
+ --build-arg RUN_WHEEL_CHECK=false
352
+ ```
353
+
354
+For (G)B300, we recommend using CUDA 13, as shown in the following command.
355
+
356
+??? console "Command"
357
+
358
+ ```bash
359
+ DOCKER_BUILDKIT=1 docker build \
360
+ --build-arg CUDA_VERSION=13.0.1 \
361
+ --build-arg BUILD_BASE_IMAGE=nvidia/cuda:13.0.1-devel-ubuntu22.04 \
362
+ --build-arg max_jobs=256 \
363
+ --build-arg nvcc_threads=2 \
364
+ --build-arg RUN_WHEEL_CHECK=false \
365
+ --build-arg torch_cuda_arch_list='9.0 10.0+PTX' \
366
+ --platform "linux/arm64" \
367
+ --tag vllm/vllm-gb300-openai:latest \
368
+ --target vllm-openai \
369
+ -f docker/Dockerfile \
370
+ .
371
+ ```
372
+
373
+!!! note
374
+ If you are building the `linux/arm64` image on a non-ARM host (e.g., an x86_64 machine), you need to ensure your system is set up for cross-compilation using QEMU. This allows your host machine to emulate ARM64 execution.
375
+
376
+ Run the following command on your host machine to register QEMU user static handlers:
377
+
378
+ ```bash
379
+ docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
380
+ ```
381
+
382
+ After setting up QEMU, you can use the `--platform "linux/arm64"` flag in your `docker build` command.
383
+
384
+#### Use the custom-built vLLM Docker image**
385
+
386
+To run vLLM with the custom-built Docker image:
387
+
388
+```bash
389
+docker run --runtime nvidia --gpus all \
390
+ -v ~/.cache/huggingface:/root/.cache/huggingface \
391
+ -p 8000:8000 \
392
+ --env "HF_TOKEN=<secret>" \
393
+ vllm/vllm-openai <args...>
394
+```
395
+
396
+The argument `vllm/vllm-openai` specifies the image to run, and should be replaced with the name of the custom-built image (the `-t` tag from the build command).
397
+
398
+!!! note
399
+ **For version 0.4.1 and 0.4.2 only** - the vLLM docker images under these versions are supposed to be run under the root user since a library under the root user's home directory, i.e. `/root/.config/vllm/nccl/cu12/libnccl.so.2.18.1` is required to be loaded during runtime. If you are running the container under a different user, you may need to first change the permissions of the library (and all the parent directories) to allow the user to access it, then run vLLM with environment variable `VLLM_NCCL_SO_PATH=/root/.config/vllm/nccl/cu12/libnccl.so.2.18.1` .
400
401
# --8<-- [end:build-image-from-source]
402
# --8<-- [start:supported-features]
403
404
See [Feature x Hardware](../../features/README.md#feature-x-hardware) compatibility matrix for feature support information.
405
265
-# --8<-- [end:supported-features]
406
+# --8<-- [end:supported-features]
\ No newline at end of file
docs/getting_started/installation/gpu.md
+18
@@ -1,3 +1,7 @@
1
+---
2
+toc_depth: 3
3
+---
4
+
5
# GPU
6
7
vLLM is a Python library that supports the following GPU variants. Select your GPU type to see vendor specific instructions:
@@ -84,6 +88,9 @@ vLLM is a Python library that supports the following GPU variants. Select your G
88
89
### Pre-built images
90
91
+<!-- markdownlint-disable MD025 -->
92
+# --8<-- [start:pre-built-images]
93
+
94
=== "NVIDIA CUDA"
95
96
--8<-- "docs/getting_started/installation/gpu.cuda.inc.md:pre-built-images"
@@ -96,7 +103,15 @@ vLLM is a Python library that supports the following GPU variants. Select your G
103
104
--8<-- "docs/getting_started/installation/gpu.xpu.inc.md:pre-built-images"
105
106
+# --8<-- [end:pre-built-images]
107
+<!-- markdownlint-enable MD025 -->
108
+
109
+<!-- markdownlint-disable MD001 -->
110
### Build image from source
111
+<!-- markdownlint-enable MD001 -->
112
+
113
+<!-- markdownlint-disable MD025 -->
114
+# --8<-- [start:build-image-from-source]
115
116
=== "NVIDIA CUDA"
117
@@ -110,6 +125,9 @@ vLLM is a Python library that supports the following GPU variants. Select your G
125
126
--8<-- "docs/getting_started/installation/gpu.xpu.inc.md:build-image-from-source"
127
128
+# --8<-- [end:build-image-from-source]
129
+<!-- markdownlint-enable MD025 -->
130
+
131
## Supported features
132
133
=== "NVIDIA CUDA"
docs/getting_started/installation/gpu.rocm.inc.md
+62
-71
@@ -174,67 +174,44 @@ uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.15.0/rocm700
174
# --8<-- [end:build-wheel-from-source]
175
# --8<-- [start:pre-built-images]
176
177
-#### Use vLLM's Official Docker Image
178
-
177
vLLM offers an official Docker image for deployment.
178
The image can be used to run OpenAI compatible server and is available on Docker Hub as [vllm/vllm-openai-rocm](https://hub.docker.com/r/vllm/vllm-openai-rocm/tags).
179
182
-???+ console "Commands"
183
- ```bash
184
- docker run --rm \
185
- --group-add=video \
186
- --cap-add=SYS_PTRACE \
187
- --security-opt seccomp=unconfined \
188
- --device /dev/kfd \
189
- --device /dev/dri \
190
- -v ~/.cache/huggingface:/root/.cache/huggingface \
191
- --env "HF_TOKEN=$HF_TOKEN" \
192
- -p 8000:8000 \
193
- --ipc=host \
194
- vllm/vllm-openai-rocm:latest \
195
- --model Qwen/Qwen3-0.6B
196
- ```
197
-
198
-To use the docker image as base for development, you can launch it in interactive session through overriding the entrypoint.
199
-
200
-???+ console "Commands"
201
- ```bash
202
- docker run --rm -it \
203
- --group-add=video \
204
- --cap-add=SYS_PTRACE \
205
- --security-opt seccomp=unconfined \
206
- --device /dev/kfd \
207
- --device /dev/dri \
208
- -v ~/.cache/huggingface:/root/.cache/huggingface \
209
- --env "HF_TOKEN=$HF_TOKEN" \
210
- -p 8000:8000 \
211
- --ipc=host \
212
- --entrypoint bash \
213
- vllm/vllm-openai-rocm:latest
214
- ```
215
-
216
-
217
-#### Use AMD's Docker Images
218
-
219
-The [AMD Infinity hub for vLLM](https://hub.docker.com/r/rocm/vllm/tags) offers a prebuilt, optimized
220
-docker image designed for validating inference performance on the AMD Instinct™ MI300X accelerator.
221
-AMD also offers nightly prebuilt docker image from [Docker Hub](https://hub.docker.com/r/rocm/vllm-dev), which has vLLM and all its dependencies installed. The entrypoint of this docker image is `/bin/bash` (different from the vLLM's Official Docker Image).
222
-
223
-???+ console "Commands"
224
- ```bash
225
- docker pull rocm/vllm-dev:nightly # to get the latest image
226
- docker run -it --rm \
227
- --network=host \
180
+```bash
181
+docker run --rm \
182
--group-add=video \
229
- --ipc=host \
183
--cap-add=SYS_PTRACE \
184
--security-opt seccomp=unconfined \
185
--device /dev/kfd \
186
--device /dev/dri \
234
- -v <path/to/your/models>:/app/models \
235
- -e HF_HOME="/app/models" \
236
- rocm/vllm-dev:nightly
237
- ```
187
+ -v ~/.cache/huggingface:/root/.cache/huggingface \
188
+ --env "HF_TOKEN=$HF_TOKEN" \
189
+ -p 8000:8000 \
190
+ --ipc=host \
191
+ vllm/vllm-openai-rocm:latest \
192
+ --model Qwen/Qwen3-0.6B
193
+```
194
+
195
+#### Use AMD's Docker Images
196
+
197
+Prior to January 20th, 2026 when the official docker images are available on [upstream vLLM docker hub](https://hub.docker.com/v2/repositories/vllm/vllm-openai-rocm/tags/), the [AMD Infinity hub for vLLM](https://hub.docker.com/r/rocm/vllm/tags) offers a prebuilt, optimized
198
+docker image designed for validating inference performance on the AMD Instinct MI300X™ accelerator.
199
+AMD also offers nightly prebuilt docker image from [Docker Hub](https://hub.docker.com/r/rocm/vllm-dev), which has vLLM and all its dependencies installed. The entrypoint of this docker image is `/bin/bash` (different from the vLLM's Official Docker Image).
200
+
201
+```bash
202
+docker pull rocm/vllm-dev:nightly # to get the latest image
203
+docker run -it --rm \
204
+--network=host \
205
+--group-add=video \
206
+--ipc=host \
207
+--cap-add=SYS_PTRACE \
208
+--security-opt seccomp=unconfined \
209
+--device /dev/kfd \
210
+--device /dev/dri \
211
+-v <path/to/your/models>:/app/models \
212
+-e HF_HOME="/app/models" \
213
+rocm/vllm-dev:nightly
214
+```
215
216
!!! tip
217
Please check [LLM inference performance validation on AMD Instinct MI300X](https://rocm.docs.amd.com/en/latest/how-to/performance-validation/mi300x/vllm-benchmark.html)
@@ -243,7 +220,7 @@ AMD also offers nightly prebuilt docker image from [Docker Hub](https://hub.dock
220
# --8<-- [end:pre-built-images]
221
# --8<-- [start:build-image-from-source]
222
246
-Building the Docker image from source is the recommended way to use vLLM with ROCm.
223
+You can build and run vLLM from source via the provided [docker/Dockerfile.rocm](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm).
224
225
??? info "(Optional) Build an image with ROCm software stack"
226
@@ -269,8 +246,6 @@ Building the Docker image from source is the recommended way to use vLLM with RO
246
-t rocm/vllm-dev:base .
247
```
248
272
-#### Build an image with vLLM
273
-
249
First, build a docker image from [docker/Dockerfile.rocm](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm) and launch a docker container from the image.
250
It is important that the user kicks off the docker build using buildkit. Either the user put `DOCKER_BUILDKIT=1` as environment variable when calling docker build command, or the user needs to set up buildkit in the docker daemon configuration /etc/docker/daemon.json as follows and restart the daemon:
251
@@ -292,30 +267,46 @@ Their values can be passed in when running `docker build` with `--build-arg` opt
267
268
To build vllm on ROCm 7.0 for MI200 and MI300 series, you can use the default (which build a docker image with `vllm serve` as entrypoint):
269
295
-???+ console "Commands"
296
- ```bash
297
- DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.rocm -t vllm-rocm .
298
- ```
270
+```bash
271
+DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.rocm -t vllm/vllm-openai-rocm .
272
+```
273
300
-To run the above docker image `vllm-rocm`, use the below command:
274
275
+To run vLLM with the custom-built Docker image:
276
303
-???+ console "Commands"
304
- ```bash
305
- docker run -it \
306
- --network=host \
277
+```bash
278
+docker run --rm \
279
--group-add=video \
308
- --ipc=host \
280
--cap-add=SYS_PTRACE \
281
--security-opt seccomp=unconfined \
282
--device /dev/kfd \
283
--device /dev/dri \
313
- -v <path/to/model>:/app/model \
314
- vllm-rocm \
315
- --model Qwen/Qwen3-0.6B
316
- ```
284
+ -v ~/.cache/huggingface:/root/.cache/huggingface \
285
+ --env "HF_TOKEN=$HF_TOKEN" \
286
+ -p 8000:8000 \
287
+ --ipc=host \
288
+ vllm/vllm-openai-rocm <args...>
289
+```
290
+
291
+The argument `vllm/vllm-openai-rocm` specifies the image to run, and should be replaced with the name of the custom-built image (the `-t` tag from the build command).
292
+
293
+To use the docker image as base for development, you can launch it in interactive session through overriding the entrypoint.
294
318
-Where the `<path/to/model>` is the location where the model is stored, for example, the weights for llama2 or llama3 models.
295
+???+ console "Commands"
296
+ ```bash
297
+ docker run --rm -it \
298
+ --group-add=video \
299
+ --cap-add=SYS_PTRACE \
300
+ --security-opt seccomp=unconfined \
301
+ --device /dev/kfd \
302
+ --device /dev/dri \
303
+ -v ~/.cache/huggingface:/root/.cache/huggingface \
304
+ --env "HF_TOKEN=$HF_TOKEN" \
305
+ --network=host \
306
+ --ipc=host \
307
+ --entrypoint bash \
308
+ vllm/vllm-openai-rocm
309
+ ```
310
311
# --8<-- [end:build-image-from-source]
312
# --8<-- [start:supported-features]