@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 499 lines 22.4 KB
Rendered Raw
1 <!-- markdownlint-disable MD041 MD051 -->
2 --8<-- [start:installation]
3
4 vLLM supports AMD GPUs with ROCm 6.3 or above. Pre-built wheels are available for ROCm 7.0 and ROCm 7.2.1.
5
6 #### Prebuilt Wheels
7
8 | ROCm Variant | Python Version | ROCm Version | glibc Requirement | Supported Versions |
9 | ------------ | -------------- | ------------ | ----------------- | ------------------ |
10 | `rocm700` | 3.12 | 7.0 | >= 2.35 | `0.14.0` to `0.18.0` |
11 | `rocm721` | 3.12 | 7.2.1 | >= 2.35 | Nightly releases after commit `171775f306a333a9cf105bfd533bf3e113d401d9` |
12
13 --8<-- [end:installation]
14 --8<-- [start:requirements]
15
16 - GPU: MI200s (gfx90a), MI300 (gfx942), MI350 (gfx950), Radeon RX 7900 series (gfx1100/1101), Radeon RX 9000 series (gfx1200/1201), Ryzen AI MAX / AI 300 Series (gfx1151/1150)
17 - ROCm 6.3 or above
18 - MI350 requires ROCm 7.0 or above
19 - Ryzen AI MAX / AI 300 Series requires ROCm 7.0.2 or above
20
21 --8<-- [end:requirements]
22 --8<-- [start:set-up-using-python]
23
24 The vLLM wheel bundles PyTorch and all required dependencies, and you should use the included PyTorch for compatibility. Because vLLM compiles many ROCm kernels to ensure a validated, high‑performance stack, the resulting binaries may not be compatible with other ROCm or PyTorch builds.
25 If you need a different ROCm version or want to use an existing PyTorch installation, you’ll need to build vLLM from source. See [below](#build-wheel-from-source) for more details.
26
27 --8<-- [end:set-up-using-python]
28 --8<-- [start:pre-built-wheels]
29
30 !!! warning "Python 3.12 required for ROCm wheels"
31
32 ROCm pre-built wheels are only available for **Python 3.12**. If you are using a different Python version (e.g. 3.11 or 3.13), the installer **will silently fall back** to the CUDA wheel from PyPI, which will fail on AMD GPUs with errors like `libcudart.so: cannot open shared object file`.
33
34 To check your Python version: `python3 --version`
35
36 If you need Python 3.12, you can create an isolated environment with `uv`:
37
38 ```bash
39 uv venv --python 3.12 --seed --managed-python
40 source .venv/bin/activate
41 ```
42
43 To install the latest version of vLLM for Python 3.12, ROCm 7.0 and `glibc >= 2.35`.
44
45 ```bash
46 uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/ --upgrade
47 ```
48
49 !!! tip
50 You can find out about which ROCm version the latest vLLM supports by checking the `vllm` package in index in extra-index-url <https://wheels.vllm.ai/rocm/> at [https://wheels.vllm.ai/rocm/vllm](https://wheels.vllm.ai/rocm/vllm) .
51
52 Another approach is that you can use this following commands to automatically extract the wheel variants:
53
54 ```bash
55 # automatically extract the available rocm variant
56 export VLLM_ROCM_VARIANT=$(curl -s https://wheels.vllm.ai/rocm/vllm | grep -oP 'rocm\d+' | head -1)
57
58 # automatically extract the vLLM version
59 export VLLM_VERSION=$(curl -s https://wheels.vllm.ai/rocm/vllm | grep -oP 'vllm-\K[0-9.]+' | head -1)
60
61 # inspect if the ROCm version is compatible with your environment
62 echo $VLLM_ROCM_VARIANT
63 echo $VLLM_VERSION
64 ```
65
66 To install a specific version and ROCm variant of vLLM wheel.
67
68 ```bash
69 # version without the `v`
70 uv pip install vllm==${VLLM_VERSION} --extra-index-url https://wheels.vllm.ai/rocm/${VLLM_VERSION}/${VLLM_ROCM_VARIANT}
71
72 # Example
73 uv pip install vllm==0.18.0 --extra-index-url https://wheels.vllm.ai/rocm/0.18.0/rocm700
74 ```
75
76 !!! warning "Caveats for using `pip`"
77
78 We recommend leveraging `uv` to install the vLLM wheel. Using `pip` to install from custom indices is cumbersome because `pip` combines packages from `--extra-index-url` and the default index, choosing only the latest version. This makes it difficult to install a wheel from a custom index unless exact versions of all packages are specified. In contrast, `uv` gives the extra index [higher priority than the default index](https://docs.astral.sh/uv/pip/compatibility/#packages-that-exist-on-multiple-indexes).
79
80 If you insist on using `pip`, you need to specify the exact vLLM version in the package name and provide the custom index URL `https://wheels.vllm.ai/rocm/${VLLM_VERSION}/${VLLM_ROCM_VARIANT}` via `--extra-index-url`.
81
82 ```bash
83 pip install vllm==0.18.0+rocm700 --extra-index-url https://wheels.vllm.ai/rocm/0.18.0/rocm700
84 ```
85
86 #### Install the latest code
87
88 LLM inference is a fast-evolving field, and the latest code may contain bug fixes, performance improvements, and new features that are not released yet. To allow users to try the latest code without waiting for the next release, vLLM provides wheels for every commit since commit `171775f306a333a9cf105bfd533bf3e113d401d9` on <https://wheels.vllm.ai/rocm/nightly/>. The custom index to be used is `https://wheels.vllm.ai/rocm/nightly/${VLLM_ROCM_VARIANT}`
89
90 **NOTE:** The first ROCm Variant that supports nightly wheel is ROCm 7.2.1
91
92 To install from latest nightly index, run:
93
94 ```bash
95 # automatically extract the available rocm variant
96 export VLLM_ROCM_VARIANT=$(curl -s https://wheels.vllm.ai/rocm/nightly | \
97 grep -oP 'rocm\d+' | head -1 | sed 's/%2B/+/g')
98
99 # inspect if the ROCm version is compatible with your environment
100 echo $VLLM_ROCM_VARIANT
101
102 uv pip install --pre vllm \
103 --extra-index-url https://wheels.vllm.ai/rocm/nightly/${VLLM_ROCM_VARIANT} \
104 --index-strategy unsafe-best-match
105 ```
106
107 ##### Install specific revisions
108
109 If you want to access the wheels for previous commits (e.g. to bisect the behavior change, performance regression), you can specify the commit hash in the URL, example:
110
111 ```bash
112 export VLLM_COMMIT=5b8c30d62b754b575e043ce2fc0dcbf8a64f6306
113
114 export VLLM_ROCM_VARIANT=$(curl -s https://wheels.vllm.ai/rocm/${VLLM_COMMIT} | \
115 grep -oP 'rocm\d+' | head -1 | sed 's/%2B/+/g')
116
117 # Extract the version from the wheel URL
118 export VLLM_VERSION=$(curl -s https://wheels.vllm.ai/rocm/${VLLM_COMMIT}/${VLLM_ROCM_VARIANT}/vllm/ | \
119 grep -oP 'vllm-\K[^-]+' | head -1 | sed 's/%2B/+/g')
120
121 # inspect the version if it is compatible with the ROCm version of your environment
122 echo $VLLM_ROCM_VARIANT
123 echo $VLLM_VERSION
124
125 uv pip install vllm==${VLLM_VERSION} \
126 --extra-index-url https://wheels.vllm.ai/rocm/${VLLM_COMMIT}/${VLLM_ROCM_VARIANT} \
127 --index-strategy unsafe-best-match
128 ```
129
130 !!! warning "`pip` caveat"
131
132 Using `pip` to install from nightly indices is _not supported_, because `pip` combines packages from `--extra-index-url` and the default index, choosing only the latest version, which makes it difficult to install a development version prior to the released version. In contrast, `uv` gives the extra index [higher priority than the default index](https://docs.astral.sh/uv/pip/compatibility/#packages-that-exist-on-multiple-indexes).
133
134 If you insist on using `pip`, you need to specify the exact vLLM version in the package name and provide the custom index URL (which can be obtained from the web page).
135
136 ```bash
137 export VLLM_COMMIT=5b8c30d62b754b575e043ce2fc0dcbf8a64f6306
138
139 export VLLM_ROCM_VARIANT=$(curl -s https://wheels.vllm.ai/rocm/${VLLM_COMMIT} | \
140 grep -oP 'rocm\d+' | head -1 | sed 's/%2B/+/g')
141
142 # Extract the version from the wheel URL
143 export VLLM_VERSION=$(curl -s https://wheels.vllm.ai/rocm/${VLLM_COMMIT}/${VLLM_ROCM_VARIANT}/vllm/ | \
144 grep -oP 'vllm-\K[^-]+' | head -1 | sed 's/%2B/+/g')
145
146 # inspect the version if it is compatible with the ROCm version of your environment
147 echo $VLLM_ROCM_VARIANT
148 echo $VLLM_VERSION
149
150 pip install vllm==${VLLM_VERSION} \
151 --extra-index-url https://wheels.vllm.ai/rocm/${VLLM_COMMIT}/${VLLM_ROCM_VARIANT}
152 ```
153
154 --8<-- [end:pre-built-wheels]
155 --8<-- [start:build-wheel-from-source]
156
157 #### Set up using Python-only build (without compilation) {#python-only-build}
158
159 If you only need to change Python code, you can build and install vLLM without
160 compilation. Changes you make to the code will be reflected when you run vLLM:
161
162 ```bash
163 git clone https://github.com/vllm-project/vllm.git
164 cd vllm
165 VLLM_USE_PRECOMPILED=1 python3 setup.py develop
166 ```
167
168 This command will do the following:
169
170 1. Look for the current branch in your vLLM clone.
171 1. Identify the corresponding base commit in the main branch.
172 1. Detect the ROCm version in your environment and select the matching wheel
173 variant.
174 1. Download the pre-built wheel of the base commit.
175 1. Use its compiled libraries and `vllm-rs` binary in the installation.
176
177 !!! note
178 1. If you change C++, HIP, or kernel code, you cannot use Python-only build;
179 otherwise you may see an import error about a library not being found or
180 an undefined symbol.
181 2. If you rebase your development branch, it is recommended to uninstall
182 vLLM and re-run the above command to make sure your libraries are up to
183 date.
184
185 !!! tip "Rebuilding the Rust frontend"
186 If you need to recompile the `vllm-rs` Rust frontend binary, you can rebuild and
187 install it without re-running the full installation:
188
189 ```bash
190 ./tools/build_rust.sh # release build
191 ./tools/build_rust.sh --debug # faster build for development
192 ```
193
194 This will install the required Rust toolchain if needed, build the binary,
195 and place it in `vllm/vllm-rs`.
196
197 If you see an error about a wheel not being found, the wheel for your base
198 commit and ROCm patch version might not be available. Check the available
199 variants under `https://wheels.vllm.ai/rocm/<commit>/`. For example, ROCm 7.2.1
200 uses the `rocm721` variant.
201
202 There are more environment variables to control the behavior of Python-only
203 build:
204
205 - `VLLM_PRECOMPILED_WHEEL_LOCATION`: specify the exact wheel URL or local file
206 path of a pre-compiled wheel to use. All other logic to find the wheel will be
207 skipped.
208 - `VLLM_PRECOMPILED_WHEEL_COMMIT`: override the full commit hash used to
209 download the pre-compiled wheel.
210 - `VLLM_PRECOMPILED_WHEEL_VARIANT`: specify the ROCm variant subdirectory, e.g.,
211 `rocm700` or `rocm721`. If not specified, the variant is auto-detected based
212 on your system's ROCm version. An explicitly specified variant must match the
213 detected environment.
214
215 You can find more information about vLLM's wheels in
216 [Install the latest code](#install-the-latest-code).
217
218 !!! note
219 There is a possibility that your source code may have a different commit ID
220 compared to the vLLM wheel, which could potentially lead to unknown errors.
221 It is recommended to use the same commit ID for the source code as the vLLM
222 wheel you have installed. Please refer to
223 [Install the latest code](#install-the-latest-code) for instructions on how
224 to install a specified wheel.
225
226 #### Full build (with compilation) {#full-build}
227
228 !!! tip
229 - If you found that the following installation step does not work for you, please refer to [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base). Dockerfile is a form of installation steps.
230
231 0. Install prerequisites (skip if you are already in an environment/docker with the following installed):
232
233 - [ROCm](https://rocm.docs.amd.com/en/latest/deploy/linux/index.html)
234 - [PyTorch](https://pytorch.org/)
235
236 For installing PyTorch, you can start from a fresh docker image, e.g, `rocm/pytorch:rocm7.0_ubuntu22.04_py3.10_pytorch_release_2.8.0`, `rocm/pytorch-nightly`. If you are using docker image, you can skip to Step 3.
237
238 Alternatively, you can install PyTorch using PyTorch wheels. You can check PyTorch installation guide in PyTorch [Getting Started](https://pytorch.org/get-started/locally/). Example:
239
240 ```bash
241 # Install PyTorch
242 pip uninstall torch -y
243 pip install --no-cache-dir torch torchvision --index-url https://download.pytorch.org/whl/nightly/rocm7.0
244 ```
245
246 1. Install [Triton for ROCm](https://github.com/ROCm/triton.git)
247
248 Install ROCm's Triton following the instructions from [ROCm/triton](https://github.com/ROCm/triton.git)
249
250 ```bash
251 python3 -m pip install ninja cmake wheel pybind11
252 pip uninstall -y triton
253 git clone https://github.com/ROCm/triton.git
254 cd triton
255 # git checkout $TRITON_BRANCH
256 git checkout f9e5bf54
257 if [ ! -f setup.py ]; then cd python; fi
258 python3 setup.py install
259 cd ../..
260 ```
261
262 !!! note
263 - The validated `$TRITON_BRANCH` can be found in the [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base).
264 - If you see HTTP issue related to downloading packages during building triton, please try again as the HTTP error is intermittent.
265
266 2. Optionally, if you choose to use CK flash attention, you can install [flash attention for ROCm](https://github.com/Dao-AILab/flash-attention.git)
267
268 Install ROCm's flash attention (v2.8.0) following the instructions from [ROCm/flash-attention](https://github.com/Dao-AILab/flash-attention#amd-rocm-support)
269
270 For example, for ROCm 7.0, suppose your gfx arch is `gfx942`. To get your gfx architecture, run `rocminfo |grep gfx`.
271
272 ```bash
273 git clone https://github.com/Dao-AILab/flash-attention.git
274 cd flash-attention
275 # git checkout $FA_BRANCH
276 git checkout 0e60e394
277 git submodule update --init
278 GPU_ARCHS="gfx942" python3 setup.py install
279 cd ..
280 ```
281
282 !!! note
283 - The validated `$FA_BRANCH` can be found in the [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base).
284
285 3. Optionally, if you choose to build AITER yourself to use a certain branch or commit, you can build AITER using the following steps:
286
287 ```bash
288 python3 -m pip uninstall -y aiter
289 git clone --recursive https://github.com/ROCm/aiter.git
290 cd aiter
291 git checkout $AITER_BRANCH_OR_COMMIT
292 git submodule sync; git submodule update --init --recursive
293 python3 setup.py develop
294 ```
295
296 !!! note
297 - You will need to config the `$AITER_BRANCH_OR_COMMIT` for your purpose.
298 - The validated `$AITER_BRANCH_OR_COMMIT` can be found in the [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base).
299
300 4. Optionally, if you want to use MORI for EP or PD disaggregation, you can install [MORI](https://github.com/ROCm/mori) using the following steps:
301
302 ```bash
303 git clone https://github.com/ROCm/mori.git
304 cd mori
305 git checkout $MORI_BRANCH_OR_COMMIT
306 git submodule sync; git submodule update --init --recursive
307 MORI_GPU_ARCHS="gfx942;gfx950" python3 setup.py install
308 ```
309
310 !!! note
311 - You will need to config the `$MORI_BRANCH_OR_COMMIT` for your purpose.
312 - The validated `$MORI_BRANCH_OR_COMMIT` can be found in the [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base).
313
314 5. Build vLLM. For example, vLLM on ROCM 7.0 can be built with the following steps:
315
316 ???+ console "Commands"
317
318 ```bash
319 pip install --upgrade pip
320
321 # Build & install AMD SMI
322 pip install /opt/rocm/share/amd_smi
323
324 # Install dependencies
325 pip install --upgrade numba \
326 scipy \
327 huggingface-hub \
328 setuptools_scm
329 pip install -r requirements/rocm.txt
330
331 # To build for a single architecture (e.g., MI300) for faster installation (recommended):
332 export PYTORCH_ROCM_ARCH="gfx942"
333
334 # To build vLLM for multiple arch MI210/MI250/MI300, use this instead
335 # export PYTORCH_ROCM_ARCH="gfx90a;gfx942"
336
337 python3 setup.py develop
338 ```
339
340 This may take 5-10 minutes. Currently, `pip install .` does not work for ROCm when installing vLLM from source.
341
342 !!! tip
343 - The ROCm version of PyTorch, ideally, should match the ROCm driver version.
344
345 !!! tip
346 - For MI300x (gfx942) users, to achieve optimal performance, please refer to [MI300x tuning guide](https://rocm.docs.amd.com/en/latest/how-to/tuning-guides/mi300x/index.html) for performance optimization and tuning tips on system and workflow level.
347 For vLLM, please refer to [vLLM performance optimization](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/vllm-optimization.html).
348
349 --8<-- [end:build-wheel-from-source]
350 --8<-- [start:pre-built-images]
351
352 vLLM offers official Docker images for deployment.
353 The images can be used to run OpenAI compatible server and are available on Docker Hub as [vllm/vllm-openai-rocm](https://hub.docker.com/r/vllm/vllm-openai-rocm/tags).
354
355 - `vllm/vllm-openai-rocm:latest` — stable release
356 - `vllm/vllm-openai-rocm:nightly` — preview build from the latest development branch, use this if you want the latest features and fixes
357
358 ```bash
359 docker run --rm \
360 --group-add=video \
361 --cap-add=SYS_PTRACE \
362 --security-opt seccomp=unconfined \
363 --device /dev/kfd \
364 --device /dev/dri \
365 -v ~/.cache/huggingface:/root/.cache/huggingface \
366 --env "HF_TOKEN=$HF_TOKEN" \
367 -p 8000:8000 \
368 --ipc=host \
369 vllm/vllm-openai-rocm:<tag> \
370 --model Qwen/Qwen3-0.6B
371 ```
372
373 To use the docker image as base for development, you can launch it in interactive session through overriding the entrypoint.
374
375 ???+ console "Commands"
376 ```bash
377 docker run --rm -it \
378 --group-add=video \
379 --cap-add=SYS_PTRACE \
380 --security-opt seccomp=unconfined \
381 --device /dev/kfd \
382 --device /dev/dri \
383 -v ~/.cache/huggingface:/root/.cache/huggingface \
384 --env "HF_TOKEN=$HF_TOKEN" \
385 --network=host \
386 --ipc=host \
387 --entrypoint /bin/bash \
388 vllm/vllm-openai-rocm:<tag>
389 ```
390
391 #### Use AMD's Docker Images (Deprecated)
392
393 !!! warning "Deprecated"
394 AMD's Docker images (`rocm/vllm` and `rocm/vllm-dev`) are deprecated in favor of the official vLLM Docker images above (`vllm/vllm-openai-rocm`). Please migrate to the official images.
395
396 Prior to January 20th, 2026 when the official docker images became available on [upstream vLLM docker hub](https://hub.docker.com/v2/repositories/vllm/vllm-openai-rocm/tags/), the [AMD Infinity hub for vLLM](https://hub.docker.com/r/rocm/vllm/tags) offered a prebuilt, optimized
397 docker image designed for validating inference performance on the AMD Instinct MI300X™ accelerator.
398 AMD also offered nightly prebuilt docker image from [Docker Hub](https://hub.docker.com/r/rocm/vllm-dev), which has vLLM and all its dependencies installed. The entrypoint of this docker image is `/bin/bash` (different from the vLLM's Official Docker Image).
399
400 !!! tip
401 Please check [LLM inference performance validation on AMD Instinct MI300X](https://rocm.docs.amd.com/en/latest/how-to/performance-validation/mi300x/vllm-benchmark.html)
402 for instructions on how to use this prebuilt docker image.
403
404 --8<-- [end:pre-built-images]
405 --8<-- [start:build-image-from-source]
406
407 You can build and run vLLM from source via the provided [docker/Dockerfile.rocm](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm).
408
409 ??? info "(Optional) Build an image with ROCm software stack"
410
411 Build a docker image from [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base) which setup ROCm software stack needed by the vLLM.
412 **This step is optional as this rocm_base image is usually prebuilt and store at [Docker Hub](https://hub.docker.com/r/rocm/vllm-dev) under tag `rocm/vllm-dev:base` to speed up user experience.**
413 If you choose to build this rocm_base image yourself, the steps are as follows.
414
415 It is important that the user kicks off the docker build using buildkit. Either the user put `DOCKER_BUILDKIT=1` as environment variable when calling docker build command, or the user needs to set up buildkit in the docker daemon configuration `/etc/docker/daemon.json` as follows and restart the daemon:
416
417 ```json
418 {
419 "features": {
420 "buildkit": true
421 }
422 }
423 ```
424
425 To build vllm on ROCm 7.0 for MI200 and MI300 series, you can use the default:
426
427 ```bash
428 DOCKER_BUILDKIT=1 docker build \
429 -f docker/Dockerfile.rocm_base \
430 -t rocm/vllm-dev:base .
431 ```
432
433 First, build a docker image from [docker/Dockerfile.rocm](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm) and launch a docker container from the image.
434 It is important that the user kicks off the docker build using buildkit. Either the user put `DOCKER_BUILDKIT=1` as environment variable when calling docker build command, or the user needs to set up buildkit in the docker daemon configuration /etc/docker/daemon.json as follows and restart the daemon:
435
436 ```json
437 {
438 "features": {
439 "buildkit": true
440 }
441 }
442 ```
443
444 [docker/Dockerfile.rocm](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm) uses ROCm 7.0 by default, but also supports ROCm 5.7, 6.0, 6.1, 6.2, 6.3, and 6.4, in older vLLM branches.
445 It provides flexibility to customize the build of docker image using the following arguments:
446
447 - `BASE_IMAGE`: specifies the base image used when running `docker build`. The default value `rocm/vllm-dev:base` is an image published and maintained by AMD. It is being built using [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base)
448 - `ARG_PYTORCH_ROCM_ARCH`: Allows to override the gfx architecture values from the base docker image
449
450 Their values can be passed in when running `docker build` with `--build-arg` options.
451
452 To build vllm on ROCm 7.0 for MI200 and MI300 series, you can use the default (which build a docker image with `vllm serve` as entrypoint):
453
454 ```bash
455 DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.rocm -t vllm/vllm-openai-rocm .
456 ```
457
458 To run vLLM with the custom-built Docker image:
459
460 ```bash
461 docker run --rm \
462 --group-add=video \
463 --cap-add=SYS_PTRACE \
464 --security-opt seccomp=unconfined \
465 --device /dev/kfd \
466 --device /dev/dri \
467 -v ~/.cache/huggingface:/root/.cache/huggingface \
468 --env "HF_TOKEN=$HF_TOKEN" \
469 -p 8000:8000 \
470 --ipc=host \
471 vllm/vllm-openai-rocm <args...>
472 ```
473
474 The argument `vllm/vllm-openai-rocm` specifies the image to run, and should be replaced with the name of the custom-built image (the `-t` tag from the build command).
475
476 To use the docker image as base for development, you can launch it in interactive session through overriding the entrypoint.
477
478 ???+ console "Commands"
479 ```bash
480 docker run --rm -it \
481 --group-add=video \
482 --cap-add=SYS_PTRACE \
483 --security-opt seccomp=unconfined \
484 --device /dev/kfd \
485 --device /dev/dri \
486 -v ~/.cache/huggingface:/root/.cache/huggingface \
487 --env "HF_TOKEN=$HF_TOKEN" \
488 --network=host \
489 --ipc=host \
490 --entrypoint bash \
491 vllm/vllm-openai-rocm
492 ```
493
494 --8<-- [end:build-image-from-source]
495 --8<-- [start:supported-features]
496
497 See [Feature x Hardware](../../features/README.md#feature-x-hardware) compatibility matrix for feature support information.
498
499 --8<-- [end:supported-features]