A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 9, 23:00 UTC
[Doc] [ROCm] Update ROCm getting started doc (#32580)
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com> Co-authored-by: Hongxia Yang <hongxia.yang@amd.com> Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
TJian committed
Jan 21, 2026 at 01:20 UTC
c025263ddd1c35532151fa1334f64c0aa7acc4ba
2 files changed
+96
-42
docs/getting_started/installation/gpu.rocm.inc.md
+80
-15
@@ -1,9 +1,6 @@
1
# --8<-- [start:installation]
2
3
-vLLM supports AMD GPUs with ROCm 6.3 or above, and torch 2.8.0 and above.
4
-
5
-!!! tip
6
- [Docker](#set-up-using-docker) is the recommended way to use vLLM on ROCm.
3
+vLLM supports AMD GPUs with ROCm 6.3 or above. Pre-built wheels are available for ROCm 7.0.
4
5
# --8<-- [end:installation]
6
# --8<-- [start:requirements]
@@ -16,12 +13,36 @@ vLLM supports AMD GPUs with ROCm 6.3 or above, and torch 2.8.0 and above.
13
# --8<-- [end:requirements]
14
# --8<-- [start:set-up-using-python]
15
19
-There is no extra information on creating a new Python environment for this device.
16
+The vLLM wheel bundles PyTorch and all required dependencies, and you should use the included PyTorch for compatibility. Because vLLM compiles many ROCm kernels to ensure a validated, high‑performance stack, the resulting binaries may not be compatible with other ROCm or PyTorch builds.
17
+If you need a different ROCm version or want to use an existing PyTorch installation, you’ll need to build vLLM from source. See [below](#build-wheel-from-source) for more details.
18
19
# --8<-- [end:set-up-using-python]
20
# --8<-- [start:pre-built-wheels]
21
24
-Currently, there are no pre-built ROCm wheels.
22
+To install the latest version of vLLM for Python 3.12, ROCm 7.0 and `glibc >= 2.35`.
23
+
24
+```bash
25
+uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
26
+```
27
+
28
+!!! tip
29
+ You can find out about which ROCm version the latest vLLM supports by checking the index in extra-index-url [https://wheels.vllm.ai/rocm/](https://wheels.vllm.ai/rocm/) .
30
+
31
+To install a specific version and ROCm variant of vLLM wheel.
32
+
33
+```bash
34
+uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.14.0/rocm700
35
+```
36
+
37
+!!! warning "Caveats for using `pip`"
38
+
39
+ We recommend leveraging `uv` to install vLLM wheel. Using `pip` to install from custom indices is cumbersome, because `pip` combines packages from `--extra-index-url` and the default index, choosing only the latest version, which makes it difficult to install wheel from custom index if exact versions of all packages are specified exactly. In contrast, `uv` gives the extra index [higher priority than the default index](https://docs.astral.sh/uv/pip/compatibility/#packages-that-exist-on-multiple-indexes).
40
+
41
+ If you insist on using `pip`, you have to specify the exact vLLM version and full URL of the wheel path `https://wheels.vllm.ai/rocm/<version>/<rocm-variant>` (which can be obtained from the web page).
42
+
43
+ ```bash
44
+ pip install vllm==0.14.0+rocm700 --extra-index-url https://wheels.vllm.ai/rocm/0.14.0/rocm700
45
+ ```
46
47
# --8<-- [end:pre-built-wheels]
48
# --8<-- [start:build-wheel-from-source]
@@ -84,7 +105,7 @@ Currently, there are no pre-built ROCm wheels.
105
- The validated `$FA_BRANCH` can be found in the [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base).
106
107
87
-3. If you choose to build AITER yourself to use a certain branch or commit, you can build AITER using the following steps:
108
+3. Optionally, if you choose to build AITER yourself to use a certain branch or commit, you can build AITER using the following steps:
109
110
```bash
111
python3 -m pip uninstall -y aiter
@@ -100,14 +121,14 @@ Currently, there are no pre-built ROCm wheels.
121
- The validated `$AITER_BRANCH_OR_COMMIT` can be found in the [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base).
122
123
103
-4. If you want to use MORI for EP or PD disaggregation, you can install [MORI](https://github.com/ROCm/mori) using the following steps:
124
+4. Optionally, if you want to use MORI for EP or PD disaggregation, you can install [MORI](https://github.com/ROCm/mori) using the following steps:
125
126
```bash
127
git clone https://github.com/ROCm/mori.git
128
cd mori
129
git checkout $MORI_BRANCH_OR_COMMIT
130
git submodule sync; git submodule update --init --recursive
110
- MORI_GPU_ARCHS="gfx942;gfx950" python3 install .
131
+ MORI_GPU_ARCHS="gfx942;gfx950" python3 setup.py install
132
```
133
134
!!! note
@@ -141,7 +162,7 @@ Currently, there are no pre-built ROCm wheels.
162
python3 setup.py develop
163
```
164
144
- This may take 5-10 minutes. Currently, `pip install .` does not work for ROCm installation.
165
+ This may take 5-10 minutes. Currently, `pip install .` does not work for ROCm when installing vLLM from source.
166
167
!!! tip
168
- The ROCm version of PyTorch, ideally, should match the ROCm driver version.
@@ -153,9 +174,51 @@ Currently, there are no pre-built ROCm wheels.
174
# --8<-- [end:build-wheel-from-source]
175
# --8<-- [start:pre-built-images]
176
177
+#### Use vLLM's Official Docker Image
178
+
179
+vLLM offers an official Docker image for deployment.
180
+The image can be used to run OpenAI compatible server and is available on Docker Hub as [vllm/vllm-openai-rocm](https://hub.docker.com/r/vllm/vllm-openai-rocm/tags).
181
+
182
+???+ console "Commands"
183
+ ```bash
184
+ docker run --rm \
185
+ --group-add=video \
186
+ --cap-add=SYS_PTRACE \
187
+ --security-opt seccomp=unconfined \
188
+ --device /dev/kfd \
189
+ --device /dev/dri \
190
+ -v ~/.cache/huggingface:/root/.cache/huggingface \
191
+ --env "HF_TOKEN=$HF_TOKEN" \
192
+ -p 8000:8000 \
193
+ --ipc=host \
194
+ vllm/vllm-openai-rocm:latest \
195
+ --model Qwen/Qwen3-0.6B
196
+ ```
197
+
198
+To use the docker image as base for development, you can launch it in interactive session through overriding the entrypoint.
199
+
200
+???+ console "Commands"
201
+ ```bash
202
+ docker run --rm -it \
203
+ --group-add=video \
204
+ --cap-add=SYS_PTRACE \
205
+ --security-opt seccomp=unconfined \
206
+ --device /dev/kfd \
207
+ --device /dev/dri \
208
+ -v ~/.cache/huggingface:/root/.cache/huggingface \
209
+ --env "HF_TOKEN=$HF_TOKEN" \
210
+ -p 8000:8000 \
211
+ --ipc=host \
212
+ --entrypoint bash \
213
+ vllm/vllm-openai-rocm:latest
214
+ ```
215
+
216
+
217
+#### Use AMD's Docker Images
218
+
219
The [AMD Infinity hub for vLLM](https://hub.docker.com/r/rocm/vllm/tags) offers a prebuilt, optimized
220
docker image designed for validating inference performance on the AMD Instinct™ MI300X accelerator.
158
-AMD also offers nightly prebuilt docker image from [Docker Hub](https://hub.docker.com/r/rocm/vllm-dev), which has vLLM and all its dependencies installed.
221
+AMD also offers nightly prebuilt docker image from [Docker Hub](https://hub.docker.com/r/rocm/vllm-dev), which has vLLM and all its dependencies installed. The entrypoint of this docker image is `/bin/bash` (different from the vLLM's Official Docker Image).
222
223
???+ console "Commands"
224
```bash
@@ -188,7 +251,7 @@ Building the Docker image from source is the recommended way to use vLLM with RO
251
**This step is optional as this rocm_base image is usually prebuilt and store at [Docker Hub](https://hub.docker.com/r/rocm/vllm-dev) under tag `rocm/vllm-dev:base` to speed up user experience.**
252
If you choose to build this rocm_base image yourself, the steps are as follows.
253
191
- It is important that the user kicks off the docker build using buildkit. Either the user put DOCKER_BUILDKIT=1 as environment variable when calling docker build command, or the user needs to set up buildkit in the docker daemon configuration /etc/docker/daemon.json as follows and restart the daemon:
254
+ It is important that the user kicks off the docker build using buildkit. Either the user put `DOCKER_BUILDKIT=1` as environment variable when calling docker build command, or the user needs to set up buildkit in the docker daemon configuration `/etc/docker/daemon.json` as follows and restart the daemon:
255
256
```json
257
{
@@ -211,7 +274,7 @@ Building the Docker image from source is the recommended way to use vLLM with RO
274
First, build a docker image from [docker/Dockerfile.rocm](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm) and launch a docker container from the image.
275
It is important that the user kicks off the docker build using buildkit. Either the user put `DOCKER_BUILDKIT=1` as environment variable when calling docker build command, or the user needs to set up buildkit in the docker daemon configuration /etc/docker/daemon.json as follows and restart the daemon:
276
214
-```bash
277
+```json
278
{
279
"features": {
280
"buildkit": true
@@ -227,7 +290,7 @@ It provides flexibility to customize the build of docker image using the followi
290
291
Their values can be passed in when running `docker build` with `--build-arg` options.
292
230
-To build vllm on ROCm 7.0 for MI200 and MI300 series, you can use the default:
293
+To build vllm on ROCm 7.0 for MI200 and MI300 series, you can use the default (which build a docker image with `vllm serve` as entrypoint):
294
295
???+ console "Commands"
296
```bash
@@ -236,6 +299,7 @@ To build vllm on ROCm 7.0 for MI200 and MI300 series, you can use the default:
299
300
To run the above docker image `vllm-rocm`, use the below command:
301
302
+
303
???+ console "Commands"
304
```bash
305
docker run -it \
@@ -247,7 +311,8 @@ To run the above docker image `vllm-rocm`, use the below command:
311
--device /dev/kfd \
312
--device /dev/dri \
313
-v <path/to/model>:/app/model \
250
- vllm-rocm
314
+ vllm-rocm \
315
+ --model Qwen/Qwen3-0.6B
316
```
317
318
Where the `<path/to/model>` is the location where the model is stored, for example, the weights for llama2 or llama3 models.
docs/getting_started/quickstart.md
+16
-27
@@ -43,25 +43,21 @@ This guide will help you quickly get started with vLLM to perform:
43
44
=== "AMD ROCm"
45
46
- Use a pre-built docker image from Docker Hub. The public stable image is [rocm/vllm:latest](https://hub.docker.com/r/rocm/vllm). There is also a development image at [rocm/vllm-dev](https://hub.docker.com/r/rocm/vllm-dev).
47
-
48
- The `-v` flag in the `docker run` command below mounts a local directory into the container. Replace `<path/to/your/models>` with the path on your host machine to the directory containing your models. The models will then be accessible inside the container at `/app/models`.
49
-
50
- ???+ console "Commands"
51
- ```bash
52
- docker pull rocm/vllm-dev:nightly # to get the latest image
53
- docker run -it --rm \
54
- --network=host \
55
- --group-add=video \
56
- --ipc=host \
57
- --cap-add=SYS_PTRACE \
58
- --security-opt seccomp=unconfined \
59
- --device /dev/kfd \
60
- --device /dev/dri \
61
- -v <path/to/your/models>:/app/models \
62
- -e HF_HOME="/app/models" \
63
- rocm/vllm-dev:nightly
64
- ```
46
+ If you are using AMD GPUs, you can install vLLM using `uv`.
47
+
48
+ It's recommended to use [uv](https://docs.astral.sh/uv/), as it gives the extra index [higher priority than the default index](https://docs.astral.sh/uv/pip/compatibility/#packages-that-exist-on-multiple-indexes). `uv` is also a very fast Python environment manager, to create and manage Python environments. Please follow the [documentation](https://docs.astral.sh/uv/#getting-started) to install `uv`. After installing `uv`, you can create a new Python environment and install vLLM using the following commands:
49
+
50
+ ```bash
51
+ uv venv --python 3.12 --seed
52
+ source .venv/bin/activate
53
+ uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
54
+ ```
55
+
56
+ !!! note
57
+ It currently supports Python 3.12, ROCm 7.0 and `glibc >= 2.35`.
58
+
59
+ !!! note
60
+ Note that, previously, docker images were published using AMD's docker release pipeline and were located `rocm/vlm-dev`. This is being deprecated by using vLLM's docker release pipeline.
61
62
=== "Google TPU"
63
@@ -294,14 +290,7 @@ python script.py --attention-backend FLASHINFER
290
Some of the available backend options include:
291
292
- On NVIDIA CUDA: `FLASH_ATTN` or `FLASHINFER`.
297
-- On AMD ROCm: `TRITON_ATTN`, `ROCM_ATTN`, `ROCM_AITER_FA` or `ROCM_AITER_UNIFIED_ATTN`.
298
-
299
-For AMD ROCm, you can further control the specific Attention implementation using the following options:
300
-
301
-- Triton Unified Attention: Set the environment variables `VLLM_ROCM_USE_AITER=0 VLLM_ROCM_USE_AITER_MHA=0` and pass `--attention-config.use_prefill_decode_attention=false` as a CLI argument.
302
-- AITER Unified Attention: Set the environment variables `VLLM_ROCM_USE_AITER=1 VLLM_USE_AITER_UNIFIED_ATTENTION=1 VLLM_ROCM_USE_AITER_MHA=0` and pass `--attention-config.use_prefill_decode_attention=false` as a CLI argument.
303
-- Triton Prefill-Decode Attention: Set the environment variables `VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MHA=0` and pass `--attention-config.use_prefill_decode_attention=true` as a CLI argument.
304
-- AITER Multi-head Attention: Set the environment variables `VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MHA=1` and pass `--attention-config.use_prefill_decode_attention=false` as a CLI argument.
293
+- On AMD ROCm: `TRITON_ATTN`, `ROCM_ATTN`, `ROCM_AITER_FA`, `ROCM_AITER_UNIFIED_ATTN`, `TRITON_MLA`, `ROCM_AITER_MLA` or `ROCM_AITER_TRITON_MLA`.
294
295
!!! warning
296
There are no pre-built vllm wheels containing Flash Infer, so you must install it in your environment first. Refer to the [Flash Infer official docs](https://docs.flashinfer.ai/) or see [docker/Dockerfile](../../docker/Dockerfile) for instructions on how to install it.