A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 01:45 UTC
[ROCm][Hardware][AMD][Doc] Documentation update for ROCm (#4376)
Co-authored-by: WoosukKwon <woosuk.kwon@berkeley.edu>
Hongxia Yang committed
Apr 25, 2024 at 21:12 UTC
cf29b7eda47d5a8c5fe6a7a53490271da8520563
1 file changed
+65
-100
docs/source/getting_started/amd-installation.rst
+65
-100
@@ -3,9 +3,7 @@
3
Installation with ROCm
4
======================
5
6
-vLLM 0.2.4 onwards supports model inferencing and serving on AMD GPUs with ROCm.
7
-At the moment AWQ quantization is not supported in ROCm, but SqueezeLLM quantization has been ported.
8
-Data types currently supported in ROCm are FP16 and BF16.
6
+vLLM supports AMD GPUs with ROCm 5.7 and 6.0.
7
8
Requirements
9
------------
@@ -13,114 +11,57 @@ Requirements
11
* OS: Linux
12
* Python: 3.8 -- 3.11
13
* GPU: MI200s (gfx90a), MI300 (gfx942), Radeon RX 7900 series (gfx1100)
16
-* Pytorch 2.0.1/2.1.1/2.2
17
-* ROCm 5.7 (Verified on python 3.10) or ROCm 6.0 (Verified on python 3.9)
14
+* ROCm 6.0 and ROCm 5.7
15
16
Installation options:
17
21
-#. :ref:`(Recommended) Quick start with vLLM pre-installed in Docker Image <quick_start_docker_rocm>`
22
-#. :ref:`Build from source <build_from_source_rocm>`
18
#. :ref:`Build from source with docker <build_from_source_docker_rocm>`
19
+#. :ref:`Build from source <build_from_source_rocm>`
20
25
-.. _quick_start_docker_rocm:
26
-
27
-(Recommended) Option 1: Quick start with vLLM pre-installed in Docker Image
28
----------------------------------------------------------------------------
29
-
30
-This option is for ROCm 5.7 only:
31
-
32
-.. code-block:: console
33
-
34
- $ docker pull embeddedllminfo/vllm-rocm:vllm-v0.2.4
35
- $ docker run -it \
36
- --network=host \
37
- --group-add=video \
38
- --ipc=host \
39
- --cap-add=SYS_PTRACE \
40
- --security-opt seccomp=unconfined \
41
- --device /dev/kfd \
42
- --device /dev/dri \
43
- -v <path/to/model>:/app/model \
44
- embeddedllminfo/vllm-rocm \
45
- bash
46
-
47
-
48
-.. _build_from_source_rocm:
49
-
50
-Option 2: Build from source
51
----------------------------
52
-
53
-You can build and install vLLM from source:
54
-
55
-Below instruction is for ROCm 5.7 only.
56
-At the time of this documentation update, PyTorch on ROCm 6.0 wheel is not yet available on the PyTorch website.
57
-
58
-0. Install prerequisites (skip if you are already in an environment/docker with the following installed):
59
-
60
-- `ROCm <https://rocm.docs.amd.com/en/latest/deploy/linux/index.html>`_
61
-- `Pytorch <https://pytorch.org/>`_
62
-
63
- .. code-block:: console
64
-
65
- $ pip install torch==2.2.0.dev20231206+rocm5.7 --index-url https://download.pytorch.org/whl/nightly/rocm5.7 # tested version
66
-
67
-
68
-1. Install `flash attention for ROCm <https://github.com/ROCmSoftwarePlatform/flash-attention/tree/flash_attention_for_rocm>`_
69
-
70
- Install ROCm's flash attention (v2.0.4) following the instructions from `ROCmSoftwarePlatform/flash-attention <https://github.com/ROCmSoftwarePlatform/flash-attention/tree/flash_attention_for_rocm#amd-gpurocm-support>`_
71
-
72
-.. note::
73
- - If you are using rocm5.7 with pytorch 2.1.0 onwards, you don't need to apply the `hipify_python.patch`. You can build the ROCm flash attention directly.
74
- - If you fail to install `ROCmSoftwarePlatform/flash-attention`, try cloning from the commit `6fd2f8e572805681cd67ef8596c7e2ce521ed3c6`.
75
- - ROCm's Flash-attention-2 (v2.0.4) does not support sliding windows attention.
76
- - You might need to downgrade the "ninja" version to 1.10 it is not used when compiling flash-attention-2 (e.g. `pip install ninja==1.10.2.4`)
77
-
78
-2. Setup `xformers==0.0.23` without dependencies, and apply patches to adapt for ROCm flash attention
21
+.. _build_from_source_docker_rocm:
22
80
- .. code-block:: console
23
+Option 1: Build from source with docker (recommended)
24
+-----------------------------------------------------
25
82
- $ pip install xformers==0.0.23 --no-deps
83
- $ bash patch_xformers.rocm.sh
26
+You can build and install vLLM from source.
27
85
-3. Build vLLM.
28
+First, build a docker image from `Dockerfile.rocm <https://github.com/vllm-project/vllm/blob/main/Dockerfile.rocm>`_ and launch a docker container from the image.
29
87
- .. code-block:: console
30
+`Dockerfile.rocm <https://github.com/vllm-project/vllm/blob/main/Dockerfile.rocm>`_ uses ROCm 6.0 by default, but also supports ROCm 5.7.
31
+It provides flexibility to customize the build of docker image using the following arguments:
32
89
- $ cd vllm
90
- $ pip install -U -r requirements-rocm.txt
91
- $ python setup.py install # This may take 5-10 minutes. Currently, `pip install .`` does not work for ROCm installation
33
+* `BASE_IMAGE`: specifies the base image used when running ``docker build``, specifically the PyTorch on ROCm base image. We have tested ROCm 5.7 and ROCm 6.0. The default is `rocm/pytorch:rocm6.0_ubuntu20.04_py3.9_pytorch_2.1.1`
34
+* `BUILD_FA`: specifies whether to build CK flash-attention. The default is 1. For `Radeon RX 7900 series (gfx1100) <https://rocm.docs.amd.com/projects/radeon/en/latest/index.html>`_, this should be set to 0 before flash-attention supports this target.
35
+* `FX_GFX_ARCHS`: specifies the GFX architecture that is used to build CK flash-attention, for example, `gfx90a;gfx942` for MI200 and MI300. The default is `gfx90a;gfx942`
36
+* `FA_BRANCH`: specifies the branch used to build the CK flash-attention in `ROCm's flash-attention repo <https://github.com/ROCmSoftwarePlatform/flash-attention>`_. The default is `ae7928c`
37
+* `BUILD_TRITON`: specifies whether to build triton flash-attention. The default value is 1.
38
39
+Their values can be passed in when running ``docker build`` with ``--build-arg`` options.
40
94
-.. _build_from_source_docker_rocm:
41
96
-Option 3: Build from source with docker
97
------------------------------------------------------
42
+To build vllm on ROCm 6.0 for MI200 and MI300 series, you can use the default:
43
99
-You can build and install vLLM from source:
44
+.. code-block:: console
45
101
-Build a docker image from `Dockerfile.rocm`, and launch a docker container.
46
+ $ docker build -f Dockerfile.rocm -t vllm-rocm .
47
103
-The `Dockerfile.rocm` is designed to support both ROCm 5.7 and ROCm 6.0 and later versions. It provides flexibility to customize the build of docker image using the following arguments:
48
+To build vllm on ROCm 6.0 for Radeon RX7900 series (gfx1100), you should specify ``BUILD_FA`` as below:
49
105
-* `BASE_IMAGE`: specifies the base image used when running ``docker build``, specifically the PyTorch on ROCm base image. We have tested ROCm 5.7 and ROCm 6.0. The default is `rocm/pytorch:rocm6.0_ubuntu20.04_py3.9_pytorch_2.1.1`
106
-* `FX_GFX_ARCHS`: specifies the GFX architecture that is used to build flash-attention, for example, `gfx90a;gfx942` for MI200 and MI300. The default is `gfx90a;gfx942`
107
-* `FA_BRANCH`: specifies the branch used to build the flash-attention in `ROCmSoftwarePlatform's flash-attention repo <https://github.com/ROCmSoftwarePlatform/flash-attention>`_. The default is `3d2b6f5`
108
-* `BUILD_FA`: specifies whether to build flash-attention. For `Radeon RX 7900 series (gfx1100) <https://rocm.docs.amd.com/projects/radeon/en/latest/index.html>`_, this should be set to 0 before flash-attention supports this target.
50
+.. code-block:: console
51
110
-Their values can be passed in when running ``docker build`` with ``--build-arg`` options.
52
+ $ docker build --build-arg BUILD_FA="0" -f Dockerfile.rocm -t vllm-rocm .
53
112
-For example, to build docker image for vllm on ROCm 5.7, you can run:
54
+To build docker image for vllm on ROCm 5.7, you can specify ``BASE_IMAGE`` as below:
55
56
.. code-block:: console
57
58
$ docker build --build-arg BASE_IMAGE="rocm/pytorch:rocm5.7_ubuntu22.04_py3.10_pytorch_2.0.1" \
59
-f Dockerfile.rocm -t vllm-rocm .
60
119
-To build vllm on ROCm 6.0, you can use the default:
61
+To run the above docker image ``vllm-rocm``, use the below command:
62
63
.. code-block:: console
64
123
- $ docker build -f Dockerfile.rocm -t vllm-rocm .
65
$ docker run -it \
66
--network=host \
67
--group-add=video \
@@ -133,7 +74,13 @@ To build vllm on ROCm 6.0, you can use the default:
74
vllm-rocm \
75
bash
76
136
-Alternatively, if you plan to install vLLM-ROCm on a local machine or start from a fresh docker image (e.g. rocm/pytorch), you can follow the steps below:
77
+Where the `<path/to/model>` is the location where the model is stored, for example, the weights for llama2 or llama3 models.
78
+
79
+
80
+.. _build_from_source_rocm:
81
+
82
+Option 2: Build from source
83
+---------------------------
84
85
0. Install prerequisites (skip if you are already in an environment/docker with the following installed):
86
@@ -141,32 +88,50 @@ Alternatively, if you plan to install vLLM-ROCm on a local machine or start from
88
- `Pytorch <https://pytorch.org/>`_
89
- `hipBLAS <https://rocm.docs.amd.com/projects/hipBLAS/en/latest/install.html>`_
90
144
-1. Install `flash attention for ROCm <https://github.com/ROCmSoftwarePlatform/flash-attention/tree/flash_attention_for_rocm>`_
91
+For installing PyTorch, you can start from a fresh docker image, e.g, `rocm6.0.2_ubuntu22.04_py3.10_pytorch_2.1.2`, `rocm/pytorch:rocm6.0_ubuntu20.04_py3.9_pytorch_2.1.1`, `rocm/pytorch-nightly`.
92
146
- Install ROCm's flash attention (v2.0.4) following the instructions from `ROCmSoftwarePlatform/flash-attention <https://github.com/ROCmSoftwarePlatform/flash-attention/tree/flash_attention_for_rocm#amd-gpurocm-support>`_
93
+Alternatively, you can install pytorch using pytorch wheels. You can check Pytorch installation guild in Pytorch `Getting Started <https://pytorch.org/get-started/locally/>`_
94
+
95
+For rocm6.0:
96
+
97
+.. code-block:: console
98
+
99
+ $ pip3 install torch --index-url https://download.pytorch.org/whl/rocm6.0
100
+
101
+
102
+For rocm5.7:
103
+
104
+.. code-block:: console
105
+
106
+ $ pip install torch --index-url https://download.pytorch.org/whl/rocm5.7
107
+
108
+
109
+1. Install `Triton flash attention for ROCm <https://github.com/ROCm/triton>`_
110
+
111
+Install ROCm's Triton flash attention (the default triton-mlir branch) following the instructions from `ROCm/triton <https://github.com/ROCm/triton/blob/triton-mlir/README.md>`_
112
+
113
+2. Optionally, if you choose to use CK flash attention, you can install `flash attention for ROCm <https://github.com/ROCm/flash-attention/tree/flash_attention_for_rocm>`_
114
+
115
+Install ROCm's flash attention (v2.0.4) following the instructions from `ROCm/flash-attention <https://github.com/ROCm/flash-attention/tree/flash_attention_for_rocm#amd-gpurocm-support>`_
116
117
.. note::
118
- If you are using rocm5.7 with pytorch 2.1.0 onwards, you don't need to apply the `hipify_python.patch`. You can build the ROCm flash attention directly.
150
- - If you fail to install `ROCmSoftwarePlatform/flash-attention`, try cloning from the commit `6fd2f8e572805681cd67ef8596c7e2ce521ed3c6`.
119
+ - If you fail to install `ROCm/flash-attention`, try cloning from the commit `6fd2f8e572805681cd67ef8596c7e2ce521ed3c6`.
120
- ROCm's Flash-attention-2 (v2.0.4) does not support sliding windows attention.
121
- You might need to downgrade the "ninja" version to 1.10 it is not used when compiling flash-attention-2 (e.g. `pip install ninja==1.10.2.4`)
122
154
-2. Setup `xformers==0.0.23` without dependencies, and apply patches to adapt for ROCm flash attention
155
-
156
- .. code-block:: console
157
-
158
- $ pip install xformers==0.0.23 --no-deps
159
- $ bash patch_xformers.rocm.sh
160
-
123
3. Build vLLM.
124
163
- .. code-block:: console
125
+.. code-block:: console
126
165
- $ cd vllm
166
- $ pip install -U -r requirements-rocm.txt
167
- $ python setup.py install # This may take 5-10 minutes.
127
+ $ cd vllm
128
+ $ pip install -U -r requirements-rocm.txt
129
+ $ python setup.py install # This may take 5-10 minutes. Currently, `pip install .`` does not work for ROCm installation
130
169
-.. note::
131
171
- - You may need to turn on the ``--enforce-eager`` flag if you experience process hang when running the `benchmark_thoughput.py` script to test your installation.
132
+.. tip::
133
134
+ - You may need to turn on the ``--enforce-eager`` flag if you experience process hang when running the `benchmark_thoughput.py` script to test your installation.
135
+ - Triton flash attention is used by default. For benchmarking purposes, it is recommended to run a warm up step before collecting perf numbers.
136
+ - To use CK flash-attention, please use this flag ``export VLLM_USE_FLASH_ATTN_TRITON=0`` to turn off triton flash attention.
137
+ - The ROCm version of pytorch, ideally, should match the ROCm driver version.