@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / cf29b7eda4
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 01:45 UTC

[ROCm][Hardware][AMD][Doc] Documentation update for ROCm (#4376)

Co-authored-by: WoosukKwon <woosuk.kwon@berkeley.edu>

Hongxia Yang committed Apr 25, 2024 at 21:12 UTC cf29b7eda47d5a8c5fe6a7a53490271da8520563
1 file changed +65 -100
docs/source/getting_started/amd-installation.rst
+65 -100
@@ -3,9 +3,7 @@
3 Installation with ROCm
4 ======================
5
6 -vLLM 0.2.4 onwards supports model inferencing and serving on AMD GPUs with ROCm.
7 -At the moment AWQ quantization is not supported in ROCm, but SqueezeLLM quantization has been ported.
8 -Data types currently supported in ROCm are FP16 and BF16.
6 +vLLM supports AMD GPUs with ROCm 5.7 and 6.0.
7
8 Requirements
9 ------------
@@ -13,114 +11,57 @@ Requirements
11 * OS: Linux
12 * Python: 3.8 -- 3.11
13 * GPU: MI200s (gfx90a), MI300 (gfx942), Radeon RX 7900 series (gfx1100)
16 -* Pytorch 2.0.1/2.1.1/2.2
17 -* ROCm 5.7 (Verified on python 3.10) or ROCm 6.0 (Verified on python 3.9)
14 +* ROCm 6.0 and ROCm 5.7
15
16 Installation options:
17
21 -#. :ref:`(Recommended) Quick start with vLLM pre-installed in Docker Image <quick_start_docker_rocm>`
22 -#. :ref:`Build from source <build_from_source_rocm>`
18 #. :ref:`Build from source with docker <build_from_source_docker_rocm>`
19 +#. :ref:`Build from source <build_from_source_rocm>`
20
25 -.. _quick_start_docker_rocm:
26 -
27 -(Recommended) Option 1: Quick start with vLLM pre-installed in Docker Image
28 ----------------------------------------------------------------------------
29 -
30 -This option is for ROCm 5.7 only:
31 -
32 -.. code-block:: console
33 -
34 - $ docker pull embeddedllminfo/vllm-rocm:vllm-v0.2.4
35 - $ docker run -it \
36 - --network=host \
37 - --group-add=video \
38 - --ipc=host \
39 - --cap-add=SYS_PTRACE \
40 - --security-opt seccomp=unconfined \
41 - --device /dev/kfd \
42 - --device /dev/dri \
43 - -v <path/to/model>:/app/model \
44 - embeddedllminfo/vllm-rocm \
45 - bash
46 -
47 -
48 -.. _build_from_source_rocm:
49 -
50 -Option 2: Build from source
51 ----------------------------
52 -
53 -You can build and install vLLM from source:
54 -
55 -Below instruction is for ROCm 5.7 only.
56 -At the time of this documentation update, PyTorch on ROCm 6.0 wheel is not yet available on the PyTorch website.
57 -
58 -0. Install prerequisites (skip if you are already in an environment/docker with the following installed):
59 -
60 -- `ROCm <https://rocm.docs.amd.com/en/latest/deploy/linux/index.html>`_
61 -- `Pytorch <https://pytorch.org/>`_
62 -
63 - .. code-block:: console
64 -
65 - $ pip install torch==2.2.0.dev20231206+rocm5.7 --index-url https://download.pytorch.org/whl/nightly/rocm5.7 # tested version
66 -
67 -
68 -1. Install `flash attention for ROCm <https://github.com/ROCmSoftwarePlatform/flash-attention/tree/flash_attention_for_rocm>`_
69 -
70 - Install ROCm's flash attention (v2.0.4) following the instructions from `ROCmSoftwarePlatform/flash-attention <https://github.com/ROCmSoftwarePlatform/flash-attention/tree/flash_attention_for_rocm#amd-gpurocm-support>`_
71 -
72 -.. note::
73 - - If you are using rocm5.7 with pytorch 2.1.0 onwards, you don't need to apply the `hipify_python.patch`. You can build the ROCm flash attention directly.
74 - - If you fail to install `ROCmSoftwarePlatform/flash-attention`, try cloning from the commit `6fd2f8e572805681cd67ef8596c7e2ce521ed3c6`.
75 - - ROCm's Flash-attention-2 (v2.0.4) does not support sliding windows attention.
76 - - You might need to downgrade the "ninja" version to 1.10 it is not used when compiling flash-attention-2 (e.g. `pip install ninja==1.10.2.4`)
77 -
78 -2. Setup `xformers==0.0.23` without dependencies, and apply patches to adapt for ROCm flash attention
21 +.. _build_from_source_docker_rocm:
22
80 - .. code-block:: console
23 +Option 1: Build from source with docker (recommended)
24 +-----------------------------------------------------
25
82 - $ pip install xformers==0.0.23 --no-deps
83 - $ bash patch_xformers.rocm.sh
26 +You can build and install vLLM from source.
27
85 -3. Build vLLM.
28 +First, build a docker image from `Dockerfile.rocm <https://github.com/vllm-project/vllm/blob/main/Dockerfile.rocm>`_ and launch a docker container from the image.
29
87 - .. code-block:: console
30 +`Dockerfile.rocm <https://github.com/vllm-project/vllm/blob/main/Dockerfile.rocm>`_ uses ROCm 6.0 by default, but also supports ROCm 5.7.
31 +It provides flexibility to customize the build of docker image using the following arguments:
32
89 - $ cd vllm
90 - $ pip install -U -r requirements-rocm.txt
91 - $ python setup.py install # This may take 5-10 minutes. Currently, `pip install .`` does not work for ROCm installation
33 +* `BASE_IMAGE`: specifies the base image used when running ``docker build``, specifically the PyTorch on ROCm base image. We have tested ROCm 5.7 and ROCm 6.0. The default is `rocm/pytorch:rocm6.0_ubuntu20.04_py3.9_pytorch_2.1.1`
34 +* `BUILD_FA`: specifies whether to build CK flash-attention. The default is 1. For `Radeon RX 7900 series (gfx1100) <https://rocm.docs.amd.com/projects/radeon/en/latest/index.html>`_, this should be set to 0 before flash-attention supports this target.
35 +* `FX_GFX_ARCHS`: specifies the GFX architecture that is used to build CK flash-attention, for example, `gfx90a;gfx942` for MI200 and MI300. The default is `gfx90a;gfx942`
36 +* `FA_BRANCH`: specifies the branch used to build the CK flash-attention in `ROCm's flash-attention repo <https://github.com/ROCmSoftwarePlatform/flash-attention>`_. The default is `ae7928c`
37 +* `BUILD_TRITON`: specifies whether to build triton flash-attention. The default value is 1.
38
39 +Their values can be passed in when running ``docker build`` with ``--build-arg`` options.
40
94 -.. _build_from_source_docker_rocm:
41
96 -Option 3: Build from source with docker
97 ------------------------------------------------------
42 +To build vllm on ROCm 6.0 for MI200 and MI300 series, you can use the default:
43
99 -You can build and install vLLM from source:
44 +.. code-block:: console
45
101 -Build a docker image from `Dockerfile.rocm`, and launch a docker container.
46 + $ docker build -f Dockerfile.rocm -t vllm-rocm .
47
103 -The `Dockerfile.rocm` is designed to support both ROCm 5.7 and ROCm 6.0 and later versions. It provides flexibility to customize the build of docker image using the following arguments:
48 +To build vllm on ROCm 6.0 for Radeon RX7900 series (gfx1100), you should specify ``BUILD_FA`` as below:
49
105 -* `BASE_IMAGE`: specifies the base image used when running ``docker build``, specifically the PyTorch on ROCm base image. We have tested ROCm 5.7 and ROCm 6.0. The default is `rocm/pytorch:rocm6.0_ubuntu20.04_py3.9_pytorch_2.1.1`
106 -* `FX_GFX_ARCHS`: specifies the GFX architecture that is used to build flash-attention, for example, `gfx90a;gfx942` for MI200 and MI300. The default is `gfx90a;gfx942`
107 -* `FA_BRANCH`: specifies the branch used to build the flash-attention in `ROCmSoftwarePlatform's flash-attention repo <https://github.com/ROCmSoftwarePlatform/flash-attention>`_. The default is `3d2b6f5`
108 -* `BUILD_FA`: specifies whether to build flash-attention. For `Radeon RX 7900 series (gfx1100) <https://rocm.docs.amd.com/projects/radeon/en/latest/index.html>`_, this should be set to 0 before flash-attention supports this target.
50 +.. code-block:: console
51
110 -Their values can be passed in when running ``docker build`` with ``--build-arg`` options.
52 + $ docker build --build-arg BUILD_FA="0" -f Dockerfile.rocm -t vllm-rocm .
53
112 -For example, to build docker image for vllm on ROCm 5.7, you can run:
54 +To build docker image for vllm on ROCm 5.7, you can specify ``BASE_IMAGE`` as below:
55
56 .. code-block:: console
57
58 $ docker build --build-arg BASE_IMAGE="rocm/pytorch:rocm5.7_ubuntu22.04_py3.10_pytorch_2.0.1" \
59 -f Dockerfile.rocm -t vllm-rocm .
60
119 -To build vllm on ROCm 6.0, you can use the default:
61 +To run the above docker image ``vllm-rocm``, use the below command:
62
63 .. code-block:: console
64
123 - $ docker build -f Dockerfile.rocm -t vllm-rocm .
65 $ docker run -it \
66 --network=host \
67 --group-add=video \
@@ -133,7 +74,13 @@ To build vllm on ROCm 6.0, you can use the default:
74 vllm-rocm \
75 bash
76
136 -Alternatively, if you plan to install vLLM-ROCm on a local machine or start from a fresh docker image (e.g. rocm/pytorch), you can follow the steps below:
77 +Where the `<path/to/model>` is the location where the model is stored, for example, the weights for llama2 or llama3 models.
78 +
79 +
80 +.. _build_from_source_rocm:
81 +
82 +Option 2: Build from source
83 +---------------------------
84
85 0. Install prerequisites (skip if you are already in an environment/docker with the following installed):
86
@@ -141,32 +88,50 @@ Alternatively, if you plan to install vLLM-ROCm on a local machine or start from
88 - `Pytorch <https://pytorch.org/>`_
89 - `hipBLAS <https://rocm.docs.amd.com/projects/hipBLAS/en/latest/install.html>`_
90
144 -1. Install `flash attention for ROCm <https://github.com/ROCmSoftwarePlatform/flash-attention/tree/flash_attention_for_rocm>`_
91 +For installing PyTorch, you can start from a fresh docker image, e.g, `rocm6.0.2_ubuntu22.04_py3.10_pytorch_2.1.2`, `rocm/pytorch:rocm6.0_ubuntu20.04_py3.9_pytorch_2.1.1`, `rocm/pytorch-nightly`.
92
146 - Install ROCm's flash attention (v2.0.4) following the instructions from `ROCmSoftwarePlatform/flash-attention <https://github.com/ROCmSoftwarePlatform/flash-attention/tree/flash_attention_for_rocm#amd-gpurocm-support>`_
93 +Alternatively, you can install pytorch using pytorch wheels. You can check Pytorch installation guild in Pytorch `Getting Started <https://pytorch.org/get-started/locally/>`_
94 +
95 +For rocm6.0:
96 +
97 +.. code-block:: console
98 +
99 + $ pip3 install torch --index-url https://download.pytorch.org/whl/rocm6.0
100 +
101 +
102 +For rocm5.7:
103 +
104 +.. code-block:: console
105 +
106 + $ pip install torch --index-url https://download.pytorch.org/whl/rocm5.7
107 +
108 +
109 +1. Install `Triton flash attention for ROCm <https://github.com/ROCm/triton>`_
110 +
111 +Install ROCm's Triton flash attention (the default triton-mlir branch) following the instructions from `ROCm/triton <https://github.com/ROCm/triton/blob/triton-mlir/README.md>`_
112 +
113 +2. Optionally, if you choose to use CK flash attention, you can install `flash attention for ROCm <https://github.com/ROCm/flash-attention/tree/flash_attention_for_rocm>`_
114 +
115 +Install ROCm's flash attention (v2.0.4) following the instructions from `ROCm/flash-attention <https://github.com/ROCm/flash-attention/tree/flash_attention_for_rocm#amd-gpurocm-support>`_
116
117 .. note::
118 - If you are using rocm5.7 with pytorch 2.1.0 onwards, you don't need to apply the `hipify_python.patch`. You can build the ROCm flash attention directly.
150 - - If you fail to install `ROCmSoftwarePlatform/flash-attention`, try cloning from the commit `6fd2f8e572805681cd67ef8596c7e2ce521ed3c6`.
119 + - If you fail to install `ROCm/flash-attention`, try cloning from the commit `6fd2f8e572805681cd67ef8596c7e2ce521ed3c6`.
120 - ROCm's Flash-attention-2 (v2.0.4) does not support sliding windows attention.
121 - You might need to downgrade the "ninja" version to 1.10 it is not used when compiling flash-attention-2 (e.g. `pip install ninja==1.10.2.4`)
122
154 -2. Setup `xformers==0.0.23` without dependencies, and apply patches to adapt for ROCm flash attention
155 -
156 - .. code-block:: console
157 -
158 - $ pip install xformers==0.0.23 --no-deps
159 - $ bash patch_xformers.rocm.sh
160 -
123 3. Build vLLM.
124
163 - .. code-block:: console
125 +.. code-block:: console
126
165 - $ cd vllm
166 - $ pip install -U -r requirements-rocm.txt
167 - $ python setup.py install # This may take 5-10 minutes.
127 + $ cd vllm
128 + $ pip install -U -r requirements-rocm.txt
129 + $ python setup.py install # This may take 5-10 minutes. Currently, `pip install .`` does not work for ROCm installation
130
169 -.. note::
131
171 - - You may need to turn on the ``--enforce-eager`` flag if you experience process hang when running the `benchmark_thoughput.py` script to test your installation.
132 +.. tip::
133
134 + - You may need to turn on the ``--enforce-eager`` flag if you experience process hang when running the `benchmark_thoughput.py` script to test your installation.
135 + - Triton flash attention is used by default. For benchmarking purposes, it is recommended to run a warm up step before collecting perf numbers.
136 + - To use CK flash-attention, please use this flag ``export VLLM_USE_FLASH_ATTN_TRITON=0`` to turn off triton flash attention.
137 + - The ROCm version of pytorch, ideally, should match the ROCm driver version.