A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 20:45 UTC
[Docs][ROCm] Detailed instructions to build from source (#6680)
Woosuk Kwon committed
Jul 24, 2024 at 01:07 UTC
ccc4a73257b61d5f1249999f6a804d60c2d1a518
1 file changed
+29
docs/source/getting_started/amd-installation.rst
+29
@@ -107,6 +107,35 @@ Alternatively, wheels intended for vLLM use can be accessed under the releases.
107
$ python setup.py develop # This may take 5-10 minutes. Currently, `pip install .`` does not work for ROCm installation
108
109
110
+.. tip::
111
+
112
+ For example, vLLM v0.5.3 on ROCM 6.1 can be built with the following steps:
113
+
114
+ .. code-block:: console
115
+
116
+ $ pip install --upgrade pip
117
+
118
+ $ # Install PyTorch
119
+ $ pip uninstall torch -y
120
+ $ pip install --no-cache-dir --pre torch==2.5.0.dev20240710 --index-url https://download.pytorch.org/whl/nightly/rocm6.1
121
+
122
+ $ # Build & install AMD SMI
123
+ $ pip install /opt/rocm/share/amd_smi
124
+
125
+ $ # Install dependencies
126
+ $ pip install --upgrade numba scipy huggingface-hub[cli]
127
+ $ pip install "numpy<2"
128
+ $ pip install -r requirements-rocm.txt
129
+
130
+ $ # Apply the patch to ROCM 6.1 (requires root permission)
131
+ $ wget -N https://github.com/ROCm/vllm/raw/fa78403/rocm_patch/libamdhip64.so.6 -P /opt/rocm/lib
132
+ $ rm -f "$(python3 -c 'import torch; print(torch.__path__[0])')"/lib/libamdhip64.so*
133
+
134
+ $ # Build vLLM for MI210/MI250/MI300.
135
+ $ export PYTORCH_ROCM_ARCH="gfx90a;gfx942"
136
+ $ python3 setup.py develop
137
+
138
+
139
.. tip::
140
141
- Triton flash attention is used by default. For benchmarking purposes, it is recommended to run a warm up step before collecting perf numbers.