@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / ccc4a73257
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 20:45 UTC

[Docs][ROCm] Detailed instructions to build from source (#6680)

Woosuk Kwon committed Jul 24, 2024 at 01:07 UTC ccc4a73257b61d5f1249999f6a804d60c2d1a518
1 file changed +29
docs/source/getting_started/amd-installation.rst
+29
@@ -107,6 +107,35 @@ Alternatively, wheels intended for vLLM use can be accessed under the releases.
107 $ python setup.py develop # This may take 5-10 minutes. Currently, `pip install .`` does not work for ROCm installation
108
109
110 +.. tip::
111 +
112 + For example, vLLM v0.5.3 on ROCM 6.1 can be built with the following steps:
113 +
114 + .. code-block:: console
115 +
116 + $ pip install --upgrade pip
117 +
118 + $ # Install PyTorch
119 + $ pip uninstall torch -y
120 + $ pip install --no-cache-dir --pre torch==2.5.0.dev20240710 --index-url https://download.pytorch.org/whl/nightly/rocm6.1
121 +
122 + $ # Build & install AMD SMI
123 + $ pip install /opt/rocm/share/amd_smi
124 +
125 + $ # Install dependencies
126 + $ pip install --upgrade numba scipy huggingface-hub[cli]
127 + $ pip install "numpy<2"
128 + $ pip install -r requirements-rocm.txt
129 +
130 + $ # Apply the patch to ROCM 6.1 (requires root permission)
131 + $ wget -N https://github.com/ROCm/vllm/raw/fa78403/rocm_patch/libamdhip64.so.6 -P /opt/rocm/lib
132 + $ rm -f "$(python3 -c 'import torch; print(torch.__path__[0])')"/lib/libamdhip64.so*
133 +
134 + $ # Build vLLM for MI210/MI250/MI300.
135 + $ export PYTORCH_ROCM_ARCH="gfx90a;gfx942"
136 + $ python3 setup.py develop
137 +
138 +
139 .. tip::
140
141 - Triton flash attention is used by default. For benchmarking purposes, it is recommended to run a warm up step before collecting perf numbers.