A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 20:45 UTC
[Doc][AMD][ROCm]Added tips to refer to mi300x tuning guide for mi300x users (#6754)
Hongxia Yang committed
Jul 24, 2024 at 17:32 UTC
d88c458f44f5bc0d01215310f8abb5d63fa106d4
1 file changed
+7
docs/source/getting_started/amd-installation.rst
+7
@@ -142,3 +142,10 @@ Alternatively, wheels intended for vLLM use can be accessed under the releases.
142
- Triton flash attention does not currently support sliding window attention. If using half precision, please use CK flash-attention for sliding window support.
143
- To use CK flash-attention or PyTorch naive attention, please use this flag ``export VLLM_USE_TRITON_FLASH_ATTN=0`` to turn off triton flash attention.
144
- The ROCm version of PyTorch, ideally, should match the ROCm driver version.
145
+
146
+
147
+.. tip::
148
+ - For MI300x (gfx942) users, to achieve optimal performance, please refer to `MI300x tuning guide <https://rocm.docs.amd.com/en/latest/how-to/tuning-guides/mi300x/index.html>`_ for performance optimization and tuning tips on system and workflow level.
149
+ For vLLM, please refer to `vLLM performance optimization <https://rocm.docs.amd.com/en/latest/how-to/tuning-guides/mi300x/workload.html#vllm-performance-optimization>`_.
150
+
151
+