@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / c0724fc915
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 01:45 UTC

[ROCm][Hardware][AMD] Adding Navi21 to fallback to naive attention if Triton is not used (#4658)

alexeykondrat committed May 18, 2024 at 01:09 UTC c0724fc9150329d42abaf2f0f77dc8ca91d48acb
1 file changed +3 -2
vllm/attention/backends/rocm_flash_attn.py
+3 -2
@@ -231,8 +231,9 @@ class ROCmFlashAttentionImpl(AttentionImpl):
231 self.attn_func = triton_attention
232 logger.debug("Using Triton FA in ROCmBackend")
233 else:
234 - # if not using triton, navi3x not use flash-attn either
235 - if torch.cuda.get_device_capability()[0] == 11:
234 + # if not using triton, navi3x/navi21/navi10 do not use flash-attn
235 + # either
236 + if torch.cuda.get_device_capability()[0] != 9:
237 self.use_naive_attn = True
238 else:
239 try: