@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / f48c6791b7
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 04:15 UTC

[FIX] Fix prefix test error on main (#3286)

Zhuohan Li committed Mar 8, 2024 at 17:16 UTC f48c6791b7bfc2579ad575d33ed83912f0bfb011
1 file changed -2
vllm/model_executor/layers/attention/backends/flash_attn.py
-2
@@ -103,8 +103,6 @@ class FlashAttentionBackend:
103 key_cache,
104 value_cache,
105 input_metadata,
106 - self.num_heads,
107 - self.num_kv_heads,
106 self.alibi_slopes,
107 )
108 else: