@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 9e74d9d003
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 23:15 UTC

Correct alignment in the seq_len diagram. (#5592)

Co-authored-by: Liqian Chen <liqian.chen@deeplang.ai>

Charles Riggins committed Jun 18, 2024 at 00:05 UTC 9e74d9d003d546c17dca472c3f4b48be651f1d7c
1 file changed +1 -1
vllm/attention/backends/flash_attn.py
+1 -1
@@ -83,7 +83,7 @@ class FlashAttentionMetadata(AttentionMetadata):
83 # |---------------- N iteration ---------------------|
84 # |- tokenA -|......................|-- newTokens ---|
85 # |---------- context_len ----------|
86 - # |-------------------- seq_len ----------------------|
86 + # |-------------------- seq_len ---------------------|
87 # |-- query_len ---|
88
89 # Maximum query length in the batch. None for decoding.