@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 7a0b011dd5
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 05:30 UTC

Add a 1-line docstring to explain why calling context_attention_fwd twice in test_prefix_prefill.py (#2553)

Jason Zhu committed Jan 22, 2024 at 14:47 UTC 7a0b011dd51e5c6b48e8f8f5424be0995b5cb8ee
1 file changed +1
tests/kernels/test_prefix_prefill.py
+1
@@ -125,6 +125,7 @@ def test_contexted_kv_attention(
125 v_cache = v_cache.view(-1, block_size, num_heads,
126 head_size).permute(0, 2, 3, 1).contiguous()
127
128 + # Warm up the Triton kernel by calling it once before actually measuring generation time
129 context_attention_fwd(query, k, v, output, k_cache, v_cache, block_table,
130 b_start_loc, b_seq_len, b_ctx_len, max_input_len)
131 torch.cuda.synchronize()