A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 22:35 UTC
[Doc] Remove vLLM prefix and add citation for PagedAttention (#21910)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Cyrus Leung committed
Jul 30, 2025 at 21:36 UTC
fcfd1eb9c556e295eb5708eb0f5e6ae775807775
10 files changed
+22
-11
docs/assets/design/paged_attention/k_vecs.png
renamed
docs/assets/design/paged_attention/key.png
renamed
docs/assets/design/paged_attention/logits_vec.png
renamed
docs/assets/design/paged_attention/q_vecs.png
renamed
docs/assets/design/paged_attention/query.png
renamed
docs/assets/design/paged_attention/v_vec.png
renamed
docs/assets/design/paged_attention/value.png
renamed
docs/design/paged_attention.md
+20
-9
@@ -1,7 +1,7 @@
1
-# vLLM Paged Attention
1
+# Paged Attention
2
3
!!! warning
4
- This document is being kept in the vLLM documentation for historical purposes.
4
+ This is a historical document based on the [original paper for vLLM](https://arxiv.org/abs/2309.06180).
5
It no longer describes the code used in vLLM today.
6
7
Currently, vLLM utilizes its own implementation of a multi-head query
@@ -140,7 +140,7 @@ const scalar_t* q_ptr = q + seq_idx * q_stride + head_idx * HEAD_SIZE;
140
```
141
142
<figure markdown="span">
143
- { align="center" alt="query" width="70%" }
143
+ { align="center" alt="query" width="70%" }
144
</figure>
145
146
Each thread defines its own `q_ptr` which points to the assigned
@@ -149,7 +149,7 @@ and `HEAD_SIZE` is 128, the `q_ptr` points to data that contains
149
total of 128 elements divided into 128 / 4 = 32 vecs.
150
151
<figure markdown="span">
152
- { align="center" alt="q_vecs" width="70%" }
152
+ { align="center" alt="q_vecs" width="70%" }
153
</figure>
154
155
```cpp
@@ -188,7 +188,7 @@ points to key token data based on `k_cache` at assigned block,
188
assigned head and assigned token.
189
190
<figure markdown="span">
191
- { align="center" alt="key" width="70%" }
191
+ { align="center" alt="key" width="70%" }
192
</figure>
193
194
The diagram above illustrates the memory layout for key data. It
@@ -203,7 +203,7 @@ elements for one token) that will be processed by 2 threads (one
203
thread group) separately.
204
205
<figure markdown="span">
206
- { align="center" alt="k_vecs" width="70%" }
206
+ { align="center" alt="k_vecs" width="70%" }
207
</figure>
208
209
```cpp
@@ -362,15 +362,15 @@ later steps. Now, it should store the normalized softmax result of
362
## Value
363
364
<figure markdown="span">
365
- { align="center" alt="value" width="70%" }
365
+ { align="center" alt="value" width="70%" }
366
</figure>
367
368
<figure markdown="span">
369
- { align="center" alt="logits_vec" width="50%" }
369
+ { align="center" alt="logits_vec" width="50%" }
370
</figure>
371
372
<figure markdown="span">
373
- { align="center" alt="v_vec" width="70%" }
373
+ { align="center" alt="v_vec" width="70%" }
374
</figure>
375
376
Now we need to retrieve the value data and perform dot multiplication
@@ -499,3 +499,14 @@ for (int i = 0; i < NUM_ROWS_PER_THREAD; i++) {
499
Finally, we need to iterate over different assigned head positions
500
and write out the corresponding accumulated result based on the
501
`out_ptr`.
502
+
503
+## Citation
504
+
505
+```bibtex
506
+@inproceedings{kwon2023efficient,
507
+ title={Efficient Memory Management for Large Language Model Serving with PagedAttention},
508
+ author={Woosuk Kwon and Zhuohan Li and Siyuan Zhuang and Ying Sheng and Lianmin Zheng and Cody Hao Yu and Joseph E. Gonzalez and Hao Zhang and Ion Stoica},
509
+ booktitle={Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles},
510
+ year={2023}
511
+}
512
+```
docs/design/plugin_system.md
+1
-1
@@ -1,4 +1,4 @@
1
-# vLLM's Plugin System
1
+# Plugin System
2
3
The community frequently requests the ability to extend vLLM with custom features. To facilitate this, vLLM includes a plugin system that allows users to add custom features without modifying the vLLM codebase. This document explains how plugins work in vLLM and how to create a plugin for vLLM.
4
docs/design/torch_compile.md
+1
-1
@@ -1,4 +1,4 @@
1
-# vLLM's `torch.compile` integration
1
+# `torch.compile` integration
2
3
In vLLM's V1 architecture, `torch.compile` is enabled by default and is a critical part of the framework. This document gives a simple walk-through example to show how to understand the `torch.compile` usage.
4