A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 03:15 UTC
[Doc][5/N] Move Community and API Reference to the bottom (#11896)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Co-authored-by: Simon Mo <simon.mo@hey.com>
Cyrus Leung committed
Jan 10, 2025 at 11:10 UTC
c3cf54dda4df200bc8913ed69d210a7108dfa320
3 files changed
+40
-26
README.md
+1
-1
@@ -41,7 +41,7 @@ vLLM is a fast and easy-to-use library for LLM inference and serving.
41
vLLM is fast with:
42
43
- State-of-the-art serving throughput
44
-- Efficient management of attention key and value memory with **PagedAttention**
44
+- Efficient management of attention key and value memory with [**PagedAttention**](https://blog.vllm.ai/2023/06/20/vllm.html)
45
- Continuous batching of incoming requests
46
- Fast model execution with CUDA/HIP graph
47
- Quantizations: [GPTQ](https://arxiv.org/abs/2210.17323), [AWQ](https://arxiv.org/abs/2306.00978), INT4, INT8, and FP8.
docs/source/design/automatic_prefix_caching.md
+1
-1
@@ -2,7 +2,7 @@
2
3
# Automatic Prefix Caching
4
5
-The core idea of [PagedAttention](#design-paged-attention) is to partition the KV cache of each request into KV Blocks. Each block contains the attention keys and values for a fixed number of tokens. The PagedAttention algorithm allows these blocks to be stored in non-contiguous physical memory so that we can eliminate memory fragmentation by allocating the memory on demand.
5
+The core idea of [PagedAttention](https://blog.vllm.ai/2023/06/20/vllm.html) is to partition the KV cache of each request into KV Blocks. Each block contains the attention keys and values for a fixed number of tokens. The PagedAttention algorithm allows these blocks to be stored in non-contiguous physical memory so that we can eliminate memory fragmentation by allocating the memory on demand.
6
7
To automatically cache the KV cache, we utilize the following key observation: Each KV block can be uniquely identified by the tokens within the block and the tokens in the prefix before the block.
8
docs/source/index.md
+38
-24
@@ -26,7 +26,7 @@ vLLM is a fast and easy-to-use library for LLM inference and serving.
26
vLLM is fast with:
27
28
- State-of-the-art serving throughput
29
-- Efficient management of attention key and value memory with **PagedAttention**
29
+- Efficient management of attention key and value memory with [**PagedAttention**](https://blog.vllm.ai/2023/06/20/vllm.html)
30
- Continuous batching of incoming requests
31
- Fast model execution with CUDA/HIP graph
32
- Quantization: [GPTQ](https://arxiv.org/abs/2210.17323), [AWQ](https://arxiv.org/abs/2306.00978), INT4, INT8, and FP8
@@ -54,6 +54,8 @@ For more information, check out the following:
54
55
## Documentation
56
57
+% How to start using vLLM?
58
+
59
```{toctree}
60
:caption: Getting Started
61
:maxdepth: 1
@@ -65,6 +67,8 @@ getting_started/troubleshooting
67
getting_started/faq
68
```
69
70
+% What does vLLM support?
71
+
72
```{toctree}
73
:caption: Models
74
:maxdepth: 1
@@ -75,6 +79,8 @@ models/supported_models
79
models/extensions/index
80
```
81
82
+% Additional capabilities
83
+
84
```{toctree}
85
:caption: Features
86
:maxdepth: 1
@@ -89,6 +95,8 @@ features/spec_decode
95
features/compatibility_matrix
96
```
97
98
+% Details about running vLLM
99
+
100
```{toctree}
101
:caption: Inference and Serving
102
:maxdepth: 1
@@ -104,6 +112,8 @@ serving/usage_stats
112
serving/integrations/index
113
```
114
115
+% Scaling up vLLM for production
116
+
117
```{toctree}
118
:caption: Deployment
119
:maxdepth: 1
@@ -115,6 +125,8 @@ deployment/frameworks/index
125
deployment/integrations/index
126
```
127
128
+% Making the most out of vLLM
129
+
130
```{toctree}
131
:caption: Performance
132
:maxdepth: 1
@@ -123,28 +135,7 @@ performance/optimization
135
performance/benchmarks
136
```
137
126
-% Community: User community resources
127
-
128
-```{toctree}
129
-:caption: Community
130
-:maxdepth: 1
131
-
132
-community/meetups
133
-community/sponsors
134
-```
135
-
136
-```{toctree}
137
-:caption: API Reference
138
-:maxdepth: 2
139
-
140
-api/offline_inference/index
141
-api/engine/index
142
-api/inference_params
143
-api/multimodal/index
144
-api/model/index
145
-```
146
-
147
-% Design Documents: Details about vLLM internals
138
+% Explanation of vLLM internals
139
140
```{toctree}
141
:caption: Design Documents
@@ -159,7 +150,7 @@ design/automatic_prefix_caching
150
design/multiprocessing
151
```
152
162
-% Developer Guide: How to contribute to the vLLM project
153
+% How to contribute to the vLLM project
154
155
```{toctree}
156
:caption: Developer Guide
@@ -172,6 +163,29 @@ contributing/model/index
163
contributing/vulnerability_management
164
```
165
166
+% Technical API specifications
167
+
168
+```{toctree}
169
+:caption: API Reference
170
+:maxdepth: 2
171
+
172
+api/offline_inference/index
173
+api/engine/index
174
+api/inference_params
175
+api/multimodal/index
176
+api/model/index
177
+```
178
+
179
+% Latest news and acknowledgements
180
+
181
+```{toctree}
182
+:caption: Community
183
+:maxdepth: 1
184
+
185
+community/meetups
186
+community/sponsors
187
+```
188
+
189
# Indices and tables
190
191
- {ref}`genindex`