@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / c3cf54dda4
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 03:15 UTC

[Doc][5/N] Move Community and API Reference to the bottom (#11896)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Co-authored-by: Simon Mo <simon.mo@hey.com>

Cyrus Leung committed Jan 10, 2025 at 11:10 UTC c3cf54dda4df200bc8913ed69d210a7108dfa320
3 files changed +40 -26
README.md
+1 -1
@@ -41,7 +41,7 @@ vLLM is a fast and easy-to-use library for LLM inference and serving.
41 vLLM is fast with:
42
43 - State-of-the-art serving throughput
44 -- Efficient management of attention key and value memory with **PagedAttention**
44 +- Efficient management of attention key and value memory with [**PagedAttention**](https://blog.vllm.ai/2023/06/20/vllm.html)
45 - Continuous batching of incoming requests
46 - Fast model execution with CUDA/HIP graph
47 - Quantizations: [GPTQ](https://arxiv.org/abs/2210.17323), [AWQ](https://arxiv.org/abs/2306.00978), INT4, INT8, and FP8.
docs/source/design/automatic_prefix_caching.md
+1 -1
@@ -2,7 +2,7 @@
2
3 # Automatic Prefix Caching
4
5 -The core idea of [PagedAttention](#design-paged-attention) is to partition the KV cache of each request into KV Blocks. Each block contains the attention keys and values for a fixed number of tokens. The PagedAttention algorithm allows these blocks to be stored in non-contiguous physical memory so that we can eliminate memory fragmentation by allocating the memory on demand.
5 +The core idea of [PagedAttention](https://blog.vllm.ai/2023/06/20/vllm.html) is to partition the KV cache of each request into KV Blocks. Each block contains the attention keys and values for a fixed number of tokens. The PagedAttention algorithm allows these blocks to be stored in non-contiguous physical memory so that we can eliminate memory fragmentation by allocating the memory on demand.
6
7 To automatically cache the KV cache, we utilize the following key observation: Each KV block can be uniquely identified by the tokens within the block and the tokens in the prefix before the block.
8
docs/source/index.md
+38 -24
@@ -26,7 +26,7 @@ vLLM is a fast and easy-to-use library for LLM inference and serving.
26 vLLM is fast with:
27
28 - State-of-the-art serving throughput
29 -- Efficient management of attention key and value memory with **PagedAttention**
29 +- Efficient management of attention key and value memory with [**PagedAttention**](https://blog.vllm.ai/2023/06/20/vllm.html)
30 - Continuous batching of incoming requests
31 - Fast model execution with CUDA/HIP graph
32 - Quantization: [GPTQ](https://arxiv.org/abs/2210.17323), [AWQ](https://arxiv.org/abs/2306.00978), INT4, INT8, and FP8
@@ -54,6 +54,8 @@ For more information, check out the following:
54
55 ## Documentation
56
57 +% How to start using vLLM?
58 +
59 ```{toctree}
60 :caption: Getting Started
61 :maxdepth: 1
@@ -65,6 +67,8 @@ getting_started/troubleshooting
67 getting_started/faq
68 ```
69
70 +% What does vLLM support?
71 +
72 ```{toctree}
73 :caption: Models
74 :maxdepth: 1
@@ -75,6 +79,8 @@ models/supported_models
79 models/extensions/index
80 ```
81
82 +% Additional capabilities
83 +
84 ```{toctree}
85 :caption: Features
86 :maxdepth: 1
@@ -89,6 +95,8 @@ features/spec_decode
95 features/compatibility_matrix
96 ```
97
98 +% Details about running vLLM
99 +
100 ```{toctree}
101 :caption: Inference and Serving
102 :maxdepth: 1
@@ -104,6 +112,8 @@ serving/usage_stats
112 serving/integrations/index
113 ```
114
115 +% Scaling up vLLM for production
116 +
117 ```{toctree}
118 :caption: Deployment
119 :maxdepth: 1
@@ -115,6 +125,8 @@ deployment/frameworks/index
125 deployment/integrations/index
126 ```
127
128 +% Making the most out of vLLM
129 +
130 ```{toctree}
131 :caption: Performance
132 :maxdepth: 1
@@ -123,28 +135,7 @@ performance/optimization
135 performance/benchmarks
136 ```
137
126 -% Community: User community resources
127 -
128 -```{toctree}
129 -:caption: Community
130 -:maxdepth: 1
131 -
132 -community/meetups
133 -community/sponsors
134 -```
135 -
136 -```{toctree}
137 -:caption: API Reference
138 -:maxdepth: 2
139 -
140 -api/offline_inference/index
141 -api/engine/index
142 -api/inference_params
143 -api/multimodal/index
144 -api/model/index
145 -```
146 -
147 -% Design Documents: Details about vLLM internals
138 +% Explanation of vLLM internals
139
140 ```{toctree}
141 :caption: Design Documents
@@ -159,7 +150,7 @@ design/automatic_prefix_caching
150 design/multiprocessing
151 ```
152
162 -% Developer Guide: How to contribute to the vLLM project
153 +% How to contribute to the vLLM project
154
155 ```{toctree}
156 :caption: Developer Guide
@@ -172,6 +163,29 @@ contributing/model/index
163 contributing/vulnerability_management
164 ```
165
166 +% Technical API specifications
167 +
168 +```{toctree}
169 +:caption: API Reference
170 +:maxdepth: 2
171 +
172 +api/offline_inference/index
173 +api/engine/index
174 +api/inference_params
175 +api/multimodal/index
176 +api/model/index
177 +```
178 +
179 +% Latest news and acknowledgements
180 +
181 +```{toctree}
182 +:caption: Community
183 +:maxdepth: 1
184 +
185 +community/meetups
186 +community/sponsors
187 +```
188 +
189 # Indices and tables
190
191 - {ref}`genindex`