A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 18:30 UTC
[Doc][V1] Fix V1 APC doc (#14920)
Shanshan Shen committed
Mar 18, 2025 at 16:15 UTC
d1695758b2f65fd314d1aee71ba2469ceba67a5b
1 file changed
+2
-2
docs/source/design/v1/prefix_caching.md
+2
-2
@@ -191,7 +191,7 @@ When the head block (least recently used block) of the free queue is cached, we
191
192
In this example, we assume the block size is 4 (each block can cache 4 tokens), and we have 10 blocks in the KV-cache manager in total.
193
194
-**Time 1: The cache is empty and a new request comes in.** We allocate 4 blocks. 3 of them are already full and cached. The fourth block is partially full with 2 of 4 tokens.
194
+**Time 1: The cache is empty and a new request comes in.** We allocate 4 blocks. 3 of them are already full and cached. The fourth block is partially full with 3 of 4 tokens.
195
196
:::{image} /assets/design/v1/prefix_caching/example-time-1.png
197
:alt: Example Time 1
@@ -203,7 +203,7 @@ In this example, we assume the block size is 4 (each block can cache 4 tokens),
203
:alt: Example Time 3
204
:::
205
206
-**Time 4: Request 1 comes in with the 14 prompt tokens, where the first 11 tokens are the same as request 0.** We can see that only 2 blocks (11 tokens) hit the cache, because the 3rd block only matches 3 of 4 tokens.
206
+**Time 4: Request 1 comes in with the 14 prompt tokens, where the first 10 tokens are the same as request 0.** We can see that only the first 2 blocks (8 tokens) hit the cache, because the 3rd block only matches 2 of 4 tokens.
207
208
:::{image} /assets/design/v1/prefix_caching/example-time-4.png
209
:alt: Example Time 4