A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 06:00 UTC
[Doc] Fix description in the Automatic Prefix Caching design doc (#19333)
Signed-off-by: cr7258 <chengzw258@163.com>
Se7en committed
Jun 9, 2025 at 17:30 UTC
0eca5eacd07061c5692250d0ef86267da831bd9f
1 file changed
+1
-1
docs/design/v1/prefix_caching.md
+1
-1
@@ -144,7 +144,7 @@ As a result, we will have the following components when the KV cache manager is
144
145
**Running request:** Workflow for the scheduler to schedule a running request with KV cache block allocation:
146
147
-1. The scheduler calls `kv_cache_manager.append_slots()`. It does the following steps:
147
+1. The scheduler calls `kv_cache_manager.allocate_slots()`. It does the following steps:
148
1. Compute the number of new required blocks, and return if there are no sufficient blocks to allocate.
149
2. Allocate new blocks by popping the heads of the free queue. If the head block is a cached block, this also “evicts” the block so that no other requests can reuse it anymore from now on.
150
3. Append token IDs to the slots in existing blocks as well as the new blocks. If a block is full, we add it to the Cache Block to cache it.