A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 12, 03:30 UTC
[Doc] Add allocate_slots parameter docs (#29777)
Signed-off-by: maang <maang_h@163.com> Signed-off-by: maang-h <55082429+maang-h@users.noreply.github.com> Co-authored-by: Chen Zhang <zhangch99@outlook.com>
maang-h committed
Dec 3, 2025 at 07:23 UTC
5d91d2b292be9b1d6b121d36d242d5077a031e4b
1 file changed
+3
vllm/v1/core/kv_cache_manager.py
+3
@@ -230,6 +230,9 @@ class KVCacheManager:
230
delay_cache_blocks: Whether to skip caching the blocks. This is
231
used by P/D when allocating blocks used in a KV transfer
232
which will complete in a future step.
233
+ num_encoder_tokens: The number of encoder tokens to allocate for
234
+ cross-attention in encoder-decoder models(e.g., Whisper).
235
+ For decoder-only models, this should be 0.
236
237
Blocks layout:
238
```