A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 12, 06:00 UTC
Update FAQ on interleaving sliding windows support (#29796)
Signed-off-by: Finbarr Timbers <finbarrtimbers@gmail.com>
Finbarr Timbers committed
Dec 1, 2025 at 12:15 UTC
38caf7fa1a2f57a4a791ffa4ba0f9f55d409e0d0
1 file changed
-2
docs/contributing/model/basic.md
-2
@@ -113,8 +113,6 @@ See [this page](registration.md) for instructions on how to register your new mo
113
114
### How to support models with interleaving sliding windows?
115
116
-For models with interleaving sliding windows (e.g. `google/gemma-2-2b-it` and `mistralai/Ministral-8B-Instruct-2410`), the scheduler will treat the model as a full-attention model, i.e., kv-cache of all tokens will not be dropped. This is to make sure prefix caching works with these models. Sliding window only appears as a parameter to the attention kernel computation.
117
-
116
To support a model with interleaving sliding windows, we need to take care of the following details:
117
118
- Make sure the model's `config.json` contains `layer_types`.