A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 05:30 UTC
[V1] [Hybrid] Disable prefix caching by default for hybrid or mamba-based models (#23716)
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
Thomas Parnell committed
Aug 27, 2025 at 14:51 UTC
704432af3c129b7a57fca9b059eefe214159f836
2 files changed
+11
-8
docs/usage/v1_guide.md
+6
-4
@@ -107,14 +107,16 @@ to enable simultaneous generation and embedding using the same engine instance i
107
#### Mamba Models
108
109
Models using selective state-space mechanisms instead of standard transformer attention are supported.
110
-Models that use Mamba-2 and Mamba-1 layers (e.g., `Mamba2ForCausalLM`, `MambaForCausalLM`) are supported. Please note that these models currently require disabling prefix caching in V1.
110
+Models that use Mamba-2 and Mamba-1 layers (e.g., `Mamba2ForCausalLM`, `MambaForCausalLM`) are supported.
111
+Please note that prefix caching is not yet supported for these models.
112
113
Models that combine Mamba-2 and Mamba-1 layers with standard attention layers are also supported (e.g., `BambaForCausalLM`,
113
-`Zamba2ForCausalLM`, `NemotronHForCausalLM`, `FalconH1ForCausalLM` and `GraniteMoeHybridForCausalLM`, `JambaForCausalLM`). Please note that
114
-these models currently require disabling prefix caching in V1.
114
+`Zamba2ForCausalLM`, `NemotronHForCausalLM`, `FalconH1ForCausalLM` and `GraniteMoeHybridForCausalLM`, `JambaForCausalLM`).
115
+Please note that prefix caching is not yet supported for these models.
116
117
Hybrid models with mechanisms different to Mamba are also supported (e.g, `MiniMaxText01ForCausalLM`, `MiniMaxM1ForCausalLM`).
117
-Please note that these models currently require disabling prefix caching and enforcing eager mode in V1.
118
+Please note that prefix caching is not yet supported for these models.
119
+It is also necessary to enforce eager mode for these models in V1.
120
121
#### Encoder-Decoder Models
122
vllm/model_executor/models/config.py
+5
-4
@@ -292,12 +292,13 @@ class MambaModelConfig(VerifyAndUpdateConfig):
292
return
293
294
model_config = vllm_config.model_config
295
+ cache_config = vllm_config.cache_config
296
compilation_config = vllm_config.compilation_config
297
297
- model_cls, _ = ModelRegistry.resolve_model_cls(
298
- model_config.architecture,
299
- model_config=model_config,
300
- )
298
+ # TODO(tdoublep): remove once prefix caching is enabled
299
+ cache_config.enable_prefix_caching = False
300
+ logger.info("Hybrid or mamba-based model detected: disabling prefix "
301
+ "caching since it is not yet supported.")
302
303
# TODO(tdoublep): remove as full cuda graph support is added
304
FCG_NOT_SUPPORTED_MODELS = [