A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 10, 11:30 UTC
[Doc] Improve MM models LoRA notes (#31979)
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Jee Jee Li committed
Jan 9, 2026 at 00:55 UTC
49568d5cf984f9ea0f96dc636036075ef6ddec45
1 file changed
+1
-23
docs/models/supported_models.md
+1
-23
@@ -642,29 +642,7 @@ See [this page](../features/multimodal_inputs.md) on how to pass multi-modal inp
642
For hybrid-only models such as Llama-4, Step3 and Mistral-3, a text-only mode can be enabled by setting all supported multimodal modalities to 0 (e.g, `--limit-mm-per-prompt '{"image":0}`) so that their multimodal modules will not be loaded to free up more GPU memory for KV cache.
643
644
!!! note
645
- vLLM currently only supports dynamic LoRA adapters on the language backbone of multimodal models.
646
- If you wish to use a model with LoRA in the multi-modal encoder,
647
- please merge the weights into the base model first before running it in vLLM like a regular model.
648
-
649
- ```python
650
- from peft import PeftConfig, PeftModel
651
- from transformers import AutoModelForImageTextToText, AutoProcessor
652
-
653
- def merge_and_save(model_id: str, output_dir: str):
654
- base_model = AutoModelForImageTextToText.from_pretrained(model_id)
655
- lora_model = PeftModel.from_pretrained(
656
- base_model,
657
- model_id,
658
- config=PeftConfig.from_pretrained(model_id),
659
- )
660
- model = lora_model.merge_and_unload().to(dtype=base_model.dtype)
661
- model._hf_peft_config_loaded = False # Needed to save the merged model
662
-
663
- processor = AutoProcessor.from_pretrained(model_id)
664
-
665
- model.save_pretrained(output_dir)
666
- processor.save_pretrained(output_dir)
667
- ```
645
+ vLLM currently supports adding LoRA adapters to the language backbone for most multimodal models. Additionally, vLLM now experimentally supports adding LoRA to the tower and connector modules for some multimodal models. See [this page](../features/lora.md).
646
647
### Generative Models
648