| 1 | # LoRA Adapters |
| 2 | |
| 3 | This document shows you how to use [LoRA adapters](https://arxiv.org/abs/2106.09685) with vLLM on top of a base model. |
| 4 | |
| 5 | LoRA adapters can be used with any vLLM model that implements [SupportsLoRA][vllm.model_executor.models.interfaces.SupportsLoRA]. |
| 6 | |
| 7 | Adapters can be efficiently served on a per-request basis with minimal overhead. First we download the adapter(s) and save |
| 8 | them locally with |
| 9 | |
| 10 | ```python |
| 11 | from huggingface_hub import snapshot_download |
| 12 | |
| 13 | sql_lora_path = snapshot_download(repo_id="jeeejeee/llama32-3b-text2sql-spider") |
| 14 | ``` |
| 15 | |
| 16 | Then we instantiate the base model and pass in the `enable_lora=True` flag: |
| 17 | |
| 18 | ```python |
| 19 | from vllm import LLM, SamplingParams |
| 20 | from vllm.lora.request import LoRARequest |
| 21 | |
| 22 | llm = LLM(model="meta-llama/Llama-3.2-3B-Instruct", enable_lora=True) |
| 23 | ``` |
| 24 | |
| 25 | We can now submit the prompts and call `llm.generate` with the `lora_request` parameter. The first parameter |
| 26 | of `LoRARequest` is a human identifiable name, the second parameter is a globally unique ID for the adapter and |
| 27 | the third parameter is the path to the LoRA adapter. |
| 28 | |
| 29 | ??? code |
| 30 | |
| 31 | ```python |
| 32 | sampling_params = SamplingParams( |
| 33 | temperature=0, |
| 34 | max_tokens=256, |
| 35 | stop=["[/assistant]"], |
| 36 | ) |
| 37 | |
| 38 | prompts = [ |
| 39 | "[user] Write a SQL query to answer the question based on the table schema.\n\n context: CREATE TABLE table_name_74 (icao VARCHAR, airport VARCHAR)\n\n question: Name the ICAO for lilongwe international airport [/user] [assistant]", |
| 40 | "[user] Write a SQL query to answer the question based on the table schema.\n\n context: CREATE TABLE table_name_11 (nationality VARCHAR, elector VARCHAR)\n\n question: When Anchero Pantaleone was the elector what is under nationality? [/user] [assistant]", |
| 41 | ] |
| 42 | |
| 43 | outputs = llm.generate( |
| 44 | prompts, |
| 45 | sampling_params, |
| 46 | lora_request=LoRARequest("sql_adapter", 1, sql_lora_path), |
| 47 | ) |
| 48 | ``` |
| 49 | |
| 50 | Check out [examples/features/lora/multilora_offline.py](../../examples/features/lora/multilora_offline.py) for an example of how to use LoRA adapters with the async engine and how to use more advanced configuration options. |
| 51 | |
| 52 | ## Serving LoRA Adapters |
| 53 | |
| 54 | LoRA adapted models can also be served with the Open-AI compatible vLLM server. To do so, we use |
| 55 | `--lora-modules {name}={path} {name}={path}` to specify each LoRA module when we kick off the server: |
| 56 | |
| 57 | ```bash |
| 58 | vllm serve meta-llama/Llama-3.2-3B-Instruct \ |
| 59 | --enable-lora \ |
| 60 | --lora-modules sql-lora=jeeejeee/llama32-3b-text2sql-spider |
| 61 | ``` |
| 62 | |
| 63 | The server entrypoint accepts all other LoRA configuration parameters (`max_loras`, `max_lora_rank`, `max_cpu_loras`, |
| 64 | etc.), which will apply to all forthcoming requests. Upon querying the `/models` endpoint, we should see our LoRA along |
| 65 | with its base model (if `jq` is not installed, you can follow [this guide](https://jqlang.org/download/) to install it.): |
| 66 | |
| 67 | ??? console "Command" |
| 68 | |
| 69 | ```bash |
| 70 | curl localhost:8000/v1/models | jq . |
| 71 | { |
| 72 | "object": "list", |
| 73 | "data": [ |
| 74 | { |
| 75 | "id": "meta-llama/Llama-3.2-3B-Instruct", |
| 76 | "object": "model", |
| 77 | ... |
| 78 | }, |
| 79 | { |
| 80 | "id": "sql-lora", |
| 81 | "object": "model", |
| 82 | ... |
| 83 | } |
| 84 | ] |
| 85 | } |
| 86 | ``` |
| 87 | |
| 88 | Requests can specify the LoRA adapter as if it were any other model via the `model` request parameter. The requests will be |
| 89 | processed according to the server-wide LoRA configuration (i.e. in parallel with base model requests, and potentially other |
| 90 | LoRA adapter requests if they were provided and `max_loras` is set high enough). |
| 91 | |
| 92 | The following is an example request |
| 93 | |
| 94 | ```bash |
| 95 | curl http://localhost:8000/v1/completions \ |
| 96 | -H "Content-Type: application/json" \ |
| 97 | -d '{ |
| 98 | "model": "sql-lora", |
| 99 | "prompt": "San Francisco is a", |
| 100 | "max_tokens": 7, |
| 101 | "temperature": 0 |
| 102 | }' | jq |
| 103 | ``` |
| 104 | |
| 105 | ## Dynamically serving LoRA Adapters |
| 106 | |
| 107 | In addition to serving LoRA adapters at server startup, the vLLM server supports dynamically configuring LoRA adapters at runtime through dedicated API endpoints and plugins. This feature can be particularly useful when the flexibility to change models on-the-fly is needed. |
| 108 | |
| 109 | !!! warning |
| 110 | This feature comes with security risks. It should not be used in production unless it is an isolated, fully trusted environment. |
| 111 | |
| 112 | To enable dynamic LoRA configuration, ensure that the environment variable `VLLM_ALLOW_RUNTIME_LORA_UPDATING` |
| 113 | is set to `True`. |
| 114 | |
| 115 | ```bash |
| 116 | export VLLM_ALLOW_RUNTIME_LORA_UPDATING=True |
| 117 | ``` |
| 118 | |
| 119 | ### Using API Endpoints |
| 120 | |
| 121 | Loading a LoRA Adapter: |
| 122 | |
| 123 | To dynamically load a LoRA adapter, send a POST request to the `/v1/load_lora_adapter` endpoint with the necessary |
| 124 | details of the adapter to be loaded. The request payload should include the name and path to the LoRA adapter. |
| 125 | |
| 126 | Example request to load a LoRA adapter: |
| 127 | |
| 128 | ```bash |
| 129 | curl -X POST http://localhost:8000/v1/load_lora_adapter \ |
| 130 | -H "Content-Type: application/json" \ |
| 131 | -d '{ |
| 132 | "lora_name": "sql_adapter", |
| 133 | "lora_path": "/path/to/sql-lora-adapter" |
| 134 | }' |
| 135 | ``` |
| 136 | |
| 137 | Upon a successful request, the API will respond with a `200 OK` status code from `vllm serve`, and `curl` returns the response body: `Success: LoRA adapter 'sql_adapter' added successfully`. If an error occurs, such as if the adapter |
| 138 | cannot be found or loaded, an appropriate error message will be returned. |
| 139 | |
| 140 | Unloading a LoRA Adapter: |
| 141 | |
| 142 | To unload a LoRA adapter that has been previously loaded, send a POST request to the `/v1/unload_lora_adapter` endpoint |
| 143 | with the name or ID of the adapter to be unloaded. |
| 144 | |
| 145 | Upon a successful request, the API responds with a `200 OK` status code from `vllm serve`, and `curl` returns the response body: `Success: LoRA adapter 'sql_adapter' removed successfully`. |
| 146 | |
| 147 | Example request to unload a LoRA adapter: |
| 148 | |
| 149 | ```bash |
| 150 | curl -X POST http://localhost:8000/v1/unload_lora_adapter \ |
| 151 | -H "Content-Type: application/json" \ |
| 152 | -d '{ |
| 153 | "lora_name": "sql_adapter" |
| 154 | }' |
| 155 | ``` |
| 156 | |
| 157 | ### Using Plugins |
| 158 | |
| 159 | Alternatively, you can use the LoRAResolver plugin to dynamically load LoRA adapters. LoRAResolver plugins enable you to load LoRA adapters from both local and remote sources such as local file system and S3. On every request, when there's a new model name that hasn't been loaded yet, the LoRAResolver will try to resolve and load the corresponding LoRA adapter. |
| 160 | |
| 161 | You can set up multiple LoRAResolver plugins if you want to load LoRA adapters from different sources. For example, you might have one resolver for local files and another for S3 storage. vLLM will load the first LoRA adapter that it finds. |
| 162 | |
| 163 | You can either install existing plugins or implement your own. By default, vLLM comes with a [resolver plugin to load LoRA adapters from a local directory, as well as a resolver plugin to load LoRA adapters from repositories on Hugging Face Hub](https://github.com/vllm-project/vllm/tree/main/vllm/plugins/lora_resolvers) |
| 164 | To enable either of these resolvers, you must `set VLLM_ALLOW_RUNTIME_LORA_UPDATING` to True. |
| 165 | |
| 166 | - To leverage a local directory, set `VLLM_PLUGINS` to include `lora_filesystem_resolver` and set `VLLM_LORA_RESOLVER_CACHE_DIR` to a local directory. When vLLM receives a request using a LoRA adapter `foobar`, |
| 167 | it will first look in the local directory for a directory `foobar`, and attempt to load the contents of that directory as a LoRA adapter. If successful, the request will complete as normal and that adapter will then be available for normal use on the server. |
| 168 | - To leverage repositories on Hugging Face Hub, set `VLLM_PLUGINS` to include `lora_hf_hub_resolver` and set `VLLM_LORA_RESOLVER_HF_REPO_LIST` to a comma separated list of repository IDs on Hugging Face Hub. When vLLM receives a request for the LoRA adapter `my/repo/subpath`, it will download the adapter at the `subpath` of `my/repo` if it exists and contains an `adapter_config.json`, then build a request to the cached dir for the adapter, similar to the `lora_filesystem_resolver`. Please note that enabling remote downloads is insecure and not intended for use in production environments. |
| 169 | |
| 170 | Alternatively, follow these example steps to implement your own plugin: |
| 171 | |
| 172 | 1. Implement the LoRAResolver interface. |
| 173 | |
| 174 | ??? code "Example of a simple S3 LoRAResolver implementation" |
| 175 | |
| 176 | ```python |
| 177 | import os |
| 178 | import s3fs |
| 179 | from vllm.lora.request import LoRARequest |
| 180 | from vllm.lora.resolver import LoRAResolver |
| 181 | |
| 182 | class S3LoRAResolver(LoRAResolver): |
| 183 | def __init__(self): |
| 184 | self.s3 = s3fs.S3FileSystem() |
| 185 | self.s3_path_format = os.getenv("S3_PATH_TEMPLATE") |
| 186 | self.local_path_format = os.getenv("LOCAL_PATH_TEMPLATE") |
| 187 | |
| 188 | async def resolve_lora(self, base_model_name, lora_name): |
| 189 | s3_path = self.s3_path_format.format(base_model_name=base_model_name, lora_name=lora_name) |
| 190 | local_path = self.local_path_format.format(base_model_name=base_model_name, lora_name=lora_name) |
| 191 | |
| 192 | # Download the LoRA from S3 to the local path |
| 193 | await self.s3._get( |
| 194 | s3_path, local_path, recursive=True, maxdepth=1 |
| 195 | ) |
| 196 | |
| 197 | lora_request = LoRARequest( |
| 198 | lora_name=lora_name, |
| 199 | lora_path=local_path, |
| 200 | lora_int_id=abs(hash(lora_name)), |
| 201 | ) |
| 202 | return lora_request |
| 203 | ``` |
| 204 | |
| 205 | 2. Register `LoRAResolver` plugin. |
| 206 | |
| 207 | ```python |
| 208 | from vllm.lora.resolver import LoRAResolverRegistry |
| 209 | |
| 210 | s3_resolver = S3LoRAResolver() |
| 211 | LoRAResolverRegistry.register_resolver("s3_resolver", s3_resolver) |
| 212 | ``` |
| 213 | |
| 214 | For more details, refer to the [vLLM's Plugins System](../design/plugin_system.md). |
| 215 | |
| 216 | ### In-Place LoRA Reloading |
| 217 | |
| 218 | When dynamically loading LoRA adapters, you may need to replace an existing adapter with updated weights while keeping the same name. The `load_inplace` parameter enables this functionality. This commonly occurs in asynchronous reinforcement learning setups, where adapters are continuously updated and swapped in without interrupting ongoing inference. |
| 219 | |
| 220 | When `load_inplace=True`, vLLM will replace the existing adapter with the new one. |
| 221 | |
| 222 | Example request to load or replace a LoRA adapter with the same name: |
| 223 | |
| 224 | ```bash |
| 225 | curl -X POST http://localhost:8000/v1/load_lora_adapter \ |
| 226 | -H "Content-Type: application/json" \ |
| 227 | -d '{ |
| 228 | "lora_name": "my-adapter", |
| 229 | "lora_path": "/path/to/adapter/v2", |
| 230 | "load_inplace": true |
| 231 | }' |
| 232 | ``` |
| 233 | |
| 234 | ## New format for `--lora-modules` |
| 235 | |
| 236 | In the previous version, users would provide LoRA modules via the following format, either as a key-value pair or in JSON format. For example: |
| 237 | |
| 238 | ```bash |
| 239 | --lora-modules sql-lora=jeeejeee/llama32-3b-text2sql-spider |
| 240 | ``` |
| 241 | |
| 242 | This would only include the `name` and `path` for each LoRA module, but did not provide a way to specify a `base_model_name`. |
| 243 | Now, you can specify a base_model_name alongside the name and path using JSON format. For example: |
| 244 | |
| 245 | ```bash |
| 246 | --lora-modules '{"name": "sql-lora", "path": "jeeejeee/llama32-3b-text2sql-spider", "base_model_name": "meta-llama/Llama-3.2-3B-Instruct"}' |
| 247 | ``` |
| 248 | |
| 249 | To provide the backward compatibility support, you can still use the old key-value format (name=path), but the `base_model_name` will remain unspecified in that case. |
| 250 | |
| 251 | ## Mixing 2D and 3D MoE LoRA Adapters |
| 252 | |
| 253 | To serve 2D-format(based on `megatron`) and 3D-format (based on `peft`) adapters from the same engine instance, start the server with `--enable-mixed-moe-lora-format` |
| 254 | and declare the layout of each adapter explicitly via the `is_3d_lora_weight` field. |
| 255 | |
| 256 | Server startup (static modules): |
| 257 | |
| 258 | ```bash |
| 259 | vllm serve Qwen/Qwen3.6-35B-A3B \ |
| 260 | --enable-lora \ |
| 261 | --enable-mixed-moe-lora-format \ |
| 262 | --tensor-parallel-size 4 \ |
| 263 | --enable-expert-parallel \ |
| 264 | --lora-modules \ |
| 265 | '{"name": "lora-2d", "path": "jeeejeee/qwen36-35ba3b-2d-weights-poken-lora", "is_3d_lora_weight": false}' \ |
| 266 | '{"name": "lora-3d", "path": "jeeejeee/qwen36-35ba3b-moe-all-linear-poken-lora", "is_3d_lora_weight": true}' |
| 267 | ``` |
| 268 | |
| 269 | Dynamic load via `/v1/load_lora_adapter`: |
| 270 | |
| 271 | ```bash |
| 272 | curl -X POST http://localhost:8000/v1/load_lora_adapter \ |
| 273 | -H "Content-Type: application/json" \ |
| 274 | -d '{ |
| 275 | "lora_name": "lora-3d", |
| 276 | "lora_path": "/path/to/3d-format-lora", |
| 277 | "is_3d_lora_weight": true |
| 278 | }' |
| 279 | ``` |
| 280 | |
| 281 | !!! warning "You must know your adapter's layout" |
| 282 | Under `--enable-mixed-moe-lora-format`, vLLM trusts whatever |
| 283 | `is_3d_lora_weight` the caller declares — it does **not** inspect the |
| 284 | checkpoint to verify. A wrong declaration will load weights into the |
| 285 | wrong stacked buffers and silently produce garbage outputs, with no |
| 286 | error at load time. Confirm the layout before serving: |
| 287 | |
| 288 | - **2D (per-expert, megatron-style)** → set `is_3d_lora_weight: false`. |
| 289 | Adapter keys look like `...experts.{idx}.gate_proj.lora_A.weight`, |
| 290 | `...experts.{idx}.up_proj.lora_A.weight`, |
| 291 | `...experts.{idx}.down_proj.lora_A.weight` — one set per expert. |
| 292 | - **3D (fused, peft-style)** → set `is_3d_lora_weight: true`. |
| 293 | Adapter keys look like `...experts.gate_up_proj.lora_A.weight`, |
| 294 | `...experts.down_proj.lora_A.weight` — a single tensor that stacks |
| 295 | all experts on the leading dim. |
| 296 | |
| 297 | When `--enable-mixed-moe-lora-format` is **not** set, `is_3d_lora_weight` |
| 298 | is ignored: vLLM picks the wrapper from the base model's |
| 299 | `is_3d_moe_weight` and the adapter is required to match. The field is |
| 300 | also ignored for non-MoE models. |
| 301 | |
| 302 | ## LoRA model lineage in model card |
| 303 | |
| 304 | The new format of `--lora-modules` is mainly to support the display of parent model information in the model card. Here's an explanation of how your current response supports this: |
| 305 | |
| 306 | - The `parent` field of LoRA model `sql-lora` now links to its base model `meta-llama/Llama-3.2-3B-Instruct`. This correctly reflects the hierarchical relationship between the base model and the LoRA adapter. |
| 307 | - The `root` field points to the artifact location of the lora adapter. |
| 308 | |
| 309 | ??? console "Command output" |
| 310 | |
| 311 | ```bash |
| 312 | $ curl http://localhost:8000/v1/models |
| 313 | |
| 314 | { |
| 315 | "object": "list", |
| 316 | "data": [ |
| 317 | { |
| 318 | "id": "meta-llama/Llama-3.2-3B-Instruct", |
| 319 | "object": "model", |
| 320 | "created": 1715644056, |
| 321 | "owned_by": "vllm", |
| 322 | "root": "meta-llama/Llama-3.2-3B-Instruct", |
| 323 | "parent": null, |
| 324 | "permission": [ |
| 325 | { |
| 326 | ..... |
| 327 | } |
| 328 | ] |
| 329 | }, |
| 330 | { |
| 331 | "id": "sql-lora", |
| 332 | "object": "model", |
| 333 | "created": 1715644056, |
| 334 | "owned_by": "vllm", |
| 335 | "root": "jeeejeee/llama32-3b-text2sql-spider", |
| 336 | "parent": "meta-llama/Llama-3.2-3B-Instruct", |
| 337 | "permission": [ |
| 338 | { |
| 339 | .... |
| 340 | } |
| 341 | ] |
| 342 | } |
| 343 | ] |
| 344 | } |
| 345 | ``` |
| 346 | |
| 347 | ## LoRA Support for Tower and Connector of Multi-Modal Model |
| 348 | |
| 349 | Currently, vLLM experimentally supports LoRA for the Tower and Connector components of multi-modal models. To enable this feature, you need to implement the corresponding token helper functions for the tower and connector. For more details on the rationale behind this approach, please refer to [PR 26674](https://github.com/vllm-project/vllm/pull/26674). We welcome contributions to extend LoRA support to additional models' tower and connector. Please refer to [Issue 31479](https://github.com/vllm-project/vllm/issues/31479) to check the current model support status. |
| 350 | |
| 351 | ## Default LoRA Models For Multimodal Models |
| 352 | |
| 353 | Some models, e.g., [Granite Speech](https://huggingface.co/ibm-granite/granite-speech-3.3-8b) and [Phi-4-multimodal-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct) multimodal, contain LoRA adapter(s) that are expected to always be applied when a given modality is present. This can be a bit tedious to manage with the above approaches, as it requires the user to send the `LoRARequest` (offline) or to filter requests between the base model and LoRA model (server) depending on the content of the request's multimodal data. |
| 354 | |
| 355 | To this end, we allow registration of default multimodal LoRAs to handle this automatically, where users can map each modality to a LoRA adapter to automatically apply it when the corresponding inputs are present. Note that currently, we only allow one LoRA per prompt; if several modalities are provided, each of which are registered to a given modality, none of them will be applied. |
| 356 | |
| 357 | ??? code "Example usage for offline inference" |
| 358 | |
| 359 | ```python |
| 360 | from transformers import AutoTokenizer |
| 361 | from vllm import LLM, SamplingParams |
| 362 | from vllm.assets.audio import AudioAsset |
| 363 | |
| 364 | model_id = "ibm-granite/granite-speech-3.3-2b" |
| 365 | tokenizer = AutoTokenizer.from_pretrained(model_id) |
| 366 | |
| 367 | def get_prompt(question: str, has_audio: bool): |
| 368 | """Build the input prompt to send to vLLM.""" |
| 369 | if has_audio: |
| 370 | question = f"<|audio|>{question}" |
| 371 | chat = [ |
| 372 | {"role": "user", "content": question}, |
| 373 | ] |
| 374 | return tokenizer.apply_chat_template(chat, tokenize=False) |
| 375 | |
| 376 | |
| 377 | llm = LLM( |
| 378 | model=model_id, |
| 379 | enable_lora=True, |
| 380 | max_lora_rank=64, |
| 381 | max_model_len=2048, |
| 382 | limit_mm_per_prompt={"audio": 1}, |
| 383 | # Will always pass a `LoRARequest` with the `model_id` |
| 384 | # whenever audio is contained in the request data. |
| 385 | default_mm_loras = {"audio": model_id}, |
| 386 | enforce_eager=True, |
| 387 | ) |
| 388 | |
| 389 | question = "can you transcribe the speech into a written format?" |
| 390 | prompt_with_audio = get_prompt( |
| 391 | question=question, |
| 392 | has_audio=True, |
| 393 | ) |
| 394 | audio = AudioAsset("mary_had_lamb").audio_and_sample_rate |
| 395 | |
| 396 | inputs = { |
| 397 | "prompt": prompt_with_audio, |
| 398 | "multi_modal_data": { |
| 399 | "audio": audio, |
| 400 | } |
| 401 | } |
| 402 | |
| 403 | |
| 404 | outputs = llm.generate( |
| 405 | inputs, |
| 406 | sampling_params=SamplingParams( |
| 407 | temperature=0.2, |
| 408 | max_tokens=64, |
| 409 | ), |
| 410 | ) |
| 411 | ``` |
| 412 | |
| 413 | You can also pass a json dictionary of `--default-mm-loras` mapping modalities to LoRA model IDs. For example, when starting the server: |
| 414 | |
| 415 | ```bash |
| 416 | vllm serve ibm-granite/granite-speech-3.3-2b \ |
| 417 | --max-model-len 2048 \ |
| 418 | --enable-lora \ |
| 419 | --default-mm-loras '{"audio":"ibm-granite/granite-speech-3.3-2b"}' \ |
| 420 | --max-lora-rank 64 |
| 421 | ``` |
| 422 | |
| 423 | Note: Default multimodal LoRAs are currently only available for `.generate` and chat completions. |
| 424 | |
| 425 | ## Sequence-Classification LoRA Adapters |
| 426 | |
| 427 | vLLM supports PEFT sequence-classification adapters that save a complete, single-layer linear classification head through `modules_to_save`. The saved module must be named `score` or `classifier`. |
| 428 | |
| 429 | See [classification_with_lora_offline.py](../../examples/pooling/classify/classification_with_lora_offline.py) for an offline classification example using a LoRA adapter. |
| 430 | |
| 431 | To batch adapters with different `num_labels`, set the maximum number of labels: |
| 432 | |
| 433 | ```bash |
| 434 | vllm serve model --enable-lora --max-lora-cls-labels 8 |
| 435 | ``` |
| 436 | |
| 437 | The equivalent `LLM` argument is `max_lora_cls_labels`. It defaults to the base model's `num_labels`, and each request returns its adapter's number of labels. |
| 438 | |
| 439 | This support has the following limitations: |
| 440 | |
| 441 | - A classification head stored as float32 is converted to the runtime head dtype when it is loaded. |
| 442 | - Token-classification adapters are not supported by this feature. |
| 443 | |
| 444 | ## Using Tips |
| 445 | |
| 446 | ### Configuring `max_lora_rank` |
| 447 | |
| 448 | The `--max-lora-rank` parameter controls the maximum rank allowed for LoRA adapters. This setting affects memory allocation and performance: |
| 449 | |
| 450 | - **Set it to the maximum rank** among all LoRA adapters you plan to use |
| 451 | - **Avoid setting it too high** - using a value much larger than needed wastes memory and can cause performance issues |
| 452 | |
| 453 | For example, if your LoRA adapters have ranks [16, 32, 64], use `--max-lora-rank 64` rather than 256 |
| 454 | |
| 455 | ```bash |
| 456 | # Good: matches actual maximum rank |
| 457 | vllm serve model --enable-lora --max-lora-rank 64 |
| 458 | |
| 459 | # Bad: unnecessarily high, wastes memory |
| 460 | vllm serve model --enable-lora --max-lora-rank 256 |
| 461 | ``` |
| 462 | |
| 463 | ### Restricting LoRA to Specific Modules |
| 464 | |
| 465 | The `--lora-target-modules` parameter allows you to restrict which model modules have LoRA applied at deployment time. This is useful for performance tuning when you only need LoRA on specific layers: |
| 466 | |
| 467 | ```bash |
| 468 | # Apply LoRA only to output projection layers |
| 469 | vllm serve model --enable-lora --lora-target-modules o_proj |
| 470 | |
| 471 | # Apply LoRA to multiple specific modules |
| 472 | vllm serve model --enable-lora --lora-target-modules o_proj qkv_proj down_proj |
| 473 | ``` |
| 474 | |
| 475 | When `--lora-target-modules` is not specified, LoRA will be applied to all supported modules in the model. This parameter accepts module suffixes (the last component of the module name), such as `o_proj`, `qkv_proj`, `gate_proj`, etc. |