@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 219 lines 6.91 KB
Rendered Raw
1 # LoRA Resolver Plugins
2
3 This directory contains vLLM's LoRA resolver plugins built on the `LoRAResolver` framework.
4 They automatically discover and load LoRA adapters from a specified local storage path, eliminating the need for manual configuration or server restarts.
5
6 ## Overview
7
8 LoRA Resolver Plugins provide a flexible way to dynamically load LoRA adapters at runtime. When vLLM
9 receives a request for a LoRA adapter that hasn't been loaded yet, the resolver plugins will attempt
10 to locate and load the adapter from their configured storage locations. This enables:
11
12 - **Dynamic LoRA Loading**: Load adapters on-demand without server restarts
13 - **Multiple Storage Backends**: Support for filesystem, S3, and custom backends. The built-in `lora_filesystem_resolver` requires a local storage path, while the built-in `hf_hub_resolver` will pull LoRA adapters from Huggingface Hub and proceed in an identical manner. In general, custom resolvers can be implemented to fetch from any source.
14 - **Automatic Discovery**: Seamless integration with existing LoRA workflows
15 - **Scalable Deployment**: Centralized adapter management across multiple vLLM instances
16
17 ## Prerequisites
18
19 Before using LoRA Resolver Plugins, ensure the following environment variables are configured:
20
21 ### Required Environment Variables
22
23 1. **`VLLM_ALLOW_RUNTIME_LORA_UPDATING`**: Must be set to `true` or `1` to enable dynamic LoRA loading
24 ```bash
25 export VLLM_ALLOW_RUNTIME_LORA_UPDATING=true
26 ```
27
28 2. **`VLLM_PLUGINS`**: Must include the desired resolver plugins (comma-separated list)
29 ```bash
30 export VLLM_PLUGINS=lora_filesystem_resolver
31 ```
32
33 3. **`VLLM_LORA_RESOLVER_CACHE_DIR`**: Must be set to a valid directory path for filesystem resolver
34 ```bash
35 export VLLM_LORA_RESOLVER_CACHE_DIR=/path/to/lora/adapters
36 ```
37
38 ### Optional Environment Variables
39
40 - **`VLLM_PLUGINS`**: If not set, all available plugins will be loaded. If set to empty string, no plugins will be loaded.
41
42 ## Available Resolvers
43
44 ### lora_filesystem_resolver
45
46 The filesystem resolver is installed with vLLM by default and enables loading LoRA adapters from a local directory structure.
47
48 #### Setup Steps
49
50 1. **Create the LoRA adapter storage directory**:
51 ```bash
52 mkdir -p /path/to/lora/adapters
53 ```
54
55 2. **Set environment variables**:
56 ```bash
57 export VLLM_ALLOW_RUNTIME_LORA_UPDATING=true
58 export VLLM_PLUGINS=lora_filesystem_resolver
59 export VLLM_LORA_RESOLVER_CACHE_DIR=/path/to/lora/adapters
60 ```
61
62 3. **Start vLLM server**:
63 Your base model can be `meta-llama/Llama-2-7b-hf`. Please make sure you set up the Hugging Face token in your env var `export HF_TOKEN=xxx235`.
64 ```bash
65 vllm serve your-base-model \
66 --enable-lora
67 ```
68
69 #### Directory Structure Requirements
70
71 The filesystem resolver expects LoRA adapters to be organized in the following structure:
72
73 ```text
74 /path/to/lora/adapters/
75 ├── adapter1/
76 │ ├── adapter_config.json
77 │ ├── adapter_model.bin
78 │ └── tokenizer files (if applicable)
79 ├── adapter2/
80 │ ├── adapter_config.json
81 │ ├── adapter_model.bin
82 │ └── tokenizer files (if applicable)
83 └── ...
84 ```
85
86 Each adapter directory must contain:
87
88 - **`adapter_config.json`**: Required configuration file with the following structure:
89 ```json
90 {
91 "peft_type": "LORA",
92 "base_model_name_or_path": "your-base-model-name",
93 "r": 16,
94 "lora_alpha": 32,
95 "target_modules": ["q_proj", "v_proj"],
96 "bias": "none",
97 "modules_to_save": null,
98 "use_rslora": false,
99 "use_dora": false
100 }
101 ```
102
103 - **`adapter_model.bin`**: The LoRA adapter weights file
104
105 #### Usage Example
106
107 1. **Prepare your LoRA adapter**:
108 ```bash
109 # Assuming you have a LoRA adapter in /tmp/my_lora_adapter
110 cp -r /tmp/my_lora_adapter /path/to/lora/adapters/my_sql_adapter
111 ```
112
113 2. **Verify the directory structure**:
114 ```bash
115 ls -la /path/to/lora/adapters/my_sql_adapter/
116 # Should show: adapter_config.json, adapter_model.bin, etc.
117 ```
118
119 3. **Make a request using the adapter**:
120 ```bash
121 curl http://localhost:8000/v1/completions \
122 -H "Content-Type: application/json" \
123 -d '{
124 "model": "my_sql_adapter",
125 "prompt": "Generate a SQL query for:",
126 "max_tokens": 50,
127 "temperature": 0.1
128 }'
129 ```
130
131 #### How It Works
132
133 1. When vLLM receives a request for a LoRA adapter named `my_sql_adapter`
134 2. The filesystem resolver checks if `/path/to/lora/adapters/my_sql_adapter/` exists
135 3. If found, it validates the `adapter_config.json` file
136 4. If the configuration matches the base model and is valid, the adapter is loaded
137 5. The request is processed normally with the newly loaded adapter
138 6. The adapter remains available for future requests
139
140 ## Advanced Configuration
141
142 ### Multiple Resolvers
143
144 You can configure multiple resolver plugins to load adapters from different sources:
145
146 'lora_s3_resolver' is an example of a custom resolver you would need to implement
147
148 ```bash
149 export VLLM_PLUGINS=lora_filesystem_resolver,lora_s3_resolver
150 ```
151
152 All listed resolvers are enabled; at request time, vLLM tries them in order until one succeeds.
153
154 ### Custom Resolver Implementation
155
156 To implement your own resolver plugin:
157
158 1. **Create a new resolver class**:
159 ```python
160 from vllm.lora.resolver import LoRAResolver, LoRAResolverRegistry
161 from vllm.lora.request import LoRARequest
162
163 class CustomResolver(LoRAResolver):
164 async def resolve_lora(self, base_model_name: str, lora_name: str) -> Optional[LoRARequest]:
165 # Your custom resolution logic here
166 pass
167 ```
168
169 2. **Register the resolver**:
170 ```python
171 def register_custom_resolver():
172 resolver = CustomResolver()
173 LoRAResolverRegistry.register_resolver("Custom Resolver", resolver)
174 ```
175
176 ## Troubleshooting
177
178 ### Common Issues
179
180 1. **"VLLM_LORA_RESOLVER_CACHE_DIR must be set to a valid directory"**
181 - Ensure the directory exists and is accessible
182 - Check file permissions on the directory
183
184 2. **"LoRA adapter not found"**
185 - Verify the adapter directory name matches the requested model name
186 - Check that `adapter_config.json` exists and is valid JSON
187 - Ensure `adapter_model.bin` exists in the directory
188
189 3. **"Invalid adapter configuration"**
190 - Verify `peft_type` is set to "LORA"
191 - Check that `base_model_name_or_path` matches your base model
192 - Ensure `target_modules` is properly configured
193
194 4. **"LoRA rank exceeds maximum"**
195 - Check that `r` value in `adapter_config.json` doesn't exceed `max_lora_rank` setting
196
197 ### Debugging Tips
198
199 1. **Enable debug logging**:
200 ```bash
201 export VLLM_LOGGING_LEVEL=DEBUG
202 ```
203
204 2. **Verify environment variables**:
205 ```bash
206 echo $VLLM_ALLOW_RUNTIME_LORA_UPDATING
207 echo $VLLM_PLUGINS
208 echo $VLLM_LORA_RESOLVER_CACHE_DIR
209 ```
210
211 3. **Test adapter configuration**:
212 ```bash
213 python -c "
214 import json
215 with open('/path/to/lora/adapters/my_adapter/adapter_config.json') as f:
216 config = json.load(f)
217 print('Config valid:', config)
218 "
219 ```