A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 10, 07:45 UTC
[Doc] Improve LoRA docs (#32159)
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com> Signed-off-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Jee Jee Li committed
Jan 12, 2026 at 18:19 UTC
05e8981234d307c20139ffb5180b8fe2a4caf861
1 file changed
+12
-15
docs/features/lora.md
+12
-15
@@ -10,7 +10,7 @@ them locally with
10
```python
11
from huggingface_hub import snapshot_download
12
13
-sql_lora_path = snapshot_download(repo_id="yard1/llama-2-7b-sql-lora-test")
13
+sql_lora_path = snapshot_download(repo_id="jeeejeee/llama32-3b-text2sql-spider")
14
```
15
16
Then we instantiate the base model and pass in the `enable_lora=True` flag:
@@ -19,7 +19,7 @@ Then we instantiate the base model and pass in the `enable_lora=True` flag:
19
from vllm import LLM, SamplingParams
20
from vllm.lora.request import LoRARequest
21
22
-llm = LLM(model="meta-llama/Llama-2-7b-hf", enable_lora=True)
22
+llm = LLM(model="meta-llama/Llama-3.2-3B-Instruct", enable_lora=True)
23
```
24
25
We can now submit the prompts and call `llm.generate` with the `lora_request` parameter. The first parameter
@@ -55,14 +55,11 @@ LoRA adapted models can also be served with the Open-AI compatible vLLM server.
55
`--lora-modules {name}={path} {name}={path}` to specify each LoRA module when we kick off the server:
56
57
```bash
58
-vllm serve meta-llama/Llama-2-7b-hf \
58
+vllm serve meta-llama/Llama-3.2-3B-Instruct \
59
--enable-lora \
60
- --lora-modules sql-lora=$HOME/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/snapshots/0dfa347e8877a4d4ed19ee56c140fa518470028c/
60
+ --lora-modules sql-lora=jeeejeee/llama32-3b-text2sql-spider
61
```
62
63
-!!! note
64
- The commit ID `0dfa347e8877a4d4ed19ee56c140fa518470028c` may change over time. Please check the latest commit ID in your environment to ensure you are using the correct one.
65
-
63
The server entrypoint accepts all other LoRA configuration parameters (`max_loras`, `max_lora_rank`, `max_cpu_loras`,
64
etc.), which will apply to all forthcoming requests. Upon querying the `/models` endpoint, we should see our LoRA along
65
with its base model (if `jq` is not installed, you can follow [this guide](https://jqlang.org/download/) to install it.):
@@ -75,7 +72,7 @@ with its base model (if `jq` is not installed, you can follow [this guide](https
72
"object": "list",
73
"data": [
74
{
78
- "id": "meta-llama/Llama-2-7b-hf",
75
+ "id": "meta-llama/Llama-3.2-3B-Instruct",
76
"object": "model",
77
...
78
},
@@ -218,14 +215,14 @@ Alternatively, follow these example steps to implement your own plugin:
215
In the previous version, users would provide LoRA modules via the following format, either as a key-value pair or in JSON format. For example:
216
217
```bash
221
---lora-modules sql-lora=$HOME/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/snapshots/0dfa347e8877a4d4ed19ee56c140fa518470028c/
218
+--lora-modules sql-lora=jeeejeee/llama32-3b-text2sql-spider
219
```
220
221
This would only include the `name` and `path` for each LoRA module, but did not provide a way to specify a `base_model_name`.
222
Now, you can specify a base_model_name alongside the name and path using JSON format. For example:
223
224
```bash
228
---lora-modules '{"name": "sql-lora", "path": "/path/to/lora", "base_model_name": "meta-llama/Llama-2-7b"}'
225
+--lora-modules '{"name": "sql-lora", "path": "jeeejeee/llama32-3b-text2sql-spider", "base_model_name": "meta-llama/Llama-3.2-3B-Instruct"}'
226
```
227
228
To provide the backward compatibility support, you can still use the old key-value format (name=path), but the `base_model_name` will remain unspecified in that case.
@@ -234,7 +231,7 @@ To provide the backward compatibility support, you can still use the old key-val
231
232
The new format of `--lora-modules` is mainly to support the display of parent model information in the model card. Here's an explanation of how your current response supports this:
233
237
-- The `parent` field of LoRA model `sql-lora` now links to its base model `meta-llama/Llama-2-7b-hf`. This correctly reflects the hierarchical relationship between the base model and the LoRA adapter.
234
+- The `parent` field of LoRA model `sql-lora` now links to its base model `meta-llama/Llama-3.2-3B-Instruct`. This correctly reflects the hierarchical relationship between the base model and the LoRA adapter.
235
- The `root` field points to the artifact location of the lora adapter.
236
237
??? console "Command output"
@@ -246,11 +243,11 @@ The new format of `--lora-modules` is mainly to support the display of parent mo
243
"object": "list",
244
"data": [
245
{
249
- "id": "meta-llama/Llama-2-7b-hf",
246
+ "id": "meta-llama/Llama-3.2-3B-Instruct",
247
"object": "model",
248
"created": 1715644056,
249
"owned_by": "vllm",
253
- "root": "~/.cache/huggingface/hub/models--meta-llama--Llama-2-7b-hf/snapshots/01c7f73d771dfac7d292323805ebc428287df4f9/",
250
+ "root": "meta-llama/Llama-3.2-3B-Instruct",
251
"parent": null,
252
"permission": [
253
{
@@ -263,8 +260,8 @@ The new format of `--lora-modules` is mainly to support the display of parent mo
260
"object": "model",
261
"created": 1715644056,
262
"owned_by": "vllm",
266
- "root": "~/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/snapshots/0dfa347e8877a4d4ed19ee56c140fa518470028c/",
267
- "parent": meta-llama/Llama-2-7b-hf,
263
+ "root": "jeeejeee/llama32-3b-text2sql-spider",
264
+ "parent": "meta-llama/Llama-3.2-3B-Instruct",
265
"permission": [
266
{
267
....