A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 05:30 UTC
Add documentation section about LoRA (#2834)
Philipp Moritz committed
Feb 12, 2024 at 08:24 UTC
4ca2c358b178d5e026db925a1ed9f8945010a98f
2 files changed
+53
docs/source/index.rst
+1
@@ -82,6 +82,7 @@ Documentation
82
models/supported_models
83
models/adding_model
84
models/engine_args
85
+ models/lora
86
87
.. toctree::
88
:maxdepth: 1
docs/source/models/lora.rst
new
+52
@@ -0,0 +1,52 @@
1
+.. _lora:
2
+
3
+Using LoRA adapters
4
+===================
5
+
6
+This document shows you how to use `LoRA adapters <https://arxiv.org/abs/2106.09685>`_ with vLLM on top of a base model.
7
+Adapters can be efficiently served on a per request basis with minimal overhead. First we download the adapter(s) and save
8
+them locally with
9
+
10
+.. code-block:: python
11
+
12
+ from huggingface_hub import snapshot_download
13
+
14
+ sql_lora_path = snapshot_download(repo_id="yard1/llama-2-7b-sql-lora-test")
15
+
16
+
17
+Then we instantiate the base model and pass in the ``enable_lora=True`` flag:
18
+
19
+.. code-block:: python
20
+
21
+ from vllm import LLM, SamplingParams
22
+ from vllm.lora.request import LoRARequest
23
+
24
+ llm = LLM(model="meta-llama/Llama-2-7b-hf", enable_lora=True)
25
+
26
+
27
+We can now submit the prompts and call ``llm.generate`` with the ``lora_request`` parameter. The first parameter
28
+of ``LoRARequest`` is a human identifiable name, the second parameter is a globally unique ID for the adapter and
29
+the third parameter is the path to the LoRA adapter.
30
+
31
+.. code-block:: python
32
+
33
+ sampling_params = SamplingParams(
34
+ temperature=0,
35
+ max_tokens=256,
36
+ stop=["[/assistant]"]
37
+ )
38
+
39
+ prompts = [
40
+ "[user] Write a SQL query to answer the question based on the table schema.\n\n context: CREATE TABLE table_name_74 (icao VARCHAR, airport VARCHAR)\n\n question: Name the ICAO for lilongwe international airport [/user] [assistant]",
41
+ "[user] Write a SQL query to answer the question based on the table schema.\n\n context: CREATE TABLE table_name_11 (nationality VARCHAR, elector VARCHAR)\n\n question: When Anchero Pantaleone was the elector what is under nationality? [/user] [assistant]",
42
+ ]
43
+
44
+ outputs = llm.generate(
45
+ prompts,
46
+ sampling_params,
47
+ lora_request=LoRARequest("sql_adapter", 1, sql_lora_path)
48
+ )
49
+
50
+
51
+Check out `examples/multilora_inference.py <https://github.com/vllm-project/vllm/blob/main/examples/multilora_inference.py>`_
52
+for an example of how to use LoRA adapters with the async engine and how to use more advanced configuration options.
\ No newline at end of file