@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 4ca2c358b1
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 05:30 UTC

Add documentation section about LoRA (#2834)

Philipp Moritz committed Feb 12, 2024 at 08:24 UTC 4ca2c358b178d5e026db925a1ed9f8945010a98f
2 files changed +53
docs/source/index.rst
+1
@@ -82,6 +82,7 @@ Documentation
82 models/supported_models
83 models/adding_model
84 models/engine_args
85 + models/lora
86
87 .. toctree::
88 :maxdepth: 1
docs/source/models/lora.rst new
+52
@@ -0,0 +1,52 @@
1 +.. _lora:
2 +
3 +Using LoRA adapters
4 +===================
5 +
6 +This document shows you how to use `LoRA adapters <https://arxiv.org/abs/2106.09685>`_ with vLLM on top of a base model.
7 +Adapters can be efficiently served on a per request basis with minimal overhead. First we download the adapter(s) and save
8 +them locally with
9 +
10 +.. code-block:: python
11 +
12 + from huggingface_hub import snapshot_download
13 +
14 + sql_lora_path = snapshot_download(repo_id="yard1/llama-2-7b-sql-lora-test")
15 +
16 +
17 +Then we instantiate the base model and pass in the ``enable_lora=True`` flag:
18 +
19 +.. code-block:: python
20 +
21 + from vllm import LLM, SamplingParams
22 + from vllm.lora.request import LoRARequest
23 +
24 + llm = LLM(model="meta-llama/Llama-2-7b-hf", enable_lora=True)
25 +
26 +
27 +We can now submit the prompts and call ``llm.generate`` with the ``lora_request`` parameter. The first parameter
28 +of ``LoRARequest`` is a human identifiable name, the second parameter is a globally unique ID for the adapter and
29 +the third parameter is the path to the LoRA adapter.
30 +
31 +.. code-block:: python
32 +
33 + sampling_params = SamplingParams(
34 + temperature=0,
35 + max_tokens=256,
36 + stop=["[/assistant]"]
37 + )
38 +
39 + prompts = [
40 + "[user] Write a SQL query to answer the question based on the table schema.\n\n context: CREATE TABLE table_name_74 (icao VARCHAR, airport VARCHAR)\n\n question: Name the ICAO for lilongwe international airport [/user] [assistant]",
41 + "[user] Write a SQL query to answer the question based on the table schema.\n\n context: CREATE TABLE table_name_11 (nationality VARCHAR, elector VARCHAR)\n\n question: When Anchero Pantaleone was the elector what is under nationality? [/user] [assistant]",
42 + ]
43 +
44 + outputs = llm.generate(
45 + prompts,
46 + sampling_params,
47 + lora_request=LoRARequest("sql_adapter", 1, sql_lora_path)
48 + )
49 +
50 +
51 +Check out `examples/multilora_inference.py <https://github.com/vllm-project/vllm/blob/main/examples/multilora_inference.py>`_
52 +for an example of how to use LoRA adapters with the async engine and how to use more advanced configuration options.
\ No newline at end of file