A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 04:15 UTC
multi-lora documentation fix (#3064)
Ganesh Jagadeesan committed
Feb 28, 2024 at 00:26 UTC
a8683102cc0ab9c1a0c3ae1ba2b7954f78eba1b3
1 file changed
+13
-1
docs/source/models/lora.rst
+13
-1
@@ -58,7 +58,7 @@ LoRA adapted models can also be served with the Open-AI compatible vLLM server.
58
59
.. code-block:: bash
60
61
- python -m vllm.entrypoints.api_server \
61
+ python -m vllm.entrypoints.openai.api_server \
62
--model meta-llama/Llama-2-7b-hf \
63
--enable-lora \
64
--lora-modules sql-lora=~/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/
@@ -89,3 +89,15 @@ with its base model:
89
Requests can specify the LoRA adapter as if it were any other model via the ``model`` request parameter. The requests will be
90
processed according to the server-wide LoRA configuration (i.e. in parallel with base model requests, and potentially other
91
LoRA adapter requests if they were provided and ``max_loras`` is set high enough).
92
+
93
+The following is an example request
94
+
95
+.. code-block::bash
96
+ curl http://localhost:8000/v1/completions \
97
+ -H "Content-Type: application/json" \
98
+ -d '{
99
+ "model": "sql-lora",
100
+ "prompt": "San Francisco is a",
101
+ "max_tokens": 7,
102
+ "temperature": 0
103
+ }' | jq