@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / a8683102cc
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 04:15 UTC

multi-lora documentation fix (#3064)

Ganesh Jagadeesan committed Feb 28, 2024 at 00:26 UTC a8683102cc0ab9c1a0c3ae1ba2b7954f78eba1b3
1 file changed +13 -1
docs/source/models/lora.rst
+13 -1
@@ -58,7 +58,7 @@ LoRA adapted models can also be served with the Open-AI compatible vLLM server.
58
59 .. code-block:: bash
60
61 - python -m vllm.entrypoints.api_server \
61 + python -m vllm.entrypoints.openai.api_server \
62 --model meta-llama/Llama-2-7b-hf \
63 --enable-lora \
64 --lora-modules sql-lora=~/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/
@@ -89,3 +89,15 @@ with its base model:
89 Requests can specify the LoRA adapter as if it were any other model via the ``model`` request parameter. The requests will be
90 processed according to the server-wide LoRA configuration (i.e. in parallel with base model requests, and potentially other
91 LoRA adapter requests if they were provided and ``max_loras`` is set high enough).
92 +
93 +The following is an example request
94 +
95 +.. code-block::bash
96 + curl http://localhost:8000/v1/completions \
97 + -H "Content-Type: application/json" \
98 + -d '{
99 + "model": "sql-lora",
100 + "prompt": "San Francisco is a",
101 + "max_tokens": 7,
102 + "temperature": 0
103 + }' | jq