@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 94162beb9f
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 20:45 UTC

[Doc] Fix the lora adapter path in server startup script (#6230)

Jiaxin Shan committed Jul 16, 2024 at 10:11 UTC 94162beb9f454403d68ec009bb5572ee560d7603
1 file changed +4 -1
docs/source/models/lora.rst
+4 -1
@@ -64,7 +64,10 @@ LoRA adapted models can also be served with the Open-AI compatible vLLM server.
64 python -m vllm.entrypoints.openai.api_server \
65 --model meta-llama/Llama-2-7b-hf \
66 --enable-lora \
67 - --lora-modules sql-lora=~/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/
67 + --lora-modules sql-lora=$HOME/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/snapshots/0dfa347e8877a4d4ed19ee56c140fa518470028c/
68 +
69 +.. note::
70 + The commit ID `0dfa347e8877a4d4ed19ee56c140fa518470028c` may change over time. Please check the latest commit ID in your environment to ensure you are using the correct one.
71
72 The server entrypoint accepts all other LoRA configuration parameters (``max_loras``, ``max_lora_rank``, ``max_cpu_loras``,
73 etc.), which will apply to all forthcoming requests. Upon querying the ``/models`` endpoint, we should see our LoRA along