A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 09:15 UTC
Modify README to include info on loading LLaMA (#18)
Zhuohan Li committed
Apr 1, 2023 at 01:07 UTC
e3f00d191e1c1fa1aae0a895ef74a2ed8edd1b30
1 file changed
+16
README.md
+16
@@ -53,3 +53,19 @@ python -m cacheflow.http_frontend.fastapi_frontend
53
# At another terminal
54
python -m cacheflow.http_frontend.gradio_webserver
55
```
56
+
57
+## Load LLaMA weights
58
+
59
+Since LLaMA weight is not fully public, we cannot directly download the LLaMA weights from huggingface. Therefore, you need to follow the following process to load the LLaMA weights.
60
+
61
+1. Converting LLaMA weights to huggingface format with [this script](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py).
62
+ ```bash
63
+ python src/transformers/models/llama/convert_llama_weights_to_hf.py \
64
+ --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path/llama-7b
65
+ ```
66
+ Please make sure that `llama` is included in the output directory name.
67
+2. For all the commands above, specify the model with `--model /output/path/llama-7b` to load the model. For example:
68
+ ```bash
69
+ python simple_server.py --model /output/path/llama-7b
70
+ python -m cacheflow.http_frontend.fastapi_frontend --model /output/path/llama-7b
71
+ ```