A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 04:15 UTC
[docs] Add LoRA support information for models (#3299)
Philipp Moritz committed
Mar 11, 2024 at 00:54 UTC
657061fdced8a33a60c1b09f5da2525de9da8f03
2 files changed
+28
-2
docs/source/models/lora.rst
+2
-1
@@ -92,7 +92,8 @@ LoRA adapter requests if they were provided and ``max_loras`` is set high enough
92
93
The following is an example request
94
95
-.. code-block::bash
95
+.. code-block:: bash
96
+
97
curl http://localhost:8000/v1/completions \
98
-H "Content-Type: application/json" \
99
-d '{
docs/source/models/supported_models.rst
+26
-1
@@ -8,84 +8,109 @@ The following is the list of model architectures that are currently supported by
8
Alongside each architecture, we include some popular models that use it.
9
10
.. list-table::
11
- :widths: 25 25 50
11
+ :widths: 25 25 50 5
12
:header-rows: 1
13
14
* - Architecture
15
- Models
16
- Example HuggingFace Models
17
+ - :ref:`LoRA <lora>`
18
* - :code:`AquilaForCausalLM`
19
- Aquila
20
- :code:`BAAI/Aquila-7B`, :code:`BAAI/AquilaChat-7B`, etc.
21
+ - ✅︎
22
* - :code:`BaiChuanForCausalLM`
23
- Baichuan
24
- :code:`baichuan-inc/Baichuan2-13B-Chat`, :code:`baichuan-inc/Baichuan-7B`, etc.
25
+ -
26
* - :code:`ChatGLMModel`
27
- ChatGLM
28
- :code:`THUDM/chatglm2-6b`, :code:`THUDM/chatglm3-6b`, etc.
29
+ -
30
* - :code:`DeciLMForCausalLM`
31
- DeciLM
32
- :code:`Deci/DeciLM-7B`, :code:`Deci/DeciLM-7B-instruct`, etc.
33
+ -
34
* - :code:`BloomForCausalLM`
35
- BLOOM, BLOOMZ, BLOOMChat
36
- :code:`bigscience/bloom`, :code:`bigscience/bloomz`, etc.
37
+ -
38
* - :code:`FalconForCausalLM`
39
- Falcon
40
- :code:`tiiuae/falcon-7b`, :code:`tiiuae/falcon-40b`, :code:`tiiuae/falcon-rw-7b`, etc.
41
+ -
42
* - :code:`GemmaForCausalLM`
43
- Gemma
44
- :code:`google/gemma-2b`, :code:`google/gemma-7b`, etc.
45
+ - ✅︎
46
* - :code:`GPT2LMHeadModel`
47
- GPT-2
48
- :code:`gpt2`, :code:`gpt2-xl`, etc.
49
+ -
50
* - :code:`GPTBigCodeForCausalLM`
51
- StarCoder, SantaCoder, WizardCoder
52
- :code:`bigcode/starcoder`, :code:`bigcode/gpt_bigcode-santacoder`, :code:`WizardLM/WizardCoder-15B-V1.0`, etc.
53
+ -
54
* - :code:`GPTJForCausalLM`
55
- GPT-J
56
- :code:`EleutherAI/gpt-j-6b`, :code:`nomic-ai/gpt4all-j`, etc.
57
+ -
58
* - :code:`GPTNeoXForCausalLM`
59
- GPT-NeoX, Pythia, OpenAssistant, Dolly V2, StableLM
60
- :code:`EleutherAI/gpt-neox-20b`, :code:`EleutherAI/pythia-12b`, :code:`OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5`, :code:`databricks/dolly-v2-12b`, :code:`stabilityai/stablelm-tuned-alpha-7b`, etc.
61
+ -
62
* - :code:`InternLMForCausalLM`
63
- InternLM
64
- :code:`internlm/internlm-7b`, :code:`internlm/internlm-chat-7b`, etc.
65
+ - ✅︎
66
* - :code:`InternLM2ForCausalLM`
67
- InternLM2
68
- :code:`internlm/internlm2-7b`, :code:`internlm/internlm2-chat-7b`, etc.
69
+ -
70
* - :code:`LlamaForCausalLM`
71
- LLaMA, LLaMA-2, Vicuna, Alpaca, Yi
72
- :code:`meta-llama/Llama-2-13b-hf`, :code:`meta-llama/Llama-2-70b-hf`, :code:`openlm-research/open_llama_13b`, :code:`lmsys/vicuna-13b-v1.3`, :code:`01-ai/Yi-6B`, :code:`01-ai/Yi-34B`, etc.
73
+ - ✅︎
74
* - :code:`MistralForCausalLM`
75
- Mistral, Mistral-Instruct
76
- :code:`mistralai/Mistral-7B-v0.1`, :code:`mistralai/Mistral-7B-Instruct-v0.1`, etc.
77
+ - ✅︎
78
* - :code:`MixtralForCausalLM`
79
- Mixtral-8x7B, Mixtral-8x7B-Instruct
80
- :code:`mistralai/Mixtral-8x7B-v0.1`, :code:`mistralai/Mixtral-8x7B-Instruct-v0.1`, etc.
81
+ - ✅︎
82
* - :code:`MPTForCausalLM`
83
- MPT, MPT-Instruct, MPT-Chat, MPT-StoryWriter
84
- :code:`mosaicml/mpt-7b`, :code:`mosaicml/mpt-7b-storywriter`, :code:`mosaicml/mpt-30b`, etc.
85
+ -
86
* - :code:`OLMoForCausalLM`
87
- OLMo
88
- :code:`allenai/OLMo-1B`, :code:`allenai/OLMo-7B`, etc.
89
+ -
90
* - :code:`OPTForCausalLM`
91
- OPT, OPT-IML
92
- :code:`facebook/opt-66b`, :code:`facebook/opt-iml-max-30b`, etc.
93
+ -
94
* - :code:`OrionForCausalLM`
95
- Orion
96
- :code:`OrionStarAI/Orion-14B-Base`, :code:`OrionStarAI/Orion-14B-Chat`, etc.
97
+ -
98
* - :code:`PhiForCausalLM`
99
- Phi
100
- :code:`microsoft/phi-1_5`, :code:`microsoft/phi-2`, etc.
101
+ -
102
* - :code:`QWenLMHeadModel`
103
- Qwen
104
- :code:`Qwen/Qwen-7B`, :code:`Qwen/Qwen-7B-Chat`, etc.
105
+ -
106
* - :code:`Qwen2ForCausalLM`
107
- Qwen2
108
- :code:`Qwen/Qwen2-beta-7B`, :code:`Qwen/Qwen2-beta-7B-Chat`, etc.
109
+ - ✅︎
110
* - :code:`StableLmForCausalLM`
111
- StableLM
112
- :code:`stabilityai/stablelm-3b-4e1t/` , :code:`stabilityai/stablelm-base-alpha-7b-v2`, etc.
113
+ -
114
115
If your model uses one of the above model architectures, you can seamlessly run your model with vLLM.
116
Otherwise, please refer to :ref:`Adding a New Model <adding_a_new_model>` for instructions on how to implement support for your model.