@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 657061fdce
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 04:15 UTC

[docs] Add LoRA support information for models (#3299)

Philipp Moritz committed Mar 11, 2024 at 00:54 UTC 657061fdced8a33a60c1b09f5da2525de9da8f03
2 files changed +28 -2
docs/source/models/lora.rst
+2 -1
@@ -92,7 +92,8 @@ LoRA adapter requests if they were provided and ``max_loras`` is set high enough
92
93 The following is an example request
94
95 -.. code-block::bash
95 +.. code-block:: bash
96 +
97 curl http://localhost:8000/v1/completions \
98 -H "Content-Type: application/json" \
99 -d '{
docs/source/models/supported_models.rst
+26 -1
@@ -8,84 +8,109 @@ The following is the list of model architectures that are currently supported by
8 Alongside each architecture, we include some popular models that use it.
9
10 .. list-table::
11 - :widths: 25 25 50
11 + :widths: 25 25 50 5
12 :header-rows: 1
13
14 * - Architecture
15 - Models
16 - Example HuggingFace Models
17 + - :ref:`LoRA <lora>`
18 * - :code:`AquilaForCausalLM`
19 - Aquila
20 - :code:`BAAI/Aquila-7B`, :code:`BAAI/AquilaChat-7B`, etc.
21 + - ✅︎
22 * - :code:`BaiChuanForCausalLM`
23 - Baichuan
24 - :code:`baichuan-inc/Baichuan2-13B-Chat`, :code:`baichuan-inc/Baichuan-7B`, etc.
25 + -
26 * - :code:`ChatGLMModel`
27 - ChatGLM
28 - :code:`THUDM/chatglm2-6b`, :code:`THUDM/chatglm3-6b`, etc.
29 + -
30 * - :code:`DeciLMForCausalLM`
31 - DeciLM
32 - :code:`Deci/DeciLM-7B`, :code:`Deci/DeciLM-7B-instruct`, etc.
33 + -
34 * - :code:`BloomForCausalLM`
35 - BLOOM, BLOOMZ, BLOOMChat
36 - :code:`bigscience/bloom`, :code:`bigscience/bloomz`, etc.
37 + -
38 * - :code:`FalconForCausalLM`
39 - Falcon
40 - :code:`tiiuae/falcon-7b`, :code:`tiiuae/falcon-40b`, :code:`tiiuae/falcon-rw-7b`, etc.
41 + -
42 * - :code:`GemmaForCausalLM`
43 - Gemma
44 - :code:`google/gemma-2b`, :code:`google/gemma-7b`, etc.
45 + - ✅︎
46 * - :code:`GPT2LMHeadModel`
47 - GPT-2
48 - :code:`gpt2`, :code:`gpt2-xl`, etc.
49 + -
50 * - :code:`GPTBigCodeForCausalLM`
51 - StarCoder, SantaCoder, WizardCoder
52 - :code:`bigcode/starcoder`, :code:`bigcode/gpt_bigcode-santacoder`, :code:`WizardLM/WizardCoder-15B-V1.0`, etc.
53 + -
54 * - :code:`GPTJForCausalLM`
55 - GPT-J
56 - :code:`EleutherAI/gpt-j-6b`, :code:`nomic-ai/gpt4all-j`, etc.
57 + -
58 * - :code:`GPTNeoXForCausalLM`
59 - GPT-NeoX, Pythia, OpenAssistant, Dolly V2, StableLM
60 - :code:`EleutherAI/gpt-neox-20b`, :code:`EleutherAI/pythia-12b`, :code:`OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5`, :code:`databricks/dolly-v2-12b`, :code:`stabilityai/stablelm-tuned-alpha-7b`, etc.
61 + -
62 * - :code:`InternLMForCausalLM`
63 - InternLM
64 - :code:`internlm/internlm-7b`, :code:`internlm/internlm-chat-7b`, etc.
65 + - ✅︎
66 * - :code:`InternLM2ForCausalLM`
67 - InternLM2
68 - :code:`internlm/internlm2-7b`, :code:`internlm/internlm2-chat-7b`, etc.
69 + -
70 * - :code:`LlamaForCausalLM`
71 - LLaMA, LLaMA-2, Vicuna, Alpaca, Yi
72 - :code:`meta-llama/Llama-2-13b-hf`, :code:`meta-llama/Llama-2-70b-hf`, :code:`openlm-research/open_llama_13b`, :code:`lmsys/vicuna-13b-v1.3`, :code:`01-ai/Yi-6B`, :code:`01-ai/Yi-34B`, etc.
73 + - ✅︎
74 * - :code:`MistralForCausalLM`
75 - Mistral, Mistral-Instruct
76 - :code:`mistralai/Mistral-7B-v0.1`, :code:`mistralai/Mistral-7B-Instruct-v0.1`, etc.
77 + - ✅︎
78 * - :code:`MixtralForCausalLM`
79 - Mixtral-8x7B, Mixtral-8x7B-Instruct
80 - :code:`mistralai/Mixtral-8x7B-v0.1`, :code:`mistralai/Mixtral-8x7B-Instruct-v0.1`, etc.
81 + - ✅︎
82 * - :code:`MPTForCausalLM`
83 - MPT, MPT-Instruct, MPT-Chat, MPT-StoryWriter
84 - :code:`mosaicml/mpt-7b`, :code:`mosaicml/mpt-7b-storywriter`, :code:`mosaicml/mpt-30b`, etc.
85 + -
86 * - :code:`OLMoForCausalLM`
87 - OLMo
88 - :code:`allenai/OLMo-1B`, :code:`allenai/OLMo-7B`, etc.
89 + -
90 * - :code:`OPTForCausalLM`
91 - OPT, OPT-IML
92 - :code:`facebook/opt-66b`, :code:`facebook/opt-iml-max-30b`, etc.
93 + -
94 * - :code:`OrionForCausalLM`
95 - Orion
96 - :code:`OrionStarAI/Orion-14B-Base`, :code:`OrionStarAI/Orion-14B-Chat`, etc.
97 + -
98 * - :code:`PhiForCausalLM`
99 - Phi
100 - :code:`microsoft/phi-1_5`, :code:`microsoft/phi-2`, etc.
101 + -
102 * - :code:`QWenLMHeadModel`
103 - Qwen
104 - :code:`Qwen/Qwen-7B`, :code:`Qwen/Qwen-7B-Chat`, etc.
105 + -
106 * - :code:`Qwen2ForCausalLM`
107 - Qwen2
108 - :code:`Qwen/Qwen2-beta-7B`, :code:`Qwen/Qwen2-beta-7B-Chat`, etc.
109 + - ✅︎
110 * - :code:`StableLmForCausalLM`
111 - StableLM
112 - :code:`stabilityai/stablelm-3b-4e1t/` , :code:`stabilityai/stablelm-base-alpha-7b-v2`, etc.
113 + -
114
115 If your model uses one of the above model architectures, you can seamlessly run your model with vLLM.
116 Otherwise, please refer to :ref:`Adding a New Model <adding_a_new_model>` for instructions on how to implement support for your model.