A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 01:45 UTC
[Doc] Shorten README by removing supported model list (#4796)
Zhuohan Li committed
May 13, 2024 at 16:23 UTC
ac1fbf7fd2d1fdddc7b4953eeb3acae35c62766f
2 files changed
+28
-44
README.md
+9
-38
@@ -51,41 +51,14 @@ vLLM is flexible and easy to use with:
51
- (Experimental) Prefix caching support
52
- (Experimental) Multi-lora support
53
54
-vLLM seamlessly supports many Hugging Face models, including the following architectures:
55
-
56
-- Aquila & Aquila2 (`BAAI/AquilaChat2-7B`, `BAAI/AquilaChat2-34B`, `BAAI/Aquila-7B`, `BAAI/AquilaChat-7B`, etc.)
57
-- Baichuan & Baichuan2 (`baichuan-inc/Baichuan2-13B-Chat`, `baichuan-inc/Baichuan-7B`, etc.)
58
-- BLOOM (`bigscience/bloom`, `bigscience/bloomz`, etc.)
59
-- ChatGLM (`THUDM/chatglm2-6b`, `THUDM/chatglm3-6b`, etc.)
60
-- Command-R (`CohereForAI/c4ai-command-r-v01`, etc.)
61
-- DBRX (`databricks/dbrx-base`, `databricks/dbrx-instruct` etc.)
62
-- DeciLM (`Deci/DeciLM-7B`, `Deci/DeciLM-7B-instruct`, etc.)
63
-- Falcon (`tiiuae/falcon-7b`, `tiiuae/falcon-40b`, `tiiuae/falcon-rw-7b`, etc.)
64
-- Gemma (`google/gemma-2b`, `google/gemma-7b`, etc.)
65
-- GPT-2 (`gpt2`, `gpt2-xl`, etc.)
66
-- GPT BigCode (`bigcode/starcoder`, `bigcode/gpt_bigcode-santacoder`, etc.)
67
-- GPT-J (`EleutherAI/gpt-j-6b`, `nomic-ai/gpt4all-j`, etc.)
68
-- GPT-NeoX (`EleutherAI/gpt-neox-20b`, `databricks/dolly-v2-12b`, `stabilityai/stablelm-tuned-alpha-7b`, etc.)
69
-- InternLM (`internlm/internlm-7b`, `internlm/internlm-chat-7b`, etc.)
70
-- InternLM2 (`internlm/internlm2-7b`, `internlm/internlm2-chat-7b`, etc.)
71
-- Jais (`core42/jais-13b`, `core42/jais-13b-chat`, `core42/jais-30b-v3`, `core42/jais-30b-chat-v3`, etc.)
72
-- LLaMA, Llama 2, and Meta Llama 3 (`meta-llama/Meta-Llama-3-8B-Instruct`, `meta-llama/Meta-Llama-3-70B-Instruct`, `meta-llama/Llama-2-70b-hf`, `lmsys/vicuna-13b-v1.3`, `young-geng/koala`, `openlm-research/open_llama_13b`, etc.)
73
-- MiniCPM (`openbmb/MiniCPM-2B-sft-bf16`, `openbmb/MiniCPM-2B-dpo-bf16`, etc.)
74
-- Mistral (`mistralai/Mistral-7B-v0.1`, `mistralai/Mistral-7B-Instruct-v0.1`, etc.)
75
-- Mixtral (`mistralai/Mixtral-8x7B-v0.1`, `mistralai/Mixtral-8x7B-Instruct-v0.1`, `mistral-community/Mixtral-8x22B-v0.1`, etc.)
76
-- MPT (`mosaicml/mpt-7b`, `mosaicml/mpt-30b`, etc.)
77
-- OLMo (`allenai/OLMo-1B-hf`, `allenai/OLMo-7B-hf`, etc.)
78
-- OPT (`facebook/opt-66b`, `facebook/opt-iml-max-30b`, etc.)
79
-- Orion (`OrionStarAI/Orion-14B-Base`, `OrionStarAI/Orion-14B-Chat`, etc.)
80
-- Phi (`microsoft/phi-1_5`, `microsoft/phi-2`, etc.)
81
-- Phi-3 (`microsoft/Phi-3-mini-4k-instruct`, `microsoft/Phi-3-mini-128k-instruct`, etc.)
82
-- Qwen (`Qwen/Qwen-7B`, `Qwen/Qwen-7B-Chat`, etc.)
83
-- Qwen2 (`Qwen/Qwen1.5-7B`, `Qwen/Qwen1.5-7B-Chat`, etc.)
84
-- Qwen2MoE (`Qwen/Qwen1.5-MoE-A2.7B`, `Qwen/Qwen1.5-MoE-A2.7B-Chat`, etc.)
85
-- StableLM(`stabilityai/stablelm-3b-4e1t`, `stabilityai/stablelm-base-alpha-7b-v2`, etc.)
86
-- Starcoder2(`bigcode/starcoder2-3b`, `bigcode/starcoder2-7b`, `bigcode/starcoder2-15b`, etc.)
87
-- Xverse (`xverse/XVERSE-7B-Chat`, `xverse/XVERSE-13B-Chat`, `xverse/XVERSE-65B-Chat`, etc.)
88
-- Yi (`01-ai/Yi-6B`, `01-ai/Yi-34B`, etc.)
54
+vLLM seamlessly supports most popular open-source models on HuggingFace, including:
55
+- Transformer-like LLMs (e.g., Llama)
56
+- Mixture-of-Expert LLMs (e.g., Mixtral)
57
+- Multi-modal LLMs (e.g., LLaVA)
58
+
59
+Find the full list of supported models [here](https://docs.vllm.ai/en/latest/models/supported_models.html).
60
+
61
+## Getting Started
62
63
Install vLLM with pip or [from source](https://vllm.readthedocs.io/en/latest/getting_started/installation.html#build-from-source):
64
@@ -93,9 +66,7 @@ Install vLLM with pip or [from source](https://vllm.readthedocs.io/en/latest/get
66
pip install vllm
67
```
68
96
-## Getting Started
97
-
98
-Visit our [documentation](https://vllm.readthedocs.io/en/latest/) to get started.
69
+Visit our [documentation](https://vllm.readthedocs.io/en/latest/) to learn more.
70
- [Installation](https://vllm.readthedocs.io/en/latest/getting_started/installation.html)
71
- [Quickstart](https://vllm.readthedocs.io/en/latest/getting_started/quickstart.html)
72
- [Supported Models](https://vllm.readthedocs.io/en/latest/models/supported_models.html)
docs/source/models/supported_models.rst
+19
-6
@@ -16,13 +16,21 @@ Alongside each architecture, we include some popular models that use it.
16
- Example HuggingFace Models
17
- :ref:`LoRA <lora>`
18
* - :code:`AquilaForCausalLM`
19
- - Aquila
19
+ - Aquila & Aquila2
20
- :code:`BAAI/Aquila-7B`, :code:`BAAI/AquilaChat-7B`, etc.
21
- ✅︎
22
+ * - :code:`ArcticForCausalLM`
23
+ - Arctic
24
+ - :code:`Snowflake/snowflake-arctic-base`, :code:`Snowflake/snowflake-arctic-instruct`, etc.
25
+ -
26
* - :code:`BaiChuanForCausalLM`
23
- - Baichuan
27
+ - Baichuan & Baichuan2
28
- :code:`baichuan-inc/Baichuan2-13B-Chat`, :code:`baichuan-inc/Baichuan-7B`, etc.
29
- ✅︎
30
+ * - :code:`BloomForCausalLM`
31
+ - BLOOM, BLOOMZ, BLOOMChat
32
+ - :code:`bigscience/bloom`, :code:`bigscience/bloomz`, etc.
33
+ -
34
* - :code:`ChatGLMModel`
35
- ChatGLM
36
- :code:`THUDM/chatglm2-6b`, :code:`THUDM/chatglm3-6b`, etc.
@@ -39,10 +47,6 @@ Alongside each architecture, we include some popular models that use it.
47
- DeciLM
48
- :code:`Deci/DeciLM-7B`, :code:`Deci/DeciLM-7B-instruct`, etc.
49
-
42
- * - :code:`BloomForCausalLM`
43
- - BLOOM, BLOOMZ, BLOOMChat
44
- - :code:`bigscience/bloom`, :code:`bigscience/bloomz`, etc.
45
- -
50
* - :code:`FalconForCausalLM`
51
- Falcon
52
- :code:`tiiuae/falcon-7b`, :code:`tiiuae/falcon-40b`, :code:`tiiuae/falcon-rw-7b`, etc.
@@ -135,6 +139,15 @@ Alongside each architecture, we include some popular models that use it.
139
- StableLM
140
- :code:`stabilityai/stablelm-3b-4e1t/` , :code:`stabilityai/stablelm-base-alpha-7b-v2`, etc.
141
-
142
+ * - :code:`Starcoder2ForCausalLM`
143
+ - Starcoder2
144
+ - :code:`bigcode/starcoder2-3b`, :code:`bigcode/starcoder2-7b`, :code:`bigcode/starcoder2-15b`, etc.
145
+ -
146
+ * - :code:`XverseForCausalLM`
147
+ - Xverse
148
+ - :code:`xverse/XVERSE-7B-Chat`, :code:`xverse/XVERSE-13B-Chat`, :code:`xverse/XVERSE-65B-Chat`, etc.
149
+ -
150
+
151
152
If your model uses one of the above model architectures, you can seamlessly run your model with vLLM.
153
Otherwise, please refer to :ref:`Adding a New Model <adding_a_new_model>` for instructions on how to implement support for your model.