A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 03:00 UTC
[Docs] document that mixtral 8x22b is supported (#4073)
Simon Mo committed
Apr 14, 2024 at 14:35 UTC
aceb17cf2d629175a484c3d9df355f44bd334cb3
2 files changed
+19
-19
README.md
+1
-1
@@ -72,7 +72,7 @@ vLLM seamlessly supports many Hugging Face models, including the following archi
72
- LLaMA & LLaMA-2 (`meta-llama/Llama-2-70b-hf`, `lmsys/vicuna-13b-v1.3`, `young-geng/koala`, `openlm-research/open_llama_13b`, etc.)
73
- MiniCPM (`openbmb/MiniCPM-2B-sft-bf16`, `openbmb/MiniCPM-2B-dpo-bf16`, etc.)
74
- Mistral (`mistralai/Mistral-7B-v0.1`, `mistralai/Mistral-7B-Instruct-v0.1`, etc.)
75
-- Mixtral (`mistralai/Mixtral-8x7B-v0.1`, `mistralai/Mixtral-8x7B-Instruct-v0.1`, etc.)
75
+- Mixtral (`mistralai/Mixtral-8x7B-v0.1`, `mistralai/Mixtral-8x7B-Instruct-v0.1`, `mistral-community/Mixtral-8x22B-v0.1`, etc.)
76
- MPT (`mosaicml/mpt-7b`, `mosaicml/mpt-30b`, etc.)
77
- OLMo (`allenai/OLMo-1B`, `allenai/OLMo-7B`, etc.)
78
- OPT (`facebook/opt-66b`, `facebook/opt-iml-max-30b`, etc.)
docs/source/models/supported_models.rst
+18
-18
@@ -30,23 +30,23 @@ Alongside each architecture, we include some popular models that use it.
30
* - :code:`CohereForCausalLM`
31
- Command-R
32
- :code:`CohereForAI/c4ai-command-r-v01`, etc.
33
- -
33
+ -
34
* - :code:`DbrxForCausalLM`
35
- DBRX
36
- :code:`databricks/dbrx-base`, :code:`databricks/dbrx-instruct`, etc.
37
- -
37
+ -
38
* - :code:`DeciLMForCausalLM`
39
- DeciLM
40
- :code:`Deci/DeciLM-7B`, :code:`Deci/DeciLM-7B-instruct`, etc.
41
- -
41
+ -
42
* - :code:`BloomForCausalLM`
43
- BLOOM, BLOOMZ, BLOOMChat
44
- :code:`bigscience/bloom`, :code:`bigscience/bloomz`, etc.
45
- -
45
+ -
46
* - :code:`FalconForCausalLM`
47
- Falcon
48
- :code:`tiiuae/falcon-7b`, :code:`tiiuae/falcon-40b`, :code:`tiiuae/falcon-rw-7b`, etc.
49
- -
49
+ -
50
* - :code:`GemmaForCausalLM`
51
- Gemma
52
- :code:`google/gemma-2b`, :code:`google/gemma-7b`, etc.
@@ -54,19 +54,19 @@ Alongside each architecture, we include some popular models that use it.
54
* - :code:`GPT2LMHeadModel`
55
- GPT-2
56
- :code:`gpt2`, :code:`gpt2-xl`, etc.
57
- -
57
+ -
58
* - :code:`GPTBigCodeForCausalLM`
59
- StarCoder, SantaCoder, WizardCoder
60
- :code:`bigcode/starcoder`, :code:`bigcode/gpt_bigcode-santacoder`, :code:`WizardLM/WizardCoder-15B-V1.0`, etc.
61
- -
61
+ -
62
* - :code:`GPTJForCausalLM`
63
- GPT-J
64
- :code:`EleutherAI/gpt-j-6b`, :code:`nomic-ai/gpt4all-j`, etc.
65
- -
65
+ -
66
* - :code:`GPTNeoXForCausalLM`
67
- GPT-NeoX, Pythia, OpenAssistant, Dolly V2, StableLM
68
- :code:`EleutherAI/gpt-neox-20b`, :code:`EleutherAI/pythia-12b`, :code:`OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5`, :code:`databricks/dolly-v2-12b`, :code:`stabilityai/stablelm-tuned-alpha-7b`, etc.
69
- -
69
+ -
70
* - :code:`InternLMForCausalLM`
71
- InternLM
72
- :code:`internlm/internlm-7b`, :code:`internlm/internlm-chat-7b`, etc.
@@ -93,32 +93,32 @@ Alongside each architecture, we include some popular models that use it.
93
- ✅︎
94
* - :code:`MixtralForCausalLM`
95
- Mixtral-8x7B, Mixtral-8x7B-Instruct
96
- - :code:`mistralai/Mixtral-8x7B-v0.1`, :code:`mistralai/Mixtral-8x7B-Instruct-v0.1`, etc.
96
+ - :code:`mistralai/Mixtral-8x7B-v0.1`, :code:`mistralai/Mixtral-8x7B-Instruct-v0.1`, :code:`mistral-community/Mixtral-8x22B-v0.1`, etc.
97
- ✅︎
98
* - :code:`MPTForCausalLM`
99
- MPT, MPT-Instruct, MPT-Chat, MPT-StoryWriter
100
- :code:`mosaicml/mpt-7b`, :code:`mosaicml/mpt-7b-storywriter`, :code:`mosaicml/mpt-30b`, etc.
101
- -
101
+ -
102
* - :code:`OLMoForCausalLM`
103
- OLMo
104
- :code:`allenai/OLMo-1B`, :code:`allenai/OLMo-7B`, etc.
105
- -
105
+ -
106
* - :code:`OPTForCausalLM`
107
- OPT, OPT-IML
108
- :code:`facebook/opt-66b`, :code:`facebook/opt-iml-max-30b`, etc.
109
- -
109
+ -
110
* - :code:`OrionForCausalLM`
111
- Orion
112
- :code:`OrionStarAI/Orion-14B-Base`, :code:`OrionStarAI/Orion-14B-Chat`, etc.
113
- -
113
+ -
114
* - :code:`PhiForCausalLM`
115
- Phi
116
- :code:`microsoft/phi-1_5`, :code:`microsoft/phi-2`, etc.
117
- -
117
+ -
118
* - :code:`QWenLMHeadModel`
119
- Qwen
120
- :code:`Qwen/Qwen-7B`, :code:`Qwen/Qwen-7B-Chat`, etc.
121
- -
121
+ -
122
* - :code:`Qwen2ForCausalLM`
123
- Qwen2
124
- :code:`Qwen/Qwen2-beta-7B`, :code:`Qwen/Qwen2-beta-7B-Chat`, etc.
@@ -126,11 +126,11 @@ Alongside each architecture, we include some popular models that use it.
126
* - :code:`Qwen2MoeForCausalLM`
127
- Qwen2MoE
128
- :code:`Qwen/Qwen1.5-MoE-A2.7B`, :code:`Qwen/Qwen1.5-MoE-A2.7B-Chat`, etc.
129
- -
129
+ -
130
* - :code:`StableLmForCausalLM`
131
- StableLM
132
- :code:`stabilityai/stablelm-3b-4e1t/` , :code:`stabilityai/stablelm-base-alpha-7b-v2`, etc.
133
- -
133
+ -
134
135
If your model uses one of the above model architectures, you can seamlessly run your model with vLLM.
136
Otherwise, please refer to :ref:`Adding a New Model <adding_a_new_model>` for instructions on how to implement support for your model.