@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / aceb17cf2d
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 03:00 UTC

[Docs] document that mixtral 8x22b is supported (#4073)

Simon Mo committed Apr 14, 2024 at 14:35 UTC aceb17cf2d629175a484c3d9df355f44bd334cb3
2 files changed +19 -19
README.md
+1 -1
@@ -72,7 +72,7 @@ vLLM seamlessly supports many Hugging Face models, including the following archi
72 - LLaMA & LLaMA-2 (`meta-llama/Llama-2-70b-hf`, `lmsys/vicuna-13b-v1.3`, `young-geng/koala`, `openlm-research/open_llama_13b`, etc.)
73 - MiniCPM (`openbmb/MiniCPM-2B-sft-bf16`, `openbmb/MiniCPM-2B-dpo-bf16`, etc.)
74 - Mistral (`mistralai/Mistral-7B-v0.1`, `mistralai/Mistral-7B-Instruct-v0.1`, etc.)
75 -- Mixtral (`mistralai/Mixtral-8x7B-v0.1`, `mistralai/Mixtral-8x7B-Instruct-v0.1`, etc.)
75 +- Mixtral (`mistralai/Mixtral-8x7B-v0.1`, `mistralai/Mixtral-8x7B-Instruct-v0.1`, `mistral-community/Mixtral-8x22B-v0.1`, etc.)
76 - MPT (`mosaicml/mpt-7b`, `mosaicml/mpt-30b`, etc.)
77 - OLMo (`allenai/OLMo-1B`, `allenai/OLMo-7B`, etc.)
78 - OPT (`facebook/opt-66b`, `facebook/opt-iml-max-30b`, etc.)
docs/source/models/supported_models.rst
+18 -18
@@ -30,23 +30,23 @@ Alongside each architecture, we include some popular models that use it.
30 * - :code:`CohereForCausalLM`
31 - Command-R
32 - :code:`CohereForAI/c4ai-command-r-v01`, etc.
33 - -
33 + -
34 * - :code:`DbrxForCausalLM`
35 - DBRX
36 - :code:`databricks/dbrx-base`, :code:`databricks/dbrx-instruct`, etc.
37 - -
37 + -
38 * - :code:`DeciLMForCausalLM`
39 - DeciLM
40 - :code:`Deci/DeciLM-7B`, :code:`Deci/DeciLM-7B-instruct`, etc.
41 - -
41 + -
42 * - :code:`BloomForCausalLM`
43 - BLOOM, BLOOMZ, BLOOMChat
44 - :code:`bigscience/bloom`, :code:`bigscience/bloomz`, etc.
45 - -
45 + -
46 * - :code:`FalconForCausalLM`
47 - Falcon
48 - :code:`tiiuae/falcon-7b`, :code:`tiiuae/falcon-40b`, :code:`tiiuae/falcon-rw-7b`, etc.
49 - -
49 + -
50 * - :code:`GemmaForCausalLM`
51 - Gemma
52 - :code:`google/gemma-2b`, :code:`google/gemma-7b`, etc.
@@ -54,19 +54,19 @@ Alongside each architecture, we include some popular models that use it.
54 * - :code:`GPT2LMHeadModel`
55 - GPT-2
56 - :code:`gpt2`, :code:`gpt2-xl`, etc.
57 - -
57 + -
58 * - :code:`GPTBigCodeForCausalLM`
59 - StarCoder, SantaCoder, WizardCoder
60 - :code:`bigcode/starcoder`, :code:`bigcode/gpt_bigcode-santacoder`, :code:`WizardLM/WizardCoder-15B-V1.0`, etc.
61 - -
61 + -
62 * - :code:`GPTJForCausalLM`
63 - GPT-J
64 - :code:`EleutherAI/gpt-j-6b`, :code:`nomic-ai/gpt4all-j`, etc.
65 - -
65 + -
66 * - :code:`GPTNeoXForCausalLM`
67 - GPT-NeoX, Pythia, OpenAssistant, Dolly V2, StableLM
68 - :code:`EleutherAI/gpt-neox-20b`, :code:`EleutherAI/pythia-12b`, :code:`OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5`, :code:`databricks/dolly-v2-12b`, :code:`stabilityai/stablelm-tuned-alpha-7b`, etc.
69 - -
69 + -
70 * - :code:`InternLMForCausalLM`
71 - InternLM
72 - :code:`internlm/internlm-7b`, :code:`internlm/internlm-chat-7b`, etc.
@@ -93,32 +93,32 @@ Alongside each architecture, we include some popular models that use it.
93 - ✅︎
94 * - :code:`MixtralForCausalLM`
95 - Mixtral-8x7B, Mixtral-8x7B-Instruct
96 - - :code:`mistralai/Mixtral-8x7B-v0.1`, :code:`mistralai/Mixtral-8x7B-Instruct-v0.1`, etc.
96 + - :code:`mistralai/Mixtral-8x7B-v0.1`, :code:`mistralai/Mixtral-8x7B-Instruct-v0.1`, :code:`mistral-community/Mixtral-8x22B-v0.1`, etc.
97 - ✅︎
98 * - :code:`MPTForCausalLM`
99 - MPT, MPT-Instruct, MPT-Chat, MPT-StoryWriter
100 - :code:`mosaicml/mpt-7b`, :code:`mosaicml/mpt-7b-storywriter`, :code:`mosaicml/mpt-30b`, etc.
101 - -
101 + -
102 * - :code:`OLMoForCausalLM`
103 - OLMo
104 - :code:`allenai/OLMo-1B`, :code:`allenai/OLMo-7B`, etc.
105 - -
105 + -
106 * - :code:`OPTForCausalLM`
107 - OPT, OPT-IML
108 - :code:`facebook/opt-66b`, :code:`facebook/opt-iml-max-30b`, etc.
109 - -
109 + -
110 * - :code:`OrionForCausalLM`
111 - Orion
112 - :code:`OrionStarAI/Orion-14B-Base`, :code:`OrionStarAI/Orion-14B-Chat`, etc.
113 - -
113 + -
114 * - :code:`PhiForCausalLM`
115 - Phi
116 - :code:`microsoft/phi-1_5`, :code:`microsoft/phi-2`, etc.
117 - -
117 + -
118 * - :code:`QWenLMHeadModel`
119 - Qwen
120 - :code:`Qwen/Qwen-7B`, :code:`Qwen/Qwen-7B-Chat`, etc.
121 - -
121 + -
122 * - :code:`Qwen2ForCausalLM`
123 - Qwen2
124 - :code:`Qwen/Qwen2-beta-7B`, :code:`Qwen/Qwen2-beta-7B-Chat`, etc.
@@ -126,11 +126,11 @@ Alongside each architecture, we include some popular models that use it.
126 * - :code:`Qwen2MoeForCausalLM`
127 - Qwen2MoE
128 - :code:`Qwen/Qwen1.5-MoE-A2.7B`, :code:`Qwen/Qwen1.5-MoE-A2.7B-Chat`, etc.
129 - -
129 + -
130 * - :code:`StableLmForCausalLM`
131 - StableLM
132 - :code:`stabilityai/stablelm-3b-4e1t/` , :code:`stabilityai/stablelm-base-alpha-7b-v2`, etc.
133 - -
133 + -
134
135 If your model uses one of the above model architectures, you can seamlessly run your model with vLLM.
136 Otherwise, please refer to :ref:`Adding a New Model <adding_a_new_model>` for instructions on how to implement support for your model.