A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 22:00 UTC
[Doc] Reorganize Supported Models by Type (#6167)
Roger Wang committed
Jul 5, 2024 at 22:59 UTC
175c43eca4e6a50e160c386c6668ae4645c0b5d1
2 files changed
+36
-18
docs/source/models/supported_models.rst
+34
-17
@@ -7,6 +7,8 @@ vLLM supports a variety of generative Transformer models in `HuggingFace Transfo
7
The following is the list of model architectures that are currently supported by vLLM.
8
Alongside each architecture, we include some popular models that use it.
9
10
+Decoder-only Language Models
11
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
12
.. list-table::
13
:widths: 25 25 50 5
14
:header-rows: 1
@@ -95,14 +97,6 @@ Alongside each architecture, we include some popular models that use it.
97
- LLaMA, Llama 2, Meta Llama 3, Vicuna, Alpaca, Yi
98
- :code:`meta-llama/Meta-Llama-3-8B-Instruct`, :code:`meta-llama/Meta-Llama-3-70B-Instruct`, :code:`meta-llama/Llama-2-13b-hf`, :code:`meta-llama/Llama-2-70b-hf`, :code:`openlm-research/open_llama_13b`, :code:`lmsys/vicuna-13b-v1.3`, :code:`01-ai/Yi-6B`, :code:`01-ai/Yi-34B`, etc.
99
- ✅︎
98
- * - :code:`LlavaForConditionalGeneration`
99
- - LLaVA-1.5
100
- - :code:`llava-hf/llava-1.5-7b-hf`, :code:`llava-hf/llava-1.5-13b-hf`, etc.
101
- -
102
- * - :code:`LlavaNextForConditionalGeneration`
103
- - LLaVA-NeXT
104
- - :code:`llava-hf/llava-v1.6-mistral-7b-hf`, :code:`llava-hf/llava-v1.6-vicuna-7b-hf`, etc.
105
- -
100
* - :code:`MiniCPMForCausalLM`
101
- MiniCPM
102
- :code:`openbmb/MiniCPM-2B-sft-bf16`, :code:`openbmb/MiniCPM-2B-dpo-bf16`, etc.
@@ -143,10 +137,6 @@ Alongside each architecture, we include some popular models that use it.
137
- Phi-3-Small
138
- :code:`microsoft/Phi-3-small-8k-instruct`, :code:`microsoft/Phi-3-small-128k-instruct`, etc.
139
-
146
- * - :code:`Phi3VForCausalLM`
147
- - Phi-3-Vision
148
- - :code:`microsoft/Phi-3-vision-128k-instruct`, etc.
149
- -
140
* - :code:`QWenLMHeadModel`
141
- Qwen
142
- :code:`Qwen/Qwen-7B`, :code:`Qwen/Qwen-7B-Chat`, etc.
@@ -172,14 +162,40 @@ Alongside each architecture, we include some popular models that use it.
162
- :code:`xverse/XVERSE-7B-Chat`, :code:`xverse/XVERSE-13B-Chat`, :code:`xverse/XVERSE-65B-Chat`, etc.
163
-
164
165
+.. note::
166
+ Currently, the ROCm version of vLLM supports Mistral and Mixtral only for context lengths up to 4096.
167
+
168
+.. _supported_vlms:
169
+
170
+Vision Language Models
171
+^^^^^^^^^^^^^^^^^^^^^^^
172
+
173
+.. list-table::
174
+ :widths: 25 25 50 5
175
+ :header-rows: 1
176
+
177
+ * - Architecture
178
+ - Models
179
+ - Example HuggingFace Models
180
+ - :ref:`LoRA <lora>`
181
+ * - :code:`LlavaForConditionalGeneration`
182
+ - LLaVA-1.5
183
+ - :code:`llava-hf/llava-1.5-7b-hf`, :code:`llava-hf/llava-1.5-13b-hf`, etc.
184
+ -
185
+ * - :code:`LlavaNextForConditionalGeneration`
186
+ - LLaVA-NeXT
187
+ - :code:`llava-hf/llava-v1.6-mistral-7b-hf`, :code:`llava-hf/llava-v1.6-vicuna-7b-hf`, etc.
188
+ -
189
+ * - :code:`Phi3VForCausalLM`
190
+ - Phi-3-Vision
191
+ - :code:`microsoft/Phi-3-vision-128k-instruct`, etc.
192
+ -
193
194
If your model uses one of the above model architectures, you can seamlessly run your model with vLLM.
177
-Otherwise, please refer to :ref:`Adding a New Model <adding_a_new_model>` for instructions on how to implement support for your model.
195
+Otherwise, please refer to :ref:`Adding a New Model <adding_a_new_model>` and :ref:`Adding a New Multimodal Model <adding_a_new_multimodal_model>`
196
+for instructions on how to implement support for your model.
197
Alternatively, you can raise an issue on our `GitHub <https://github.com/vllm-project/vllm/issues>`_ project.
198
180
-.. note::
181
- Currently, the ROCm version of vLLM supports Mistral and Mixtral only for context lengths up to 4096.
182
-
199
.. tip::
200
The easiest way to check if your model is supported is to run the program below:
201
@@ -210,8 +226,9 @@ Alternatively, you can raise an issue on our `GitHub <https://github.com/vllm-pr
226
output = llm.generate("Hello, my name is")
227
print(output)
228
229
+
230
Model Support Policy
214
----------------------
231
+=====================
232
233
At vLLM, we are committed to facilitating the integration and support of third-party models within our ecosystem. Our approach is designed to balance the need for robustness and the practical limitations of supporting a wide range of models. Here’s how we manage third-party model support:
234
docs/source/models/vlm.rst
+2
-1
@@ -3,7 +3,8 @@
3
Using VLMs
4
==========
5
6
-vLLM provides experimental support for Vision Language Models (VLMs). This document shows you how to run and serve these models using vLLM.
6
+vLLM provides experimental support for Vision Language Models (VLMs). See the :ref:`list of supported VLMs here <supported_vlms>`.
7
+This document shows you how to run and serve these models using vLLM.
8
9
.. important::
10
We are actively iterating on VLM support. Expect breaking changes to VLM usage and development in upcoming releases without prior deprecation.