@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 4506641212
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 18:15 UTC

[Doc] Section for Multimodal Language Models (#7719)

Roger Wang committed Aug 20, 2024 at 23:24 UTC 450664121277240b1c52611dbb7381ef2cc75355
1 file changed +13 -3
docs/source/models/supported_models.rst
+13 -3
@@ -177,51 +177,61 @@ Decoder-only Language Models
177
178 .. _supported_vlms:
179
180 -Vision Language Models
181 -^^^^^^^^^^^^^^^^^^^^^^^
180 +Multimodal Language Models
181 +^^^^^^^^^^^^^^^^^^^^^^^^^^^^
182
183 .. list-table::
184 - :widths: 25 25 50 5
184 + :widths: 25 25 25 25 5
185 :header-rows: 1
186
187 * - Architecture
188 - Models
189 + - Supported Modality(ies)
190 - Example HuggingFace Models
191 - :ref:`LoRA <lora>`
192 * - :code:`Blip2ForConditionalGeneration`
193 - BLIP-2
194 + - Image
195 - :code:`Salesforce/blip2-opt-2.7b`, :code:`Salesforce/blip2-opt-6.7b`, etc.
196 -
197 * - :code:`ChameleonForConditionalGeneration`
198 - Chameleon
199 + - Image
200 - :code:`facebook/chameleon-7b` etc.
201 -
202 * - :code:`FuyuForCausalLM`
203 - Fuyu
204 + - Image
205 - :code:`adept/fuyu-8b` etc.
206 -
207 * - :code:`InternVLChatModel`
208 - InternVL2
209 + - Image
210 - :code:`OpenGVLab/InternVL2-4B`, :code:`OpenGVLab/InternVL2-8B`, etc.
211 -
212 * - :code:`LlavaForConditionalGeneration`
213 - LLaVA-1.5
214 + - Image
215 - :code:`llava-hf/llava-1.5-7b-hf`, :code:`llava-hf/llava-1.5-13b-hf`, etc.
216 -
217 * - :code:`LlavaNextForConditionalGeneration`
218 - LLaVA-NeXT
219 + - Image
220 - :code:`llava-hf/llava-v1.6-mistral-7b-hf`, :code:`llava-hf/llava-v1.6-vicuna-7b-hf`, etc.
221 -
222 * - :code:`PaliGemmaForConditionalGeneration`
223 - PaliGemma
224 + - Image
225 - :code:`google/paligemma-3b-pt-224`, :code:`google/paligemma-3b-mix-224`, etc.
226 -
227 * - :code:`Phi3VForCausalLM`
228 - Phi-3-Vision
229 + - Image
230 - :code:`microsoft/Phi-3-vision-128k-instruct`, etc.
231 -
232 * - :code:`MiniCPMV`
233 - MiniCPM-V
234 + - Image
235 - :code:`openbmb/MiniCPM-V-2` (see note), :code:`openbmb/MiniCPM-Llama3-V-2_5`, :code:`openbmb/MiniCPM-V-2_6`, etc.
236 -
237