@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / af7c4a92e6
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 05:45 UTC

[Doc][V1] Add V1 support column for multimodal models (#10998)

Signed-off-by: Roger Wang <ywang@roblox.com>

Roger Wang committed Dec 8, 2024 at 22:29 UTC af7c4a92e654684066e61518d6ed90feda983635
1 file changed +25 -1
docs/source/models/supported_models.rst
+25 -1
@@ -495,7 +495,7 @@ Text Generation
495 ---------------
496
497 .. list-table::
498 - :widths: 25 25 15 25 5 5
498 + :widths: 25 25 15 20 5 5 5
499 :header-rows: 1
500
501 * - Architecture
@@ -504,47 +504,55 @@ Text Generation
504 - Example HF Models
505 - :ref:`LoRA <lora>`
506 - :ref:`PP <distributed_serving>`
507 + - V1
508 * - :code:`AriaForConditionalGeneration`
509 - Aria
510 - T + I
511 - :code:`rhymes-ai/Aria`
512 -
513 - ✅︎
514 + -
515 * - :code:`Blip2ForConditionalGeneration`
516 - BLIP-2
517 - T + I\ :sup:`E`
518 - :code:`Salesforce/blip2-opt-2.7b`, :code:`Salesforce/blip2-opt-6.7b`, etc.
519 -
520 - ✅︎
521 + -
522 * - :code:`ChameleonForConditionalGeneration`
523 - Chameleon
524 - T + I
525 - :code:`facebook/chameleon-7b` etc.
526 -
527 - ✅︎
528 + -
529 * - :code:`FuyuForCausalLM`
530 - Fuyu
531 - T + I
532 - :code:`adept/fuyu-8b` etc.
533 -
534 - ✅︎
535 + -
536 * - :code:`ChatGLMModel`
537 - GLM-4V
538 - T + I
539 - :code:`THUDM/glm-4v-9b` etc.
540 - ✅︎
541 - ✅︎
542 + -
543 * - :code:`H2OVLChatModel`
544 - H2OVL
545 - T + I\ :sup:`E+`
546 - :code:`h2oai/h2ovl-mississippi-800m`, :code:`h2oai/h2ovl-mississippi-2b`, etc.
547 -
548 - ✅︎
549 + -
550 * - :code:`Idefics3ForConditionalGeneration`
551 - Idefics3
552 - T + I
553 - :code:`HuggingFaceM4/Idefics3-8B-Llama3` etc.
554 - ✅︎
555 + -
556 -
557 * - :code:`InternVLChatModel`
558 - InternVL 2.5, Mono-InternVL, InternVL 2.0
@@ -552,96 +560,112 @@ Text Generation
560 - :code:`OpenGVLab/InternVL2_5-4B`, :code:`OpenGVLab/Mono-InternVL-2B`, :code:`OpenGVLab/InternVL2-4B`, etc.
561 -
562 - ✅︎
563 + - ✅︎
564 * - :code:`LlavaForConditionalGeneration`
565 - LLaVA-1.5
566 - T + I\ :sup:`E+`
567 - :code:`llava-hf/llava-1.5-7b-hf`, :code:`TIGER-Lab/Mantis-8B-siglip-llama3` (see note), etc.
568 -
569 - ✅︎
570 + - ✅︎
571 * - :code:`LlavaNextForConditionalGeneration`
572 - LLaVA-NeXT
573 - T + I\ :sup:`E+`
574 - :code:`llava-hf/llava-v1.6-mistral-7b-hf`, :code:`llava-hf/llava-v1.6-vicuna-7b-hf`, etc.
575 -
576 - ✅︎
577 + -
578 * - :code:`LlavaNextVideoForConditionalGeneration`
579 - LLaVA-NeXT-Video
580 - T + V
581 - :code:`llava-hf/LLaVA-NeXT-Video-7B-hf`, etc.
582 -
583 - ✅︎
584 + -
585 * - :code:`LlavaOnevisionForConditionalGeneration`
586 - LLaVA-Onevision
587 - T + I\ :sup:`+` + V\ :sup:`+`
588 - :code:`llava-hf/llava-onevision-qwen2-7b-ov-hf`, :code:`llava-hf/llava-onevision-qwen2-0.5b-ov-hf`, etc.
589 -
590 - ✅︎
591 + -
592 * - :code:`MiniCPMV`
593 - MiniCPM-V
594 - T + I\ :sup:`E+`
595 - :code:`openbmb/MiniCPM-V-2` (see note), :code:`openbmb/MiniCPM-Llama3-V-2_5`, :code:`openbmb/MiniCPM-V-2_6`, etc.
596 - ✅︎
597 - ✅︎
598 + -
599 * - :code:`MllamaForConditionalGeneration`
600 - Llama 3.2
601 - T + I\ :sup:`+`
602 - :code:`meta-llama/Llama-3.2-90B-Vision-Instruct`, :code:`meta-llama/Llama-3.2-11B-Vision`, etc.
603 -
604 -
605 + -
606 * - :code:`MolmoForCausalLM`
607 - Molmo
608 - T + I
609 - :code:`allenai/Molmo-7B-D-0924`, :code:`allenai/Molmo-72B-0924`, etc.
610 -
611 - ✅︎
612 + - ✅︎
613 * - :code:`NVLM_D_Model`
614 - NVLM-D 1.0
615 - T + I\ :sup:`E+`
616 - :code:`nvidia/NVLM-D-72B`, etc.
617 -
618 - ✅︎
619 + - ✅︎
620 * - :code:`PaliGemmaForConditionalGeneration`
621 - PaliGemma
622 - T + I\ :sup:`E`
623 - :code:`google/paligemma-3b-pt-224`, :code:`google/paligemma-3b-mix-224`, etc.
624 -
625 - ✅︎
626 + -
627 * - :code:`Phi3VForCausalLM`
628 - Phi-3-Vision, Phi-3.5-Vision
629 - T + I\ :sup:`E+`
630 - :code:`microsoft/Phi-3-vision-128k-instruct`, :code:`microsoft/Phi-3.5-vision-instruct` etc.
631 -
632 - ✅︎
633 + - ✅︎
634 * - :code:`PixtralForConditionalGeneration`
635 - Pixtral
636 - T + I\ :sup:`+`
637 - :code:`mistralai/Pixtral-12B-2409`, :code:`mistral-community/pixtral-12b` etc.
638 -
639 - ✅︎
640 + - ✅︎
641 * - :code:`QWenLMHeadModel`
642 - Qwen-VL
643 - T + I\ :sup:`E+`
644 - :code:`Qwen/Qwen-VL`, :code:`Qwen/Qwen-VL-Chat`, etc.
645 - ✅︎
646 - ✅︎
647 + -
648 * - :code:`Qwen2AudioForConditionalGeneration`
649 - Qwen2-Audio
650 - T + A\ :sup:`+`
651 - :code:`Qwen/Qwen2-Audio-7B-Instruct`
652 -
653 - ✅︎
654 + -
655 * - :code:`Qwen2VLForConditionalGeneration`
656 - Qwen2-VL
657 - T + I\ :sup:`E+` + V\ :sup:`E+`
658 - :code:`Qwen/Qwen2-VL-2B-Instruct`, :code:`Qwen/Qwen2-VL-7B-Instruct`, :code:`Qwen/Qwen2-VL-72B-Instruct`, etc.
659 - ✅︎
660 - ✅︎
661 + -
662 * - :code:`UltravoxModel`
663 - Ultravox
664 - T + A\ :sup:`E+`
665 - :code:`fixie-ai/ultravox-v0_3`
666 -
667 - ✅︎
668 + -
669
670 | :sup:`E` Pre-computed embeddings can be inputted for this modality.
671 | :sup:`+` Multiple items can be inputted per text prompt for this modality.