A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 09:30 UTC
[Doc] Move additional tips/notes to the top (#9647)
Cyrus Leung committed
Oct 24, 2024 at 17:43 UTC
b979143d5bbe35192b55875f04a24de4108eb514
1 file changed
+39
-40
docs/source/models/supported_models.rst
+39
-40
@@ -3,10 +3,47 @@
3
Supported Models
4
================
5
6
-vLLM supports a variety of generative Transformer models in `HuggingFace (HF) Transformers <https://huggingface.co/models>`_.
7
-The following is the list of model architectures that are currently supported by vLLM.
6
+vLLM supports a variety of generative and embedding models from `HuggingFace (HF) Transformers <https://huggingface.co/models>`_.
7
+This page lists the model architectures that are currently supported by vLLM.
8
Alongside each architecture, we include some popular models that use it.
9
10
+For other models, you can check the :code:`config.json` file inside the model repository.
11
+If the :code:`"architectures"` field contains a model architecture listed below, then it should be supported in theory.
12
+
13
+.. tip::
14
+ The easiest way to check if your model is really supported at runtime is to run the program below:
15
+
16
+ .. code-block:: python
17
+
18
+ from vllm import LLM
19
+
20
+ llm = LLM(model=...) # Name or path of your model
21
+ output = llm.generate("Hello, my name is")
22
+ print(output)
23
+
24
+ If vLLM successfully generates text, it indicates that your model is supported.
25
+
26
+Otherwise, please refer to :ref:`Adding a New Model <adding_a_new_model>` and :ref:`Enabling Multimodal Inputs <enabling_multimodal_inputs>`
27
+for instructions on how to implement your model in vLLM.
28
+Alternatively, you can `open an issue on GitHub <https://github.com/vllm-project/vllm/issues/new/choose>`_ to request vLLM support.
29
+
30
+.. note::
31
+ To use models from `ModelScope <https://www.modelscope.cn>`_ instead of HuggingFace Hub, set an environment variable:
32
+
33
+ .. code-block:: shell
34
+
35
+ $ export VLLM_USE_MODELSCOPE=True
36
+
37
+ And use with :code:`trust_remote_code=True`.
38
+
39
+ .. code-block:: python
40
+
41
+ from vllm import LLM
42
+
43
+ llm = LLM(model=..., revision=..., trust_remote_code=True) # Name or path of your model
44
+ output = llm.generate("Hello, my name is")
45
+ print(output)
46
+
47
Text-only Language Models
48
^^^^^^^^^^^^^^^^^^^^^^^^^
49
@@ -515,44 +552,6 @@ Multimodal Embedding
552
Some model architectures support both generation and embedding tasks.
553
In this case, you have to pass :code:`--task embedding` to run the model in embedding mode.
554
518
-----
519
-
520
-If your model uses one of the above model architectures, you can seamlessly run your model with vLLM.
521
-Otherwise, please refer to :ref:`Adding a New Model <adding_a_new_model>` and :ref:`Enabling Multimodal Inputs <enabling_multimodal_inputs>`
522
-for instructions on how to implement support for your model.
523
-Alternatively, you can raise an issue on our `GitHub <https://github.com/vllm-project/vllm/issues>`_ project.
524
-
525
-.. tip::
526
- The easiest way to check if your model is supported is to run the program below:
527
-
528
- .. code-block:: python
529
-
530
- from vllm import LLM
531
-
532
- llm = LLM(model=...) # Name or path of your model
533
- output = llm.generate("Hello, my name is")
534
- print(output)
535
-
536
- If vLLM successfully generates text, it indicates that your model is supported.
537
-
538
-.. tip::
539
- To use models from `ModelScope <https://www.modelscope.cn>`_ instead of HuggingFace Hub, set an environment variable:
540
-
541
- .. code-block:: shell
542
-
543
- $ export VLLM_USE_MODELSCOPE=True
544
-
545
- And use with :code:`trust_remote_code=True`.
546
-
547
- .. code-block:: python
548
-
549
- from vllm import LLM
550
-
551
- llm = LLM(model=..., revision=..., trust_remote_code=True) # Name or path of your model
552
- output = llm.generate("Hello, my name is")
553
- print(output)
554
-
555
-
555
Model Support Policy
556
=====================
557