@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / b979143d5b
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 09:30 UTC

[Doc] Move additional tips/notes to the top (#9647)

Cyrus Leung committed Oct 24, 2024 at 17:43 UTC b979143d5bbe35192b55875f04a24de4108eb514
1 file changed +39 -40
docs/source/models/supported_models.rst
+39 -40
@@ -3,10 +3,47 @@
3 Supported Models
4 ================
5
6 -vLLM supports a variety of generative Transformer models in `HuggingFace (HF) Transformers <https://huggingface.co/models>`_.
7 -The following is the list of model architectures that are currently supported by vLLM.
6 +vLLM supports a variety of generative and embedding models from `HuggingFace (HF) Transformers <https://huggingface.co/models>`_.
7 +This page lists the model architectures that are currently supported by vLLM.
8 Alongside each architecture, we include some popular models that use it.
9
10 +For other models, you can check the :code:`config.json` file inside the model repository.
11 +If the :code:`"architectures"` field contains a model architecture listed below, then it should be supported in theory.
12 +
13 +.. tip::
14 + The easiest way to check if your model is really supported at runtime is to run the program below:
15 +
16 + .. code-block:: python
17 +
18 + from vllm import LLM
19 +
20 + llm = LLM(model=...) # Name or path of your model
21 + output = llm.generate("Hello, my name is")
22 + print(output)
23 +
24 + If vLLM successfully generates text, it indicates that your model is supported.
25 +
26 +Otherwise, please refer to :ref:`Adding a New Model <adding_a_new_model>` and :ref:`Enabling Multimodal Inputs <enabling_multimodal_inputs>`
27 +for instructions on how to implement your model in vLLM.
28 +Alternatively, you can `open an issue on GitHub <https://github.com/vllm-project/vllm/issues/new/choose>`_ to request vLLM support.
29 +
30 +.. note::
31 + To use models from `ModelScope <https://www.modelscope.cn>`_ instead of HuggingFace Hub, set an environment variable:
32 +
33 + .. code-block:: shell
34 +
35 + $ export VLLM_USE_MODELSCOPE=True
36 +
37 + And use with :code:`trust_remote_code=True`.
38 +
39 + .. code-block:: python
40 +
41 + from vllm import LLM
42 +
43 + llm = LLM(model=..., revision=..., trust_remote_code=True) # Name or path of your model
44 + output = llm.generate("Hello, my name is")
45 + print(output)
46 +
47 Text-only Language Models
48 ^^^^^^^^^^^^^^^^^^^^^^^^^
49
@@ -515,44 +552,6 @@ Multimodal Embedding
552 Some model architectures support both generation and embedding tasks.
553 In this case, you have to pass :code:`--task embedding` to run the model in embedding mode.
554
518 -----
519 -
520 -If your model uses one of the above model architectures, you can seamlessly run your model with vLLM.
521 -Otherwise, please refer to :ref:`Adding a New Model <adding_a_new_model>` and :ref:`Enabling Multimodal Inputs <enabling_multimodal_inputs>`
522 -for instructions on how to implement support for your model.
523 -Alternatively, you can raise an issue on our `GitHub <https://github.com/vllm-project/vllm/issues>`_ project.
524 -
525 -.. tip::
526 - The easiest way to check if your model is supported is to run the program below:
527 -
528 - .. code-block:: python
529 -
530 - from vllm import LLM
531 -
532 - llm = LLM(model=...) # Name or path of your model
533 - output = llm.generate("Hello, my name is")
534 - print(output)
535 -
536 - If vLLM successfully generates text, it indicates that your model is supported.
537 -
538 -.. tip::
539 - To use models from `ModelScope <https://www.modelscope.cn>`_ instead of HuggingFace Hub, set an environment variable:
540 -
541 - .. code-block:: shell
542 -
543 - $ export VLLM_USE_MODELSCOPE=True
544 -
545 - And use with :code:`trust_remote_code=True`.
546 -
547 - .. code-block:: python
548 -
549 - from vllm import LLM
550 -
551 - llm = LLM(model=..., revision=..., trust_remote_code=True) # Name or path of your model
552 - output = llm.generate("Hello, my name is")
553 - print(output)
554 -
555 -
555 Model Support Policy
556 =====================
557