@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 096827c284
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 05:30 UTC

[Docs] Add notes on ROCm-supported models (#2087)

Woosuk Kwon committed Dec 13, 2023 at 09:45 UTC 096827c2846e7a769cf20e34c46b8eada444cc1e
1 file changed +10 -3
docs/source/models/supported_models.rst
+10 -3
@@ -73,6 +73,10 @@ If your model uses one of the above model architectures, you can seamlessly run
73 Otherwise, please refer to :ref:`Adding a New Model <adding_a_new_model>` for instructions on how to implement support for your model.
74 Alternatively, you can raise an issue on our `GitHub <https://github.com/vllm-project/vllm/issues>`_ project.
75
76 +.. note::
77 + Currently, the ROCm version of vLLM does not support Mixtral.
78 + Additionally, it only supports Mistral for context lengths up to 4096.
79 +
80 .. tip::
81 The easiest way to check if your model is supported is to run the program below:
82
@@ -84,12 +88,17 @@ Alternatively, you can raise an issue on our `GitHub <https://github.com/vllm-pr
88 output = llm.generate("Hello, my name is")
89 print(output)
90
87 - To use model from www.modelscope.cn
91 + If vLLM successfully generates text, it indicates that your model is supported.
92 +
93 +.. tip::
94 + To use models from `ModelScope <www.modelscope.cn>`_ instead of HuggingFace Hub, set an environment variable:
95
96 .. code-block:: shell
97
98 $ export VLLM_USE_MODELSCOPE=True
99
100 + And use with :code:`trust_remote_code=True`.
101 +
102 .. code-block:: python
103
104 from vllm import LLM
@@ -97,5 +106,3 @@ Alternatively, you can raise an issue on our `GitHub <https://github.com/vllm-pr
106 llm = LLM(model=..., revision=..., trust_remote_code=True) # Name or path of your model
107 output = llm.generate("Hello, my name is")
108 print(output)
100 -
101 - If vLLM successfully generates text, it indicates that your model is supported.