A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 08:00 UTC
Add and list supported models in README (#161)
Zhuohan Li committed
Jun 20, 2023 at 10:57 UTC
0b32a987ddb0d73dbf366c51eb5a0b7daf0d3ec0
3 files changed
+15
-1
README.md
+7
@@ -39,6 +39,13 @@ vLLM is flexible and easy to use with:
39
- Streaming outputs
40
- OpenAI-compatible API server
41
42
+vLLM seamlessly supports many Huggingface models, including the following architectures:
43
+
44
+- GPT-2 (e.g., `gpt2`, `gpt2-xl`, etc.)
45
+- GPTNeoX (e.g., `EleutherAI/gpt-neox-20b`, `databricks/dolly-v2-12b`, `stabilityai/stablelm-tuned-alpha-7b`, etc.)
46
+- LLaMA (e.g., `lmsys/vicuna-13b-v1.3`, `young-geng/koala`, `openlm-research/open_llama_13b`, etc.)
47
+- OPT (e.g., `facebook/opt-66b`, `facebook/opt-iml-max-30b`, etc.)
48
+
49
Install vLLM with pip or [from source](https://llm-serving-cacheflow.readthedocs-hosted.com/en/latest/getting_started/installation.html#build-from-source):
50
51
```bash
docs/source/conf.py
+2
@@ -53,7 +53,9 @@ copybutton_prompt_is_regexp = True
53
#
54
html_title = project
55
html_theme = 'sphinx_book_theme'
56
+html_logo = 'assets/logos/vllm-logo-text-light.png'
57
html_theme_options = {
58
+ 'logo_only': True,
59
'path_to_docs': 'docs/source',
60
'repository_url': 'https://github.com/WoosukKwon/vllm',
61
'use_repository_button': True,
docs/source/models/supported_models.rst
+6
-1
@@ -8,19 +8,24 @@ The following is the list of model architectures that are currently supported by
8
Alongside each architecture, we include some popular models that use it.
9
10
.. list-table::
11
- :widths: 25 75
11
+ :widths: 25 25 50
12
:header-rows: 1
13
14
* - Architecture
15
- Models
16
+ - Example HuggingFace Models
17
* - :code:`GPT2LMHeadModel`
18
- GPT-2
19
+ - :code:`gpt2`, :code:`gpt2-xl`, etc.
20
* - :code:`GPTNeoXForCausalLM`
21
- GPT-NeoX, Pythia, OpenAssistant, Dolly V2, StableLM
22
+ - :code:`EleutherAI/gpt-neox-20b`, :code:`EleutherAI/pythia-12b`, :code:`OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5`, :code:`databricks/dolly-v2-12b`, :code:`stabilityai/stablelm-tuned-alpha-7b`, etc.
23
* - :code:`LlamaForCausalLM`
24
- LLaMA, Vicuna, Alpaca, Koala, Guanaco
25
+ - :code:`openlm-research/open_llama_13b`, :code:`lmsys/vicuna-13b-v1.3`, :code:`young-geng/koala`, :code:`JosephusCheung/Guanaco`, etc.
26
* - :code:`OPTForCausalLM`
27
- OPT, OPT-IML
28
+ - :code:`facebook/opt-66b`, :code:`facebook/opt-iml-max-30b`, etc.
29
30
If your model uses one of the above model architectures, you can seamlessly run your model with vLLM.
31
Otherwise, please refer to :ref:`Adding a New Model <adding_a_new_model>` for instructions on how to implement support for your model.