A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 09:15 UTC
Document supported models (#127)
Woosuk Kwon committed
Jun 2, 2023 at 22:35 UTC
62ec38ea4148bb8147f346f7e01cab6b8a2ec7b6
5 files changed
+58
-3
cacheflow/entrypoints/llm.py
+3
-1
@@ -39,11 +39,13 @@ class LLM:
39
40
def generate(
41
self,
42
- prompts: List[str],
42
+ prompts: Union[str, List[str]],
43
sampling_params: Optional[SamplingParams] = None,
44
prompt_token_ids: Optional[List[List[int]]] = None,
45
use_tqdm: bool = True,
46
) -> List[RequestOutput]:
47
+ if isinstance(prompts, str):
48
+ prompts = [prompts]
49
if sampling_params is None:
50
# Use default sampling params.
51
sampling_params = SamplingParams()
docs/README.md
+1
-2
@@ -14,7 +14,6 @@ make html
14
## Open the docs with your brower
15
16
```bash
17
-cd build/html
18
-python -m http.server
17
+python -m http.server -d build/html/
18
```
19
Launch your browser and open localhost:8000.
docs/source/index.rst
+7
@@ -10,3 +10,10 @@ Documentation
10
11
getting_started/installation
12
getting_started/quickstart
13
+
14
+.. toctree::
15
+ :maxdepth: 1
16
+ :caption: Models
17
+
18
+ models/supported_models
19
+ models/adding_model
docs/source/models/adding_model.rst
new
+7
@@ -0,0 +1,7 @@
1
+.. _adding_a_new_model:
2
+
3
+Adding a New Model
4
+==================
5
+
6
+
7
+Placeholder
docs/source/models/supported_models.rst
new
+40
@@ -0,0 +1,40 @@
1
+.. _supported_models:
2
+
3
+Supported Models
4
+================
5
+
6
+CacheFlow supports a variety of generative Transformer models in `HuggingFace Transformers <https://github.com/huggingface/transformers>`_.
7
+The following is the list of model architectures that are currently supported by CacheFlow.
8
+Alongside each architecture, we include some popular models that use it.
9
+
10
+.. list-table::
11
+ :widths: 25 75
12
+ :header-rows: 1
13
+
14
+ * - Architecture
15
+ - Models
16
+ * - :code:`GPT2LMHeadModel`
17
+ - GPT-2
18
+ * - :code:`GPTNeoXForCausalLM`
19
+ - GPT-NeoX, Pythia, OpenAssistant, Dolly V2, StableLM
20
+ * - :code:`LlamaForCausalLM`
21
+ - LLaMA, Vicuna, Alpaca, Koala
22
+ * - :code:`OPTForCausalLM`
23
+ - OPT, OPT-IML
24
+
25
+If your model uses one of the above model architectures, you can seamlessly run your model with CacheFlow.
26
+Otherwise, please refer to :ref:`Adding a New Model <adding_a_new_model>` for instructions on how to implement support for your model.
27
+Alternatively, you can raise an issue on our `GitHub <https://github.com/WoosukKwon/cacheflow/issues>`_ project.
28
+
29
+.. tip::
30
+ The easiest way to check if your model is supported is to run the program below:
31
+
32
+ .. code-block:: python
33
+
34
+ from cacheflow import LLM
35
+
36
+ llm = LLM(model=...) # Name or path of your model
37
+ output = llm.generate("Hello, my name is")
38
+ print(output)
39
+
40
+ If CacheFlow successfully generates text, it indicates that your model is supported.