@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 62ec38ea41
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 09:15 UTC

Document supported models (#127)

Woosuk Kwon committed Jun 2, 2023 at 22:35 UTC 62ec38ea4148bb8147f346f7e01cab6b8a2ec7b6
5 files changed +58 -3
cacheflow/entrypoints/llm.py
+3 -1
@@ -39,11 +39,13 @@ class LLM:
39
40 def generate(
41 self,
42 - prompts: List[str],
42 + prompts: Union[str, List[str]],
43 sampling_params: Optional[SamplingParams] = None,
44 prompt_token_ids: Optional[List[List[int]]] = None,
45 use_tqdm: bool = True,
46 ) -> List[RequestOutput]:
47 + if isinstance(prompts, str):
48 + prompts = [prompts]
49 if sampling_params is None:
50 # Use default sampling params.
51 sampling_params = SamplingParams()
docs/README.md
+1 -2
@@ -14,7 +14,6 @@ make html
14 ## Open the docs with your brower
15
16 ```bash
17 -cd build/html
18 -python -m http.server
17 +python -m http.server -d build/html/
18 ```
19 Launch your browser and open localhost:8000.
docs/source/index.rst
+7
@@ -10,3 +10,10 @@ Documentation
10
11 getting_started/installation
12 getting_started/quickstart
13 +
14 +.. toctree::
15 + :maxdepth: 1
16 + :caption: Models
17 +
18 + models/supported_models
19 + models/adding_model
docs/source/models/adding_model.rst new
+7
@@ -0,0 +1,7 @@
1 +.. _adding_a_new_model:
2 +
3 +Adding a New Model
4 +==================
5 +
6 +
7 +Placeholder
docs/source/models/supported_models.rst new
+40
@@ -0,0 +1,40 @@
1 +.. _supported_models:
2 +
3 +Supported Models
4 +================
5 +
6 +CacheFlow supports a variety of generative Transformer models in `HuggingFace Transformers <https://github.com/huggingface/transformers>`_.
7 +The following is the list of model architectures that are currently supported by CacheFlow.
8 +Alongside each architecture, we include some popular models that use it.
9 +
10 +.. list-table::
11 + :widths: 25 75
12 + :header-rows: 1
13 +
14 + * - Architecture
15 + - Models
16 + * - :code:`GPT2LMHeadModel`
17 + - GPT-2
18 + * - :code:`GPTNeoXForCausalLM`
19 + - GPT-NeoX, Pythia, OpenAssistant, Dolly V2, StableLM
20 + * - :code:`LlamaForCausalLM`
21 + - LLaMA, Vicuna, Alpaca, Koala
22 + * - :code:`OPTForCausalLM`
23 + - OPT, OPT-IML
24 +
25 +If your model uses one of the above model architectures, you can seamlessly run your model with CacheFlow.
26 +Otherwise, please refer to :ref:`Adding a New Model <adding_a_new_model>` for instructions on how to implement support for your model.
27 +Alternatively, you can raise an issue on our `GitHub <https://github.com/WoosukKwon/cacheflow/issues>`_ project.
28 +
29 +.. tip::
30 + The easiest way to check if your model is supported is to run the program below:
31 +
32 + .. code-block:: python
33 +
34 + from cacheflow import LLM
35 +
36 + llm = LLM(model=...) # Name or path of your model
37 + output = llm.generate("Hello, my name is")
38 + print(output)
39 +
40 + If CacheFlow successfully generates text, it indicates that your model is supported.